This is a great example how to apply the problem solving tools help to save time of a process. Good job.

In our company, the CI/CD processes are very long (up to 2.5 hours). Since this is a blocking step in the release flow it causes holdups of our fast-paced releases. Eventually, we are getting a backlog of unreleased features, which could be mitigated by releasing many things at once, but that comes with additional risks.

If a related test is failing, try to identify what feature was affected and how, and disable it temporarily until fixed.

Create a dashboard for periodic builds, so it's visible for everyone and everyone sees that there is a problem.

Use the exactly same package for prod deployment as for staging with proper config parameter.

Use of feature flag for everything that we deliver, so things can be turned off whenever needed without affecting the user

Execute the full build once a day or so, for monitoring

Deploy first, then execute tests (or at least in parallel)

Add an additional requirement of PR: a report from the engineer which includes tests results, linters,  etc...

Create a separate CI process only for the things which are not affecting functionality. Such as linters.

Stop building all the sub projects together. We only need to build and check the sub-project that was affected

Within a separate steps of the build, combine the tasks that rely on the same preparation steps into one.

Execute the build only related to the changed of that build.

Put the responsibility of code quality on the engineers

In our company, the CI/CD processes are very long (over 2 hours). Since this is a blocking step in the release flow it causes holdups of our fast-paced releases. Eventually, we are getting a backlog of unreleased features, which could be mitigated by releasing many things at once, but that comes with additional risks.

Speeding up CI/CD processes

Projects solving problems in the software development world and related processes.

The velocity or speed of an object or the rate of any kind of process or action.  The speed may be relative or absolute, linear or rotational.

Speed

Local quality

Partial or excessive action

Cheap short-living objects

Discarding and recovering

How difficult it is to make measurements on an object or system. Complex costly, time-consuming, labor-consuming inspection or analysis operations.  The increasing cost of measuring to a satisfactory quality level.  Also includes ease of inspection.

Difficulty of detecting and measuring

Our cycle is very restrictive and blocks the developer from moving fast

Over time the CI process was build by adding more into it, and we are failing the build if anything fails

Current development, test, review, deployment cycle is very long

We cannot move fast enough with our development process

Beef up CI/CD machines. A possible short-term solution to buy us some time. But it is not infinitely scalable.

Allow the process to be looser. Manage the risk somewhere else.

Rethink the release process altogether. Remove redundant steps. Maybe we are doing things twice.

Try to find a way to optimize the processes. Maybe run things in parallel. This might get expensive.