Green/Blue Deployments with several development teams

Question

I am working in an AKS cluster environment with several clusters separated from each other.

I've been reading about Green/Blue deployments about testing a change in one cluster and then rerouting traffic to that cluster once everything is ok. For that I will have an additional cluster for each existing cluster.

My question is about having several teams deploying changes. If testing is made on a change of a single team, how do you suggest handling several teams all wanting to make changes at the same time? Also, how do you make sure a certain change testing doesn't starve other changes from being deployed and tested?

Answer 1

How many are "several"? My experience in a somewhat similar environment (*) including a few dozen teams and many experiments running in parallel is that there is no simple technical solution and it is not always possible to know what caused a regression or improvement.

What you can do is have a central, easily accessible and easy to understand, board showing what deployments are running at the moment and what they include, and educate people to check it before they deploy.

You can get bonus points by:

• Be able to lock parts of the system, either in a soft (e.g. the above board) or hard way

• Have a coordinator role, this can be a full time job or a rotating role. The later has the benefit of increasing the shared responsibility feeling. Alternatively this can be a meeting or even a mailing list where changes are published

• Have a "deployment schedule" per team or area

Note that most of the solutions might lead to lower deployment frequency and challenges in implementing a real CI/CD

(*) A/B testing in production of a complex system, not micro services or clusters but each module was developed separately while keeping an agreed API. The final pre-release testing was done using another A/B test group of testers.