In previous posts I’ve been written about why you should be doing Continuous Delivery and how to avoid branches using a very simple technique called Branch By Abstraction. As we have already discussed in these articles, there is nothing wrong with branches, the problem is all about merges. So, adopting toggles may help us to have only a unique development branch. But there are some difficulties around this adoption. After a pair of years trying to find an alternative for a client we have reached a solution which allowed us to do Continuous Delivery combining both branches, Feature Toggles and a bit of automation. The idea of this post is to expose our experience in order to allow other people to experiment and suggest improvements on this solution.
The problem
Working with branches has been shown much more complex than it should be. In short I would say: the bigger is your team, the more complex is your merge. The biggest the time between merges, the more conflicts will appear. Have done a refactoring?!? Merging will be a chaos.
Merges take us too much time and are an error prone alternative, mainly when we are talking about large development teams. So, why not using a unique development branch? Adopting an unique development branch knowing that the entire code may go to production at any moment may be an easy task for companies such as Google, Facebook, Amazon, Flicker, etc (of course, they have the best software engineering the money can pay). But, in real world (a.k.a. enterprise world) it seem to be much more complex. Why? In my experience of years developing software for enterprises it is easy to notice that, generally, the teams are formed by 1/5 experts, 2/5 developers, 2/5 junior programmers or trainees. In addition, enterprises tend to add a lot of people in their projects in order to reach "high productivity" (what compromise control and quality, of course).
So, it is extremely hard to put this kind of team working on a unique branch and request them to produce production ready code.
The solution
Although we think it would be ideal to have a team of experts using Branch By Abstraction technique in a unique development branch, we know that, in real world, it hardly will be applicable. So, we have been trying to mix branches, toggles and a bit of automation in order to find a solution which fits our needs. For the happiness of our team we have found a very simple and interesting alternative. So, let me explain it and how the idea came out.
We were used to have two development branches. The Trunk, for major developments and PS for Production Support. While developers were committing new features in Trunk, the production support team were fixing bugs and implementing small improvements in PS. Initially we were used to merge PS to Trunk as soon as a new version was released (usually one merge per week basis) and, from Trunk to PS whenever business requested. Frequently, those merges took the integration team one or more days to finalize. Sometimes, bugs were introduced and some of them would not be seen until a new release was generated from Trunk (what could take months or even years). Knowing this, we have decided to use a basic premise of continuous delivery: "if it hurts, do it more often". Thinking like that we have decided to merge once a day, but we still have problems. So, we have tried to merge twice a day. Of course it got better, but nobody likes merges and do it twice a day is a pain in the ass. Then, we question ourselves: why not perform a merge per commit base? and why not automate it? At this point we decided to develop a tool and the result was the following.
In the above image, it is possible to see two developers. Dev 1 is committing in PS and Dev 2 is committing in Trunk. Note that Dev 1 have done two commits (revision 1 and revision 2) which were merged automatically to Trunk by a tool we have developed. Then Dev 2 tried to make a commit but the version control blocked it due to a conflict. So, he must first resolve the conflict in his working copy and then commit his new feature (the revision 3 in the image) in Trunk.
Whenever the team decides this feature (
revision 3) is production ready, the developer which have committed it (in this case Dev 2) must be pushing the revision to PS in order to release it as soon as possible.
A more complex scenario where the automatic merge fail can be seen in the below picture.
In this case, when Dev 1 tries to commit revision 3, our automatic merging tool shows the conflict in RED in a dashboard and stops merging. At this point, the integration team, along with the developer which has performed the commit (in this case, Dev 1), must analyze and fix the problem. After that, the merging tool must be advised to keep doing merges. As we have seen in the first picture, every time a feature is considered production ready in Trunk it must be pushed to PS in order to be released.
Advantages
- Developers have time to finalize its features (in Trunk) without being afraid of putting unfinished code in production
- There are fewer cases which actually require Feature Toggles
- Once they are rarely used to manage unfinished code
- The team which use Feature Toggles are more prepared
- Toggles are usually implemented by the production support team
- The integration team has more free time to improve the whole integration and delivery process
- Less issues resulted from merges
- More control about what is done and what is not
- Done means running in production without problems
- Developers have more responsibility
- Developers must be responsible from coding to monitoring production
- Forces the team to have focus
- New features developed in Trunk must go to production as soon as possible in order to make the automated merge tool work properly
Not so good, but not disadvantages
- New features must be pushed manually from Trunk to PS
- Although we think this task can also be automated, we like the idea of developers being involved/responsible up to production.
- The integration team is still involved to resolve conflicts
- However, the time spent in this task is nothing compared to some years ago
- Conflicts must be resolved as soon as possible
- In order to not accumulate automatic merges and keep the whole system working seamless
- Branches must be as similar as possible
- In order to avoid automatic merges conflicts. So, features must be pushed from Trunk to PS as soon as possible.
This solution is way better than others we have tried up until now. It works in real world, developer is involved from the beginning until the end, the number of Feature Toggles is reduced, the integration team has more free time to improve the process and it forces the whole team to have focus on the current development features (soon I'll be writing more about what other strategies we have used to improve focus).
Are you doing or trying to do Continuous Delivery? How do you work? What do you think of this solution? Do you like it? Do you think it may work for you? Please, leave your comments. I'll strongly appreciate them.
