Is it possible to do Continuous Delivery using branches? We think so!

In previous posts I’ve been written about why you should be doing Continuous Delivery and how to avoid branches using a very simple technique called Branch By Abstraction. As we have already discussed in these articles, there is nothing wrong with branches, the problem is all about merges. So, adopting toggles may help us to have only a unique development branch. But there are some difficulties around this adoption. After a pair of years trying to find an alternative for a client we have reached a solution which allowed us to do Continuous Delivery combining both branches, Feature Toggles and a bit of automation. The idea of this post is to expose our experience in order to allow other people to experiment and suggest improvements on this solution.

The problem

Working with branches has been shown much more complex than it should be. In short I would say: the bigger is your team, the more complex is your merge. The biggest the time between merges, the more conflicts will appear. Have done a refactoring?!? Merging will be a chaos.

Merges take us too much time and are an error prone alternative, mainly when we are talking about large development teams. So, why not using a unique development branch? Adopting an unique development branch knowing that the entire code may go to production at any moment may be an easy task for companies such as Google, Facebook, Amazon, Flicker, etc (of course, they have the best software engineering the money can pay). But, in real world (a.k.a. enterprise world) it seem to be much more complex. Why? In my experience of years developing software for enterprises it is easy to notice that, generally, the teams are formed by 1/5 experts, 2/5 developers, 2/5 junior programmers or trainees. In addition, enterprises tend to add a lot of people in their projects in order to reach "high productivity" (what compromise control and quality, of course).

So, it is extremely hard to put this kind of team working on a unique branch and request them to produce production ready code.

The solution

Although we think it would be ideal to have a team of experts using Branch By Abstraction technique in a unique development branch, we know that, in real world, it hardly will be applicable. So, we have been trying to mix branches, toggles and a bit of automation in order to find a solution which fits our needs. For the happiness of our team we have found a very simple and interesting alternative. So, let me explain it and how the idea came out.

We were used to have two development branches. The Trunk, for major developments and PS for Production Support. While developers were committing new features in Trunk, the production support team were fixing bugs and implementing small improvements in PS. Initially we were used to merge PS to Trunk as soon as a new version was released (usually one merge per week basis) and, from Trunk to PS whenever business requested. Frequently, those merges took the integration team one or more days to finalize. Sometimes, bugs were introduced and some of them would not be seen until a new release was generated from Trunk (what could take months or even years). Knowing this, we have decided to use a basic premise of continuous delivery: "if it hurts, do it more often".  Thinking like that we have decided to merge once a day, but we still have problems. So, we have tried to merge twice a day. Of course it got better, but nobody likes merges and do it twice a day is a pain in the ass. Then, we question ourselves: why not perform a merge per commit base? and why not automate it? At this point we decided to develop a tool and the result was the following.

In the above image, it is possible to see two developers. Dev 1 is committing in PS and Dev 2 is committing in Trunk. Note that Dev 1 have done two commits (revision 1 and revision 2) which were merged automatically to Trunk by a tool we have developed. Then Dev 2 tried to make a commit but the version control blocked it due to a conflict. So, he must first resolve the conflict in his working copy and then commit his new feature (the revision 3 in the image) in Trunk.

Whenever the team decides this feature (

revision 3) is production ready, the developer which have committed it (in this case Dev 2) must be pushing the revision to PS in order to release it as soon as possible.

A more complex scenario where the automatic merge fail can be seen in the below picture.

In this case, when Dev 1 tries to commit revision 3, our automatic merging tool shows the conflict in RED in a dashboard and stops merging. At this point, the integration team, along with the developer which has performed the commit (in this case, Dev 1), must analyze and fix the problem. After that, the merging tool must be advised to keep doing merges. As we have seen in the first picture, every time a feature is considered production ready in Trunk it must be pushed to PS in order to be released.

Advantages

• Developers have time to finalize its features (in Trunk) without being afraid of putting unfinished code in production
• There are fewer cases which actually require Feature Toggles
• Once they are rarely used to manage unfinished code
• The team which use Feature Toggles are more prepared
• Toggles are usually implemented by the production support team
• The integration team has more free time to improve the whole integration and delivery process
• Less issues resulted from merges
• More control about what is done and what is not
• Done means running in production without problems
• Developers have more responsibility
• Developers must be responsible from coding to monitoring production
• Forces the team to have focus
• New features developed in Trunk must go to production as soon as possible in order to make the automated merge tool work properly

Not so good, but not disadvantages

• New features must be pushed manually from Trunk to PS
• Although we think this task can also be automated, we like the idea of developers being involved/responsible up to production.
• The integration team is still involved to resolve conflicts
• However, the time spent in this task is nothing compared to some years ago
• Conflicts must be resolved as soon as possible
• In order to not accumulate automatic merges and keep the whole system working seamless
• Branches must be as similar as possible
• In order to avoid automatic merges conflicts. So, features must be pushed from Trunk to PS as soon as possible.

This solution is way better than others we have tried up until now. It works in real world, developer is involved from the beginning until the end, the number of Feature Toggles is reduced, the integration team has more free time to improve the process and it forces the whole team to have focus on the current development features (soon I'll be writing more about what other strategies we have used to improve focus).

Are you doing or trying to do Continuous Delivery? How do you work? What do you think of this solution? Do you like it? Do you think it may work for you? Please, leave your comments. I'll strongly appreciate them.

Feature toggles: good or bad?

Everyone who have read at least once about continuous integration or continuous delivery knows that it is not possible or,  at least it is very difficult, to do if you use more than one development branches. If you have never ever faced any problems using two or more development branches you are a fucking lucky guy or, probably, you are not used to refactor your code as you should be (in these rare cases, I strongly recommend you to take a look in this Martin Fowler & Mike Mason interview and Jez Humber Continuous Delivery book). On the other hand, if you have found yourself in an endless merging task or if you have been feeling insecure about some merging results. Do not panic! Keep reading. You are not alone. :-)

As you may already know, there is no problem with branches. The problems are all about merges. So, to overcome big and error prone merges the best strategy I would recommend is Branch by Abstraction. According to Martin Fowler, Branch by Abstraction is “a term coined by Paul Hammant to describe a technique for making structural changes to a code-base without a Feature Branch in a source-code control system”. To make that real, we are used to implement new features, or updating existing ones, using a very simple technique known as Feature Toggles. In short, a feature toggle is a way you develop your software in order to make it easy to enable or disable features through configuration changes.

This kind of strategy is widely used by companies such as Facebook, Flicker, Google, Amazon, and, of course, the ones I’ve been consulting. A very simple example of feature toggles can be seeing in the Chrome web browser by typing chrome://flags in the URL input. In this page you can enable or disable features of your browse and help Google to evolve it. ;-)

After some years of experience trying to improve clients’ delivery processes, we have found some advantages and disadvantages of using toggles instead of version control system branches. So, the main idea of this post is to share and discuss our experience with the community.

The advantages we have found are:

• Ability to turn features On and Off at any moment
• If you have found yourself adding a new bug in production, for example, you could just turn your feature Off.
• Move forward, never backward
• Once you can disable incomplete or buggy features, you can move on and fix it instead of rolling back your entire app (which in some cases is extremely painful).
• Ability to put unfinished code in production
• At the first glance it may sound a bit weird, but it helps you to reduce the amount of stuff you put in production at once. Small batches reduce the risk of going live (see Reducing IT risk using continuous delivery )
• Reduce or eliminate the necessity to create branches in a source-code control repository
• Your integration team will be very thankful, believe me ;-). This means less or no merges any more. Given them more time to making smarter tasks than merging.
• Ability to make experiments that could not be done outside of production environment
• For example, those intermittent issues which are impossible to reproduce outside of production.
• Ability to enable a feature for only one group of users
• For example, you can enable a feature for a specific region, role, user group, or any other rule you need.
• Ability to enable and disable the features while the app is up and running
• But you may set toggles at build time

The disadvantages are:

• There are cases in which it is not possible or, at least,  it is very difficult to implement a feature toggle
• Examples in the Java world
• Static configuration files (i.e. web.xml, ejb-jar.xml e persistence.xml),
• JPA mapping,
• etc.
• Every toggle requires an engineering process
• First, avoid doing toggles for everything.
• Second, you must be sure you need it.
• Third, if you realize it is extremely necessary, it is important to provide a good design for its implementation.
• Toggle On means: a pretty new technical debt
• Once you enable your toggle and it is working 100% in production you have a technical debt that should be removed as soon as possible
• If you do not remove the technical debt your software will be harder to improve, maintain, test and deploy. This happens because your code must care about all possible situations (i.e. toggle On and Off)
• Features loosing priorities
• Sometimes people forget to enable a feature toggle just because that feature loses its priority. It is very ease to happen, unfortunately. At least in my experience.
• Sometimes people find an issue in production and then disable the toggle, but never fix it. In this case, the toggle remains there disabled forever.
• Feature toggle could easily get out of control
• Sometimes people lose the control of what is enable in which environment
• The number of toggles will grow faster than you think. Take care.

Finally, the discussion about this subject

Any technique has its advantages and disadvantages. However, if you have noticed, most of disadvantages are a reflect of not given the right control feature toggles should have. With a good toggle management, frequent releases and focus (do one thing at a time) it is possible to solve most of these issues.

To help overcome these difficulties, we have been fulfilling the following information for every feature toggle:

1. The Value
1. Usually enable/disable, but can be a complex data structure informing the region it is enabled, the user grup, or etc
2. A detailed description
1. Detailed information about what this feature toggle does and why it is necessary
3. The deadline (very important)
1. When this toggle should die and how you should perform the clean up
4. The responsible (very important)
1. The engineer/team responsible to monitor and remove the toggle. Usually, this responsibility is given to the one which have chosen to add it.

Besides that we have developed a dashboard, which is revised periodically, which provides all toggles information in order to get the full benefits this technique could provide us. Moreover, as Martin Fowler already said “while feature toggles are a valuable tool in the box, they are a second-best option. The best thing to do with such features is to find a way to gradually release them into production as you are building them”.

If you like or not, please leave your comment. I will be very happy to listen to other teams experiences.