quarta-feira, 17 de abril de 2013

Is it possible to do Continuous Delivery using branches? We think so!

In previous posts I’ve been written about why you should be doing Continuous Delivery and how to avoid branches using a very simple technique called Branch By Abstraction. As we have already discussed in these articles, there is nothing wrong with branches, the problem is all about merges. So, adopting toggles may help us to have only a unique development branch. But there are some difficulties around this adoption. After a pair of years trying to find an alternative for a client we have reached a solution which allowed us to do Continuous Delivery combining both branches, Feature Toggles and a bit of automation. The idea of this post is to expose our experience in order to allow other people to experiment and suggest improvements on this solution.

The problem

Working with branches has been shown much more complex than it should be. In short I would say: the bigger is your team, the more complex is your merge. The biggest the time between merges, the more conflicts will appear. Have done a refactoring?!? Merging will be a chaos.

Merges take us too much time and are an error prone alternative, mainly when we are talking about large development teams. So, why not using a unique development branch? Adopting an unique development branch knowing that the entire code may go to production at any moment may be an easy task for companies such as Google, Facebook, Amazon, Flicker, etc (of course, they have the best software engineering the money can pay). But, in real world (a.k.a. enterprise world) it seem to be much more complex. Why? In my experience of years developing software for enterprises it is easy to notice that, generally, the teams are formed by 1/5 experts, 2/5 developers, 2/5 junior programmers or trainees. In addition, enterprises tend to add a lot of people in their projects in order to reach "high productivity" (what compromise control and quality, of course).

So, it is extremely hard to put this kind of team working on a unique branch and request them to produce production ready code.

The solution

Although we think it would be ideal to have a team of experts using Branch By Abstraction technique in a unique development branch, we know that, in real world, it hardly will be applicable. So, we have been trying to mix branches, toggles and a bit of automation in order to find a solution which fits our needs. For the happiness of our team we have found a very simple and interesting alternative. So, let me explain it and how the idea came out.

We were used to have two development branches. The Trunk, for major developments and PS for Production Support. While developers were committing new features in Trunk, the production support team were fixing bugs and implementing small improvements in PS. Initially we were used to merge PS to Trunk as soon as a new version was released (usually one merge per week basis) and, from Trunk to PS whenever business requested. Frequently, those merges took the integration team one or more days to finalize. Sometimes, bugs were introduced and some of them would not be seen until a new release was generated from Trunk (what could take months or even years). Knowing this, we have decided to use a basic premise of continuous delivery: "if it hurts, do it more often".  Thinking like that we have decided to merge once a day, but we still have problems. So, we have tried to merge twice a day. Of course it got better, but nobody likes merges and do it twice a day is a pain in the ass. Then, we question ourselves: why not perform a merge per commit base? and why not automate it? At this point we decided to develop a tool and the result was the following.

In the above image, it is possible to see two developers. Dev 1 is committing in PS and Dev 2 is committing in Trunk. Note that Dev 1 have done two commits (revision 1 and revision 2) which were merged automatically to Trunk by a tool we have developed. Then Dev 2 tried to make a commit but the version control blocked it due to a conflict. So, he must first resolve the conflict in his working copy and then commit his new feature (the revision 3 in the image) in Trunk.
Whenever the team decides this feature (
revision 3) is production ready, the developer which have committed it (in this case Dev 2) must be pushing the revision to PS in order to release it as soon as possible.

A more complex scenario where the automatic merge fail can be seen in the below picture.

In this case, when Dev 1 tries to commit revision 3, our automatic merging tool shows the conflict in RED in a dashboard and stops merging. At this point, the integration team, along with the developer which has performed the commit (in this case, Dev 1), must analyze and fix the problem. After that, the merging tool must be advised to keep doing merges. As we have seen in the first picture, every time a feature is considered production ready in Trunk it must be pushed to PS in order to be released.


  • Developers have time to finalize its features (in Trunk) without being afraid of putting unfinished code in production
  • There are fewer cases which actually require Feature Toggles
    • Once they are rarely used to manage unfinished code
  • The team which use Feature Toggles are more prepared
    • Toggles are usually implemented by the production support team
  • The integration team has more free time to improve the whole integration and delivery process
  • Less issues resulted from merges
  • More control about what is done and what is not
    • Done means running in production without problems
  • Developers have more responsibility
    • Developers must be responsible from coding to monitoring production
  • Forces the team to have focus
    • New features developed in Trunk must go to production as soon as possible in order to make the automated merge tool work properly

Not so good, but not disadvantages

  • New features must be pushed manually from Trunk to PS
    • Although we think this task can also be automated, we like the idea of developers being involved/responsible up to production.
  • The integration team is still involved to resolve conflicts
    • However, the time spent in this task is nothing compared to some years ago
  • Conflicts must be resolved as soon as possible
    • In order to not accumulate automatic merges and keep the whole system working seamless
  • Branches must be as similar as possible
    • In order to avoid automatic merges conflicts. So, features must be pushed from Trunk to PS as soon as possible.

This solution is way better than others we have tried up until now. It works in real world, developer is involved from the beginning until the end, the number of Feature Toggles is reduced, the integration team has more free time to improve the process and it forces the whole team to have focus on the current development features (soon I'll be writing more about what other strategies we have used to improve focus).

Are you doing or trying to do Continuous Delivery? How do you work? What do you think of this solution? Do you like it? Do you think it may work for you? Please, leave your comments. I'll strongly appreciate them.

domingo, 7 de abril de 2013

Jboss 7: a love–hate relationship

I've been using JBoss Application Server professionally since 2002. During this period, I've also used Tomcat, Jetty, Glassfish, IBM WebSphere and the already dead OC4J. Between these options I would say my personal choice always was JBoss. Do not misunderstand me; Tomcat and Jetty are both great and lightweight web containers. But the point of this post is not to discussion anyone's preferences, but my experiences with the new version of JBoss Application Server (AS).

First of all, I would like to discuss the good points and improvements JBoss 7 has over its older versions. So, my "loves" are:
  • Strict Java EE 6 compliant
    • Reduce vendor locked once JBoss development team has chosen to implement “only” the JavaEE spec.
  • Completely redesigned architecture (very welcome)
    • Modular design
      • Say goodbye to microcontainers
      • They have created an abstraction built on top of OSGI
      • According to JBoss guys, it is “prepared to support Jigsaw”
    • Modular dependencies
      • Best way to manage your app dependencies
    • Modular classloading
      • Say goodbye to the UnifiedClassloader. \o/
      • classloading happens on demand
  • Two execution modes
    • standalone
      • Normally used for development
      • Allows to organize and manage your cluster(s) as you are already used to
    • domain
      • Allows to organize and manage your cluster(s) in an easy and centralized way
  • Better security support
    • Allows to provide separate inet interfaces for apps and management consoles
  • Central point of configuration
    • the whole configuration is centralized in only one file (or two for domain mode)
    • Profile definition are easier and “smarter”
      • It simplifies a lot the AS slimming. \o/
      • If you do not use a specific feature, JBoss do not load it
      • You can reuse profile definitions. There is no need to be coping jars around anymore. \o/
  • Better management solutions
    • Simple and easy management interfaces
      • Web Console (GWT) with only the most important data
      • Rest interface
      • Command line
      • JMX (actually I’ve never used this, there is no need any more)
    • A centralized management interface for domain mode
    • Persistent configuration through any management interface available
      • Remember... In older versions of JBoss, the configuration through jmx-console was transient. So, this is a very welcome feature
  • Faster startup
    • Modules were optimized to minimize the startup time
    • Modules and packages are indexed
    • Only required modules are loaded
      • JBoss knows your dependency graph, so there is no reason to load JMS module, for example, if you do not use it. Even though you have configured it to be loaded. Actually, it seems to work pretty well.
    • Parallel startup
      • Once JBoss 7 knows your dependency graph, it can use all cores to start your apps
      • To resolve correctly your dependencies in older versions of JBoss, you have to inform the order you apps must start. So there was no way to start them parallelly
  • Memory diet
    • As we have seen before, only required modules are loaded
      • This means, less memory usage
      • Less memory usage == better scalability and performance, once you will have more free memory available and less Full GC executions during runtime
  • Better support for integration tests through Arquillian

Besides all these good stuff, let me show my "hates":
  • Strict Java EE 6 compliant
    • Wait.. You should be thinking: “at first you said it was a good stuff, and now a bad stuff?!?!”. The reason for this is that there are functionalities which are impossible to implement using pure Java EE spec and, most of them, were removed. :-(.  Some of them, community yell so much they have decided to add an alternative. Take as an example Container Interceptors which is coming in JBoss AS 7.2 / JBoss EAP 6.1
  • Completely redesigned architecture
    • Again.. the new design modular is wonderful, but for those who would like to migrate from an older versions of JBoss, will notice it is not as easy and seamless as it should be
  • No backward compatible in JBoss specific features
    • JBoss 7 has broken compatibility with everything that was not in JavaEE spec. So if you use JBoss specific features, be careful, your migration may be more complex than you think it would be
  • No JBoss AOP any more :-(
  • The migration documents are useless
    • The difficulties we have faced do not fit in this post, so I’m planning to write one or more posts to help community understand what they will be facing during the JBoss migration
  • Annoying bugs deploying applications through management consoles
    • Although I think the fix would be coming, it has been forcing us to restart the server at each deploy
  • JBoss EAP Alpha
    • The community version of JBoss (i.e. JBoss AS) will now be called as JBoss EAP Alpha. Why?!? Marketing purpose, of course. Imagine you saying to a client they should be using an Alpha version in production
      UPDATED: Just received an e-mail from Red Hat saing
      • JBoss AS, is now called Wildfly. See by yourself
  • JBoss EAP Alpha is not as stable as the JBoss AS Final
    • Although they say EAP Alpha is stable as the AS Final, it is not complete true. If you take a quick look on JBoss modules, you will easily notice they are using unfinished/unstable libraries.
      • For example, JBoss EAP 6 Alpha is bundled with hibernate-core-4.2.0.CR1. I don’t recall any CR lib in older versions of JBoss AS.
      • Obviously it is very ease to people with certain knowledge to modify. But most of people will be using JBoss without any module structure change, for sure.
  • Domain controller is not fault tolerant
    • Whenever the domain controller dies, other nodes cannot be restarted or managed. If you try to restart them, they will give you an error saying they cannot connect to domain controller. Hope the fix would be coming.
  • Does not provide load balancing for EJB Timer
    • Only fault tolerance
    • In some clients, we were forced to keep using quartz because the huge number of jobs they have
  • Last, but not least. JBoss EAP support is a pain in the ass
    • I will make no comments about this. Just try by yourself you will understand what I’m talking about.
    • My advice is: be prepared, mainly if you have gastritis. :-)

Even with all these issues, I still think JBoss 7 / EAP 6 is one of the best application server available, but I think the move Red Hat is taking would be pushing the community away from them. Actually, I know people which are looking for Jboss alternatives. Some friends are working in a migration from JBoss 5.1 to TomEE, another once told me: “if I have to pay for an Application Server, I will be paying for a WebLogic”. He said that knowing Oracle contracts were more expensive than Red Hat ones.

Anyway, everybody knows that there are many good Application Server options out there (both free and paid ones). So, what do you think? Would you be using the JBoss EAP Alpha in a production environment? Please, leave your comment. I would love to hear it.