Continuous integration

This article is about continuous integration: the principle that whatever software you are building should build and run all the time.

Sounds easy, doesn’t it? In practice, a lot of projects get this wrong straight away. Everyone goes off to their desks and gets right on with coding their own thing. Six months later they have created an almighty mess because they’ve left it for six months to try to fit the bits together. It’s like everyone going off by themselves to make one piece in a jigsaw puzzle. By the time all the bits come together, there’s no way they’re going to fit without some serious work!

Continuous integration is one of the ways we minimize the risk of things going wrong.

One of the things that can go wrong is that someone can write some code that doesn’t fit into the rest of the system. Maybe it doesn’t pass all its tests, or it doesn’t use the same interfaces the rest of the system does. Even worse, maybe it doesn’t even compile. Actually, that can happen easily if the rest of the system is changing all the time, which is what happens whether you’re doing XP or not.

Sticking with the XP philosophy of keeping it simple, one way to approach this is to keep integrating every small thing you do with the rest of the system. If everyone is doing that, and provided everyone is releasing their code frequently, nothing can get so far out of sync that integration is difficult.

Continuous integration is the principle that the system can be compiled, built, tested and deployed at all times. Of course, there may be stages during an iteration when not all of the stories have been completed; nevertheless, what is there should always be ready to use immediately.

There are several elements to continuous integration, but they all revolve around the use of a revision control system and a tool that can compare the data (usually files) on a development computer with the baseline held in the revision control system. This difference tool may be separate or may be part of the revision control system.

The baseline

The baseline is the place where everyone’s code meets - the fully-integrated combination of code, tests, build scripts, configuration files, deployment scripts and anything else you need to create, deploy and test the whole system. The baseline is the definitive and only current version of the complete system.

Active coding doesn’t take place on the baseline itself - having everyone testing and coding away on the same files simultaneously would quickly lead to an uncontrollable mess. Instead, everyone who wants to work on the code takes a working copy, works on it, and then merges the changes they made back into the baseline again. Integration is the process of getting your changes to code, tests, etc. included in the baseline. Until your work reaches the baseline and all the tests in the baseline pass your work isn’t finished.

In XP, the baseline is incredibly important. It should build a complete working system all the time, that reflects all the programming work that has gone into the project, even work in the current iteration. If you can’t take a working copy of the baseline, run all the tests and allow your customer to sit down and use it at any time, you haven’t got a system. What the customer sees is a load of meaningless data. It doesn’t translate into business value until it’s able to build and run in a live environment.

For you, as a developer, the advantage of continuous integration is that problems get caught early, and you always have a working baseline to go back to when you’ve really gone off the rails. How many times have you coded all morning, only to realize that you’re going in the wrong direction, and then spent the whole afternoon trying to get back to where you started? This advantage of revision control systems is so important that we’d recommend you use one even if there are only two people in your XP team.

Revision control systems

The only really good way of storing the baseline and allowing people to take working copies and merge them into the baseline is to use a revision control system. These often have a variety of names. Sometimes they’re called version control systems or configuration management systems. They may be able to perform other functions as well. Not all of these extra functions are useful in XP, and some of them actually get in the way. We’ll look at choosing a revision control system later in this chapter.

Revision control systems perform a number of functions. If you’re not used to these, it can seem a bit confusing and even like a lot of overhead at first, but a decent revision control system is essential to team-based software development. Duncan uses one for practically every file he ever produces, whether he’s working in a team or not.

The first and most important function of a revision control system is to provide a place to store data. Specifically, that data is usually text files containing program source code (your code and tests), configuration files, and build and deployment scripts.

Strangely, many revision control systems don’t store the baseline in a usable form. That is, you can’t just go and compile from the baseline itself - you have to take a working copy. This is actually a good thing, because it makes it less likely that some disaster will befall the revision control system’s master copy of the baseline. If that goes, you’re in the nasty position of not having any working baseline. The reason, though, is that the revision control system needs to keep extra data about changes to the files, so they often end up containing a series of differences, rather than the complete file.

So revision control systems also allow you to take a working copy of the baseline, which is what you will edit, compile and deploy. In XP, that working copy will include files that contain source code, tests, deployment scripts and everything that the baseline should contain. Taking a working copy of the baseline is sometimes called “checking out”. We’ll actually be calling it “catching up” or “loading the baseline” because taking the copy is not the important part; it’s catching up with the on-going development process.

The final major function is that you need to be able to move your working copy of the baseline back into the baseline stored in the revision control system, otherwise you wouldn’t be able to get any data into it! This is sometimes called “checking in”, but we’ll call it “releasing” because by doing it you’re making your working copy available to the whole team.

Catching up and releasing turn out to be a bit tricky because everyone is working away on the same tests and code at the same time. We’ll look at this issue in a bit more detail in a moment.

When you’re choosing tools look for ones that can work without enforced file locking and don’t restrict you to a single check-out out a time (or treat the first check-out as special in some way). Finally avoid systems that have no support for integrating two copies of a file from different sources. You should be able to see why that’s so when we come to the actual process of integration a little later in this chapter.

Branches and the trunk

If you develop software using a revision control system, your changes accumulate over time into what we’ll call a stream. A stream is really just a sequence of snapshots of a system’s development over time. Every time you commit some changes to the system, your new system gets added as the latest revision in the stream.

Many revision control systems allow for “branches”. A branch is where the development stream divides. It’s a bit like taking a working copy but storing it in the revision control system as though it were an alternative baseline - which it what it is used for. You can then take working copies of that branch, work on them and commit them to the branch again. All the development streams together end up looking like a tree.

Often, you won’t need to worry too much about branches, but it is a good idea to know how to use them and what they do when the need arises. We’ll look at where you might want to use branches later in the chapter.

The main branch where development work normally takes place is called the trunk. The most current revision on the trunk is often called the head revision or just head. Usually there’s nothing particularly special about the trunk except that it’s the default branch. In other words, if you don’t specify some other branch, you get the trunk. That’s good because this is the place where development normally occurs. Similarly, if you don’t specify any revision, you get the head (latest) revision, which is also what you normally want.

The process of integration

Actually integrating your work with the baseline can be a tricky business. Good tools can save you from nightmarish problems here. Bad tools will promote mistakes. We’ll go through the two most important steps of the process of integrating, and then we’ll look at tool selection.

Catching up

If you started by loading the baseline before working on your story and you’re lucky, no-one else will have released any code to the baseline by the time you finish. If that happens, you can just go right ahead and release your code, because it is based on the current revision of the baseline.

The problems start when another pair has released code while you’ve been working. If the team contains more than a couple of pairs of developers, most of the time you will not be lucky enough to get your changes back into the baseline before someone else has changed it.

Let’s assume you and another pair of developers both started from the same baseline. It doesn’t matter actually, but it makes it easier to understand what’s going on.

Let’s imagine the other pair release their stuff first, so their new code, plus the old baseline is now the new baseline. That means your code is not based on the current baseline - it’s based on an old one. If you just release your code straight away, you will be undoing their work. If everyone was doing this, the team would never get anywhere!

Before you can release, then, you need to catch up with what the other guys have done. In other words, you need to catch up with all the changes they made after you loaded the original baseline. If you do that, your working code will contain:

  • the code from the original baseline you loaded (which is now out of date)
  • all your new changes that you want to release
  • all the changes made by team A that have happened since you loaded your old copy of the baseline.

Together, all these things make up the new baseline, which is going to consist of your working copy of the baseline, plus everyone else’s changes merged in.

Releasing

If either your new code is based on the baseline (no-one else released anything since you started work), or you’ve caught up with the baseline, you are ready to release your code. Actually, both these requirements are really the same thing: your working code needs to be based on the current baseline in order for you to be able to release it.

So if, after you catch up but before you release someone else releases some code, you can’t release your code! It’s not derived from the current baseline any more because of that extra release. The best thing you can do in this situation is sigh, shake your head and start catching up again. Fortunately, you can take steps to make sure it doesn’t happen that often.

One thing you can do to prevent this is to make the time you spend catching up shorter. There are two ways to do that:

  1. Catch up faster.
  2. Catch up with less changes.

There’s a limit to how fast you can catch up without making it likely you’ll make a lot of mistakes, but good tools really help out here. The best area for improvement is to catch up with less changes. The best way of doing that is to encourage everyone in the team to keep their releases as small as they can.

If you keep catching up with each other on a regular basis, catching up should be easy. A good rule of thumb here is that if you do a whole day’s work without releasing it, really start looking for a way to get your code released, because you’re going too long on it. In practice, it’s better to release far more often than that.

There are a few good reasons why releasing code often helps:

  1. If you release often you catch up often, so regardless of what everyone else is doing you should have an easier time of it because you’re catching up with less.
  2. If you release often, the rest of the team will have an easier time of it, because they will be able to catch up with you in more but smaller steps. So if everyone is also releasing their code often you will get that benefit as well, but only if you catch up often yourself.
  3. Always thinking about getting that next release into the baseline helps you to break your work down into smaller chunks of implementation. That helps you to get the implementation right, by modularising the steps you’ll take. It also helps to reduce the risk of going off on an ill-thought-out tangent and losing all the work you’ve done.

Locking instead of catching up

One idea that might have occurred to you straight away is that by locking the files you want to work on, you can avoid this whole catching up problem. Unfortunately, that causes some problems of its own, and it doesn’t even solve the whole integration problem.

The big problem locking the files you need to work on causes is the team getting deadlocked. As we mentioned before it often seems to be the case that different pairs of developers need to work on related stories, and those stories often touch upon the same parts of the system. If pairs were to lock the files they worked on, there would be no way for those stories to be done simultaneously. This could slow down development a lot.

Worse, it’s usually hard to predict up front all the parts of the system you’ll need to change, so you would need to lock files as you go. What happens when one pair has file A locked and wants to start working on file B, and another has file B locked and wants to start working on A?

The problem here is that neither can progress without giving one up their lock and undoing all the work they’ve done so far. If this happened often, development would slow down to a snail’s pace because only one pair would actually be able to make progress. If that happens, the simplest thing you can do is just have two people working and forget the locking!

What we’re really using in XP is really an optimistic form of concurrency - where we hope that changes won’t conflict and be prepared to spend some time fixing up the cases where they do - instead of pessimistic concurrency where we completely prevent the possibility of conflicts, but at much greater average cost.

Making small releases

We’ve already mentioned some good reasons for wanting to make releases as small as you can (within reason). Let’s look at how you might do that.

The first and simplest bit of advice we can give is to sit down at the start of a task and take a couple of minutes to think through when you are going to release. For very small tasks, you may be able to release only when the task is complete. That’s ok, because in that case the task is very small, and the integration effort will also be small.

You might try to create small, single-release tasks during estimation. Whether you find it easy to do that often is largely a consequence of how you go about breaking down stories into tasks, and the granularity that results from doing that. We don’t think you should worry too much about doing this - it can be too much problem-solving up front. If you go too far with this it can be time-consuming and so much can change before the

Of course, during the planning game, if you can see a really good way to break a story down into single-release tasks, go ahead and do it. This will often be a good structure for working on the story. It’s just that we don’t think you should get hung up on doing this all the time. Sometimes you will need to be closer to the task to see the best way of doing it.

It gets a bit more challenging when the task is a little larger. A common case is where you discover the need for refactoring as you are going along. These can easily get out of hand if the refactoring is interspersed with coding. It’s a good idea in this situation to divide the task into coding releases and refactoring releases, and only do one thing at a time. Of course, that means that you sometimes have to code something in a system that isn’t primed to accept it, but it is usually better to get it coded and then clean it up. Make sure you don’t get tempted to stop until the refactoring is done, though. You’ll have to do it eventually anyway, and it’s easier to do it now while you still understand what’s going on than in a month when all you’ll see is a confusing mess of code.

Once you get used to making small releases it is usually easy for a pair of developers to figure out how to go about making small releases. The skill is mainly in setting yourself little milestones, and trying not to code outside the milestones. Each milestone makes a release.

  • keeps you focussed
  • stops a big build up of unreleased code
  • helps prevent “divergent” code, where everyone goes off in all different directions and has difficulty getting their code back together again.

Choosing tools

Catching-up and releasing can be some of the most confusing and error-prone stages of the development process. This is where your code and tests meet everyone else’s. The confusion arises because everyone is working simultaneously on the same code.

If you’re lucky, you will all be working in separate parts of the system. Tests rarely seem to overlap much, but it often seems to be the case that several pairs of developers are working in the same area of the system at the same time. It happens because there are often common aspects to the stories that the team gets asked to do during an iteration.

There are several aspects to choosing tools you might be interested in:

  • how well the revision control system integrates with your development environment
  • how well the catch-up tool integrates with your development environment
  • whether the revision control system protects you against accidental reversions of the baseline
  • whether the catch-up tool shows 3-way or only 2-way differences
  • whether revisions of the baseline are limited to individual components like files, or whether the revision control system understands the idea of revisions of the whole baseline
  • whether the revision control system uses the names of files to identify things. So, for example, if you rename Fred.java to Freda.java (the Sex Change refactoring, which is not documented by Martin Fowler’s book), does the revision control system get confused and think that Fred.java has been deleted and a new Freda.java file (revision 1) has been created? Most do, and it can be confusing.

2-way and 3-way comparison

When you’re comparing your work with the baseline, there are two common ways of doing it: 2-way and 3-way comparison. 2-way comparison compares your file with the file that’s currently in the baseline (usually). 3-way comparison compares your file and the current file in the baseline with their closest common ancestor. Sounds a lot more complicated, doesn’t it?

In practice, 3-way comparison can give you much better information when you’re trying to catch up. It doesn’t only know where part of your revision differs from the baseline, it also knows whether you’ve just changed it, or the baseline has moved on since you loaded it, or both.

As an example of this, imagine we took a working copy of the baseline when it looked like this:

Here’s the common ancestor. It contains three lines. Here is the third.

And now imagine that while we were working, someone else updated it to this:

Here’s the revision from the baseline. It’s been changed. It contains three lines. Here is the third.

Now let’s say we edit our file. We’ll add another line.

Here’s the common ancestor. It contains three lines. Here is the third. Here is the fourth line.

If we compare this with the baseline using 2-way comparison, it will tell us that two lines are different (shown in bold):

Here’s the common ancestor. It contains three lines. Here is the third. Here is the fourth line.

If we compare using 3-way comparison, it will tell us (bold) whether we changed something or whether (italic) the baseline changed under us.

Here’s the common ancestor. It contains three lines. Here is the third. Here is the fourth line.

In this case, we have a bit more information about has happened. This makes it a little bit easier to realize that we need to catch up with changes to the line “Here’s the common ancestor.”. Otherwise, it might be tempting to think that we’ve changed it ourselves and now it needs to be released. This kind of retrograde release error is common with 2-way comparison tools. It’s “retrograde” because it actually undoes useful changes that have been made in the baseline since you took your working copy.

In fact, many systems which understand 3-way comparison will automatically catch up with simple changes like this for you. So you’d see something more like this:

Here’s the revision from the baseline. It’s been changed. It contains three lines. Here is the third. Here is the fourth line.

Here, the tool is trying to tell you that it changed the line to catch you up with the baseline. You probably only need to worry about the line in bold, which is the bit you really changed.

With careful use of a 2-way tool you can usually be safe from errors. Just be aware that when you see a change it might be you two that made it, or it might be that the baseline changed while you were working.

There’s one nasty case where you have to be very careful. When you change your working copy, and the baseline changes, you get a conflict. No 2-way tool can spot this. They always see it as a change, and you will often assume that you changed it (which you did) without realizing that someone else also changed it! So you go right ahead and release it. This will often be a retrograde release, undoing the other changes.

Here’s how a 3-way tool views it:

Conflict! You have: Here’s a new change I made to create a conflict. Baseline has: Here’s the revision from the baseline. It’s been changed. It contains three lines. Here is the third.

The best a 2-way tool will do is indicate there was a change on that line, which you know anyway. That’s what tricks you into doing the retrograde release.

Revision control is a kind of database

Your revision control system is a kind of database that’s specialized for software development work, and the baseline that’s stored in it is a special kind of data.

It turns out that the data stored in your revision control system is extraordinarily expensive. It all depends on whose figures you believe, but we’ve heard it said that the average programmer is producing only about 10 lines of working code a day - we’d like to think you can do better than that with XP, but we’ll take this as a starting point.

So how long is a line of code? Maybe 20 characters on average? That’s 200 bytes a day, which is 1000 bytes a week. The average wage for a developer is probably in excess of 25000 pounds Sterling a year, but most companies reckon it costs twice someone’s salary to employ them, so that’s about 1000 pounds a week. Or 1 pound per byte. Or 1 million pounds per megabyte. So it’s expensive stuff, this software. Usually, the business value of the software - or more accurately, the cost of losing it - is much higher.

Ok, those were back-of-an-envelope calculations, but given that, it’s a good idea to treat your code (specifically, your baseline) as though it were expensive data. You should:

  • back it up
  • make sure you can restore it and that the restoration and back up processes work
  • commit (release) good quality data (which is where testing and refactoring come in)
  • commit data as soon as possible so it’s protected against loss (which is where releasing often comes in) and not leaving it on the old PC in the corner which you also use as a final resting place for cups of cold coffee.

It also stands to reason that you would use a revision control system that you can rely on. It should be reliable, and it should be easy to use, so you don’t make mistakes.

Work out a release procedure

This is particularly important if your revision control system offers you few built-in protections against mistakes. Of course, the real answer here might be to switch to a better one, but there are other issues like integration with your development environment that might override such considerations.

It’s good to have a release procedure anyway. Sometimes, when the pressure is on (you’re getting close to the end of the iteration and this is the last story, or there’s a bug in the live system) it’s easy to make the most horrendous mistakes and completely break the baseline, just when you need everything to work properly.

Particularly if you (as a team) aren’t that experienced at maintaining a baseline, it’s a good idea to start with a simple release procedure and keep looking at its effectiveness. We’ve experienced a revision control system so difficult to use it took several changes to the release procedure to give a reasonably reliable release process. So be prepared to evolve the procedure as you’re going along.

Your release procedure should probably include at least the following:

  1. Make sure all the tests pass. Often when you’re coding you work against just a small part of the full test suite. Before you release, you need to make sure what you’ve been working on doesn’t break anything else in the system. This is true even if you’re absolutely, positively certain it can’t possibly break anything else, because these are the times at which your code actually does break something else.
  2. Catch up with all changes to the baseline.
  3. Make sure all the tests still run. This sanity checks the integration between your code and the rest of the system. What you’re about to release will become the new baseline. Don’t be tempted to skip the first of these test runs. The earlier you discover failing tests, the easier it is to put right. Don’t skip this step, either. Releasing a broken baseline means that no-one else can release safely until it’s fixed.
  4. (If you use a separate release machine…) Move to the release machine and load in your working copy. On some revision control systems it’s easier to load the baseline and then load each file you’ve changed. Run the tests again to make sure what you’ve loaded makes sense.
  5. Release your working copy. This has become the new baseline.
  6. For revision control systems that work by file revision numbers instead of comparing them, you may need to load the baseline on your development machine to catch up with your own changes. Yes, this is stupid - complain to your software vendor about it or vote with your feet and get a different package.

In fact, writing error-prone procedures down in this way is a good idea. Of course, if it could be done completely automatically itÕs better to automate it. With release procedures, there will be some steps that require human judgement and so can’t be automated.

Using a separate release machine

You might find that you need to allocate a separate machine to do releases. Some people find that simply walking over to the machine helps them think more clearly about whether what they’re about to release is really of good enough quality.

Whether that’s true for you or not, some revision control systems make it almost essential that you have a separate release machine because they allow and even encourage you to make silly mistakes. Here are some common problems.

  • The revision control system allows retrograde releases. That is, if you make a mistake catching up and miss a file, it will let you re-release the out-of-date file. For example, suppose you load a baseline which contains Frog.java 1.1, which you don’t change. Later you catch up when the baseline holds a newer Frog.java 1.2. Let’s say you make a mistake and miss this file, so you’ve still got 1.1. A dumb revision control system will let you release 1.1 (because it’s different from the baseline), even though it’s not a later revision than 1.2! Loading the baseline, then your changed files on a release machine helps prevent this.
  • The revision control system can’t cope with two pairs trying to release simultaneously, and corrupts the baseline. Having one release machine means only one pair can release at a time.
  • The revision control system can’t work without locking files. This can be truly painful. You will probably need to export (not check-out, which grabs a lock) your working files on your development machine to work. Then, to release, check-out (which locks) all the files on the release machine. Copy your working files back over the top of these (which updates them), then check back in, releasing all the locks.

If you have one of these, a separate release machine acts as a lock on the revision control repository that prevents these kinds of problems from occurring. As an alternative, you might find it worth having a “release token” which you have to be holding to release, but we’ve found that people tend to forget this too often. The real solution is a different revision control system.

Best practices

Ultimately, this chapter is all about where your code and tests meets everyone else’s. All the best practices we can suggest are really about not falling over each other all the time.

  1. Load the baseline before starting new tasks. Forgetting is a surprisingly easy way to waste your time with lots of catching up.
  2. We think it’s also worth running all the tests just to make sure the baseline actually works. You don’t want to have to fix the baseline on your machine and get your story done simultaneously. Maybe later on when the whole team has got into the habit of ensuring they never release broken code, you can skip this step. Even now, we usually don’t, we just load run the tests while we’re thinking about what we’re about to start doing.
  3. Catch up often if you want it to be easy. It’s tempting to think that you’ll just have to do the same amount of catching up in total if you leave it until later - which is true - but doing it in smaller steps makes it much easier, faster and less error-prone.
  4. Release often if you want to make catching up easy for everyone else and give yourself better protection against those Friday afternoon epics.
  5. Always catch up before releasing. If you don’t you will be reverting other peoples’ changes, which has been known to lead to public lynching.
  6. Always run all tests after catching up. That way, you can have confidence that your changes and the baseline don’t conflict.
  7. Review changes between catching up and releasing. This often catches things you forgot to do. It also helps ensure you aren’t reverting accidentally, which is quite easy with some revision control systems. Finally, it sanity checks your changes against changes against the new baseline-some of them might not make sense any more.
  8. If your revision control cannot safely overlap releases, allocate a separate release machine. Make sure everyone releases code only on that machine, even if the change they’re making is tiny. The release machine serves as a physical lock to prevent concurrent releases.
  9. Label your baseline each iteration with your iteration number, and check that you can recreate the baseline from a label. This is really useful when you have to go back to the baseline from a previous iteration.
  10. Never release into a broken baseline unless your release is a fix. Doing that just makes the problem of sorting it out more serious. Fixing the broken baseline should be regarded as top priority. It’s even more important than working out who to blame for it!
  11. If you want to rename something (especially files), mention it to the rest of the team first. Unfortunately, many revision control systems handle renaming of files and directories rather dumbly. Often they will treat it as a deletion and an addition. The catch-up tool will see only the deletion, because it doesn’t know the addition is related. Renaming other parts of the code can be a bit confusing for other team members, but that mostly depends on the quality of the comparison tools you’re using to catch up.
  12. Don’t store “derived” files (that is, files that are normally made >from other files) in your revision control system. What you should be storing is the scripts and any source files needed to make them. Ideally, you should have a suite of fully automated build and deploy scripts so that you can continuously build and deploy your continuously integrated code. If you store the derived files and then use those in other parts of your build process, you’re much more likely to make the mistake of not manually rebuilding the derived file, which means the out-of-date one gets used in every build until someone remembers to rebuild and re-release the derived file.

Using branches

We’ve avoided saying much about branches so far because you don’t need to master them in most situations. However, they can be very useful in a couple of situations in XP: doing spikes and working together with other XP teams on the same code.

Using branches for spikes

Sometimes you’ll want to do a spike that’s a completely self-contained thing. It doesn’t touch your normal code in any way. This is common if you’re trying to figure out how a new tool or technology works. These spikes can just be checked in to an area of your revision control system that makes it clear they are just spikes (e.g. an area called “spikes”!)

Occasionally, you’ll want to try spikes that involve the existing code. For example, you may have some whacky experimental refactoring in mind, or want to wire some new technology into your existing codebase. This is where branches can come in handy.

Before checking your new spike code in, create a new branch for it to go on. Once it’s checked in it will exist as a separate parallel stream of development. This makes it easier to bring your spike code back up to date with a newer revision of your codebase, by merging the differences from when it was created to the way the codebase is now.

This can be a bit mind-boggling, so we suggest trying it out in a safe area before you do it for real.

Using branches for multi-team development

If your XP process is so big that you’re forced to have separate teams, there are only a few sensible ways of handling it.

  1. Don’t get that big in the first place. Ok, we know. Not an option for everyone. But it’s worth asking the question: does it really need to be this big?
  2. Divide the codebase among the teams so that each team has its own “area”. That can cause problems when you need to modify something outside your area: either you have to ask the other guys to do it (which costs them time) or you have to “break in” to their area and do it yourself.
  3. Keep every team developing on a branch.

Developing on a branch is like what you do when you take a working copy at the start of working on a task and re-integrate with the baseline every time you complete the task, except that it’s scaled up.

Each team takes a copy (a branch) of the baseline and works on that. Instead of re-integrating when every task is complete, they re-integrate (merge the branches) when every iteration is complete. This can be a lengthy process if the teams work on heavily overlapping code, but it should be quite safe when you’ve got plenty of automated tests to run.

If you’re using this approach it’s probably a good idea to stagger the end dates for each team’s iteration so that you haven’t got 4 teams all trying to merge simultaneously. There’s plenty of work there already without adding to it!

One final note on the use of branches: some revision control systems allow a branch to “end” - effectively preventing any more development work being done on it. You can use this to signal that the reason the branch was created has now gone away, and the end of the branch has been merged back into whatever it was derived from.

Other systems don’t support this. That’s not always a problem, but they can then be prone to the “multiple-merge problem”. This happens when the changes from a branch get merged back into (for example) the trunk. Then some more development takes place on the branch, and another attempt to merge the changes from the branch to the trunk takes place. At this point the revision control tool can get very confused. What it needs to merge is the changes since the last merge, but some (like CVS) try to merge every change on the branch, even though the earlier ones have already been merged.

This doesn’t actually happen very often or even ever for most organizations, but it’s worth thinking about if you’re serious branch users in your team.