Development Cycle

version 0.6a 2005-10-23

This document is a proposal, so awaiting for your comments.

Everyone running source based distros wants bleeding edge: latest stable versions, quickly, without bugs. Let's face it, a quick development cycle will let some (vicious) bugs go through. The only solution is to slow down a bit the cycle, so extensive tests can be done to ensure QA. Of course, we could tend to paranoid way of doing things for stable, à-la-debian, ending in the most prehistoric source-based distro, that's not our aim either. We need a not so long development cycle, so we can ensure quality, without being (too much) outdated. Here is a way I thought of, of course not perfect, and waiting for your clever brains to help it to become perfect :-)

First, you'll have to consider some kind of a three and a half level organization: Release, Incoming, Developpers, and Maintainer, being the half part. Keep in mind too, that "Release" is the base of everything.

note: for each package move (Maintainer to Developpers, Incoming back to Maintainer etc), packagers are warned. A public "repository moves" mailing list would be great too, so people know what happens. Moreover, Automatic posts to this ML could automatically open bugs too.

Maintainer

Please note that Maintainer doesn't mean on developper's box only. It applies because it is his personal branch, hosted on the same server that serves public repositories.

This is simply the local repository of packagers. That's here where new packages are created, and pass their first very basic quality test: sources are found, patches if needed are correctly applied, code compiles on the developper's box, install is ok, man pages, doc files are in the correct directories. dependencies must have been thoroughly checked or tar and feather will gain some popularity :-). Of course, the program must have been run and quite widely used by the packager to ensure quality. Once all these steps have been verified, the package can land in Developers. By the way, with the help of a hook (refer to subversion doc.∞), we could automatically launch a QA program checking several points of the package, rejecting the commit if the package obviously sucks. In the case where it is a package update, the developer must base his work on the latest version of the package in the Release repository.

Developpers

Here we launch tests to check if packages work correctly on every architecture we support. If a package is known to be buggy in Developpers step (more on step freezes and timing below), packager is automatically warned by mail, so he can try to solve the problem before Developpers freezes. Of course, a public ML could (should) archive such mails. When Developpers freezes, if a package is still buggy, it is simply killed. If it fails only for given architectures, flags are correctly put so we prevent people to try to install it on these architectures, and goes to Incoming. Of course, if a newer package doesn't work better than the previous one, it is killed. Everything that works is pushed into Incoming. In every case, packagers receive a sum up of the state of their package list, so they know what they must work on. Bug free packages are more important than new packages. So, a maintainer who has some bugs to squash have to solve them before adding new packages. note about package killing: the corresponding commit is undone (so package is falling back to previous state), and packager has to work on it in his Maintainer repository for the next round.

Incoming

Now, we know each package works in given circumstances. That step is there to ensure every package interacts with each other flawlessly. For example, someone could have updated foo. Another would have updated bar. bar works fine with the Release version of foo, but not the Incoming one. As newer versions are prefered, it's up to the bar maintainer to find patches so it works with the newest version for foo. If it isn't possible, the package causing the most numerous problems will be downgraded, until a cleaner solution is found. Once everything works together flawlessly, Incoming is frozen, and working packages are merged into Release. The others are killed, passed back to their respective maintainers. Developpers is reopened.

Release

Everything works. :-) Unfortunately, bugs can still be found, and are to be solved in next Maintainer/Developpers/Incoming/Release round. Now, packagers base their updates on this repository. Only security and important fixes will come in. Every other changes are to use the path Maintainer->Developpers->Incoming->Release.

Steps, freezes, and timing.

1st step

Every maintainer uses his own repository, and start everything from scratch (if the package doesn't exist), or Release repository. Developpers is open for a given duration, so they can push their packages that successfully passed QA insurance tests. Incoming and Release do not change. Bugs open for Release repository are the FIRST to be worked on.

2nd step

Developpers is frozen for additions. Everything is compiled, tested, bugs reported.

3rd step

First round of testing is finished. Packagers can commit fixes for a given duration. Then, second round of testing starts. Things that still break are rolled back to packagers, everything else is pushed to Incoming. Developpers is closed, and won't ever change again. Next Developpers will be another branch, based on the next Release.

4rd step

Incoming is under fire to check if everything works fine together. Things that do not cohabit properly must be resolved by the following way: patching, and, worst case, downgrading if no better solution can be found. Bugs are reported.

5th step

A bunch of fixes can come in for a given duration. Extensive testing comes back. Things that still fail are rolled back to packagers, things that work are merged into Release. Incoming is closed. Same as Developpers, it won't change anymore.

6th step

Release is closed. Go to step 1. Contrary to Developpers and Incoming, Release is the only ever existing repository, as everything is based on it. Even if Release is considered as closed, really important and security fixes will come in before the next development cycle finishes.

Here is a quick and (really) dirty drawing, some says drawings are easier to understand than words :)

http://abtlinux.org/Wikka/images/devcycle.png

timing

Fixing time for each steps is quite difficult, and will probably be adjusted according to experience while developping. Moreover, some packages have by far more consequences on the whole packages repository than others: glibc, gcc, binutils, etc. changes could break lots of things. Actually, as Developpers and Incoming would be entirely checked, that shouldn't be a problem. Anyway, I think it's wiser to put such important changes in another branch, that would be taken care of by some people, out of the classic repositories, until they can put in fixes in the main repositories, so the last check in (gcc, or glibc or whatever important) happens flawlessly, without disturbing all the others developpers normal way of working. So, for a "normal" evolution (i.e. nothing too revolutionnary) we could have:

• 1 month for step 1.
• 1 week for step 2 (should be doable if we have a compile farm with automatic management and reports for tests).
• 2 weeks for step 3: 1 so packagers can commit their fixes. 1 week for the compile farm.
• 1 month for step 4. Humans have lots of tests to do.
• 1 month and 1 week for step 5. 1st week so fixes can be commited, the remaining month so tests are extensive.

We end up with a 4 months schedule, which looks not so bad, imho.

When to make a public/iso/advertised release ?

Several well known (=recognized quality) distros and software programs are based on a 6 months or one year schedule. Here, we have a four months schedule when it comes to packages development. Anyway, according to package manager development schedule (see below), that we will have a six months schedule. So, what to do ? We all know that real life tends to be Free Software plague, because it's difficult to keep on roadmap. Anyway, a well known quotation says "release early, release often". So, my own opinion would be to make two releases a year, one for each stable. Once a year (each two releases), we could increase the major version number, to reflect progress made. That's just personal taste, nothing scientifically proven.

Security fixes, and how to manage them.

They are a really important point: 1st, it shows how efficient and secure we are. 2nd, being too quick or too slow can ruin stability. Moreover, there are two ways to put in security fixes: one is to apply patches, another is to jump to a newer release that corrects the problem, and often brings other changes altogether. The 1st case is the easier, because test phase can be short, thanks to (quite) small changes. So, we can quickly apply fixes to all branches, when it is needed. The second one is a bit more of work: we can't simply change the version number of a package directly in Release (unless the version upgrade only fixes the problem, so it is the 1st case), there are too many risks. Here, it is a packager work. The newer version will land into Developpers or Incoming (depends on the step we are in), whereas the maintainer have to extract the security fix, so only this part can be applied to the Release repository. In the end, a security fix has at least two steps: packager tests it thoroughlly, then commits to Release. A security fix that comes with a version upgrade goes to Developpers or Incoming, and then lands into Release.

Package manager, packages. How to make them grow up together without pain ?

A package manager gaining regularly new features can be a real pain to manage, often because packages have to adapt themselves to keep in touch with the manager evolution, increasing the probability of getting new bugs. However, there is a quite simple way to avoid such problems. How could we do that ? Oddly, the solution is really simple. The answer holds in a single word: planning ! Planning ? Of course ! A clear roadmap is enough. Features are fixed. Package manager is the obligatory base so we can safely develop packages on it. So, how does this link with the package developement roadmap ? As Release repository lives 4 months before the next evolution, it lets us test the manager during that time. That could be a solution. Otherwise, we could take two months to develop it, and have four months to test it, until the next Release is out. That leads us right on a six months schedule. Whatever the package manager developement cycle will be, thanks to erics (aka the boss:), a simple solution came out. It's, in fact, so simple, that I'm a bit upset not to have found it by myself ;). We just have to have the package manager as a package. Thus, there won't be any problem of outdated manager, making some recent packages badly fail. Just give the package manager top priority, so it is updated before any other package. Disallow user control on package manager package, and things will be ok.

Package manager development cycle

here is a quick drawing, with links to package developement cycle.

http://abtlinux.org/Wikka/images/globaldevcycle.png

You can see Package Manager development cycle is a bit longer. Two reasons: the first is, manager requires a bit more attention, being the master piece of the distribution. Second one is, Release will remain mainly unchanged during six months. So, having a quite slow changing landscape, bugs will hit and we'll have time to take care of them, both for package manager and packages. For a more precise description of package manager development cycle, just take the package dev cycle drawing, and consider the manager instead of packages. At it is a quite complex piece of software, Maintainer repos will allow people to work on parts of the code.

1st step: Development starts

Duration: two months. Thanks to a clear roadmap, a fixed API, each package manager developer knows his tasks for the next Release. Development here happens in Maintainer repositories, so everyone can check his changes are correct, as the remaining code is stable, and not disturbed by recent changes.

2nd step: Changes slow down

Duration: one month and a week. Maintainer repositories are merged into Developpers, conflicts are fixed. Developpers can still commit changes in Developpers (the repo), just in case they are late on roadmap. Hopefully, that step shall be where fixing only begins.

3rd step: Feature freeze

Duration: two weeks. Last step of code in Developpers. Only fixes are allowed. There, Developpers must have reached a state where it works perfectly with stable packages.

4th step: Developpers step finished, lands into Incoming

Duration: one month. extensive testing against Incoming packages.

5th step: FIXES '''

Duration: 1 month and a week. Testing, fixing against (updated) Incoming packages. Again and again. last Incoming step.

6th step: Release is released. Developpers (re)starts

Goto step 1.