How Ubuntu is built: the inside story
2nd Oct 2011 | 11:00
Behind the scenes with Ubuntu's community manager
How Ubuntu is built: the inside story
Every six months we release a new version of Ubuntu. Each one brings together hundreds of developers, translators, testers and documentation writers to integrate the latest and greatest upstream applications, as well as new and innovative Ubuntu technologies.
Building and releasing a new operating system every six months is hectic. Really hectic. However, since 2004, when we founded the project, we've striven to get the very best out of everyone who participates in building Ubuntu, ensuring that each release is as good as it can be.
This article explains how this organised chaos works and provides an insight into how Ubuntu is put together. Everything is encased in a rigorous release cycle, made up of a series of important milestones, which are always publicly available. Let's get started by looking at the start of a new cycle.
Casting the net
Before we even start the new release, and while we're still putting the finishing touches to the previous version, we start performing a requirements gathering exercise. The teams at Canonical reach out to different stakeholders and the community to see what the next release should focus on.
What new features does Ubuntu need? What problems require fixing? Which use cases should we support? This process results in a laundry list of needs that we start to sort into prioritised areas of focus.
At the same time, we're coordinating the structure of the next Ubuntu Developer Summit. This face-to-face event happens a few weeks into the start of every new release cycle. We send almost all of Canonical's Ubuntu engineers and sponsor a number of key community members to join us. The event is open to everyone, and many business representatives join in as well.
At the Ubuntu Developer Summit we have 14 tracks covering different themes and we schedule open discussion sessions for the various areas of focus. The goal of every session is to discuss the topic, and make and document decisions and assigned actions in public blueprints. These are web pages that we use to track the work on the features in an open and transparent way. You can see the interface at http://status.ubuntu.com for the next Ubuntu release.
The first task we need to tackle in a new cycle is getting the toolchain finalised. The toolchain is the core set of developer tools used to build software for the release. This deep and dirty low-level work is performed first, and then we synchronise the Debian Unstable archive with this toolchain. This effectively builds all Debian packages against the toolchain, so we have Debian Unstable in our Ubuntu developer repository.
When this sync is complete, Ubuntu developers start applying the hundreds of Ubuntu patches to these packages that transform Debian Unstable into Ubuntu. At this point, we have a developer version of Ubuntu that looks and feels like our last release, but built against the new Debian packages and using our toolchain. Now the real work can begin.
Catch of the day
Over the following months, the developer community starts working on the features and goals agreed upon at the Ubuntu Developer Summit. Progress is made every day: developers triage and prioritise bugs, fix them, and upload fixes to the archive. New features are developed, packaged and uploaded, as well.
Each day, when an Ubuntu developer wakes up, he or she will update their system to pull in the latest packages, and then start work on their features and bugs. Typically, this workflow involves looking at the current bug list and at the highest priority bugs and resolving those, while working on the feature goals for the cycle. This feature work often involves cherry-picking specific features from upstreams that are of interest, or creating the code for these new features and building them into the release.
An important piece of this is the new development work that goes into the Ayatana project with Unity, the indicators and other innovative desktop functionality. This work is broken into two primary teams: the Design team designs the functionality based around user needs, and the Desktop team uses these designs to write the code to implement them.
When the Desktop Experience team make a release, they send the code to the Ubuntu Desktop Engineering team, who then package it and upload it to the archive, where everyone can run and test the new code.
In each release, we strive to have a new Desktop Experience team release at least once a week (usually Thursday). This weekly deadline has been useful for regular progress. Throughout this period, we release development versions of Ubuntu that we encourage the community to test, file bugs on and help improve.
Alpha 1 is shortly after the new release cycle opens and includes the new Debian packages with merges included. Alpha 2 is when you typically start seeing significant new features. Alpha 3 is usually released a few months before the final Alpha and just before Feature Freeze.
At Feature Freeze, we lock the release down so no big new features are allowed in and all developer time is focused instead on refining what's already there. This milestone effectively switches gear from cramming in new and untested features to building quality into what we already have.
The big freeze
The big freeze
Shortly after comes the User Interface Freeze, which is designed to lock down the visual changes. This typically includes theming alterations, Unity changes and other visual components.
While this is a freeze we strive to maintain, we've often struggled to get everything finalised by this date and sometimes it slips a little. Nonetheless, the User Interface Freeze again latches the team into an even more quality-focused mindset and those with an interest in Ubuntu's UI now switch to fixing bugs.
At this point in the cycle we're ready to put out a first beta release and all hands are on deck to test, review and prioritise the bugs that are reported, and fix them. In this phase, all the engineering managers and community leaders focus on getting prioritised bug lists together.
We can never fix all the bugs, so we focus our efforts on the most critical areas to ensure we deliver the best possible user experience.
Shortly after the first beta release, we provide a kernel freeze to reduce the risk of hardware regressions, and we also have a String Freeze when we don't accept any more changes to the translatable pieces of the system. After these freezes are complete, we release our second and final beta, bang the drum to encourage people to test it and encourage users to file bugs for problems they see.
We continue to review these reports and prioritise them. The goal now is about fixing as many as we can before the Final Freeze. After a few final bits of buffing up, we kick out the new release and everyone in the Ubuntu community sleeps for an entire weekend.
A few days later, we start all over again…
The U Team
GO, TEAM!:The Ubuntu community at the Ubuntu Developer Summit in Budapest in May 2011
So far we've focused on the development of Ubuntu, but there are many teams who are working around the clock while all of this is going on.
The Information Services team runs the different servers and build farms, and so on, while the Engineering Managers provide governance and handle day-to-day decisions in the community (see Ubuntu Governance box, above right).
Other important teams include:
Web team: Prepares www.ubuntu.com for the new release, creates product showcases, prepares for the beta, and puts the professional sheen and face of Ubuntu across our web resources.
Communications team: Creates marketing materials, prepares the press for the beta and final releases, organises press calls, and liaises with different news outlets about features and interviews.
Design team: Works on Ubuntu itself, preparing visual assets for the web teams, the CD covers and brochures, and so on.
Admin team: Coordinates venues, travel and resources for the different sprints and rallies we run throughout the cycle and for the Ubuntu Developer Summit.
Finally, my team, the Community team, organises learning weeks throughout the cycle (including Ubuntu Open Week, Ubuntu Developer Week, Ubuntu Community Week, Ubuntu Cloud Days and Ubuntu App Developer Week). We also arrange worldwide events such as the Ubuntu Global Jam, Release Parties and more.
On the horizon
To provide some context, let's get a feel for what you can expect in the Ubuntu 11.10 release, scheduled for 13 October. These plans were discussed and agreed upon at the Ubuntu Developer Summit in Budapest, Hungary, back in May.
As ever, you can track the progress at http://status.ubuntu.com. Key features include:
Unity fit and finish
Ubuntu 11.04 delivered the first Unity desktop environment. In 11.10 the focus is on fit and finish, fixing reported bugs and improving on some areas of the interface. You'll see some new features, but the focus will be very much on quality.
A hugely compelling cloud deployment and management tool for making the deployment and scaling of cloud services a matter of a few commands. Ubuntu Server 11.10 will ship Ensemble as a key component and we're working to grow the availability of Ensemble formulas for popular server applications.
Thunderbird replacing Evolution
Although the final decision will be made later in the release cycle, work is ongoing to replace the Evolution email and calendar application with Mozilla Thunderbird. Work is currently going into building Unity support into Thunderbird and ensuring that the application is fully integrated.
The popular toolkit will be available in a default Ubuntu installation, thus opening up the available application and developer pool significantly to Qt developers. Qt is a powerful toolkit and first-class support will continue to make Ubuntu a strong developer platform.
We'll continue to refine with a new release of Ubuntu Developer Platform, http://developer.ubuntu.com, to reflect the different application development opportunities open to developers.
Worth the work
While we cram a huge amount into every release cycle, the organised chaos approach that the team has managed to carve out has resulted in a series of predictable releases of the distribution. In fact, since the formation of the Ubuntu project back in 2004, we've only ever delayed a release once – and that was a planned delay designed to accommodate the first long term support (LTS) release of the distro.
While it's an intense process, it's also incredibly rewarding and what makes it so are the phenomenal people who pour their heart and soul into every release. Release plans, freezes, milestones and other essential elements would be nothing if it weren't for these incredible people.
Some of you will love Ubuntu, and some will love another distro, but irrespective of your views, few could fail to be inspired by so many moving parts working together on something they believe in.
Jono Bacon is community manager at Ubuntu
First published in Linux Format Issue 149
Liked this? Then check out Inside Linux Mint
Sign up for the free weekly TechRadar newsletter
Get tech news delivered straight to your inbox. Register for the free TechRadar newsletter and stay on top of the week's biggest stories and product releases. Sign up at http://www.techradar.com/register