Quite simply, this is a tutorial on how to download, configure, build, install and reboot a new Linux kernel. It's also the first lesson in what will be a multi-lesson course on introductory Linux kernel programming, but I don't need to get into that at the moment. Instead, I'd rather just get down to business since there will be plenty of time later to take the long view, and lay out the plan and so on.
And, yes, I know this is supposed to be a course on Linux kernel programming for beginners, but we definitely need to establish the fundamentals first. So the first few lessons will concentrate on kernel configuration and build issues, and we should get around to having you write and run your first kernel module by lesson four. So be patient -- it will all be happening in due course.
And on that note, let me make a brief disclaimer. This first lesson will be a bit more terse than what I normally write since I simply want to get this done and published and let readers start playing. I'll come back over the next day or two, add a few more thoughts, clarify anything that needs clarifying -- you know, that sort of thing. But for me, it's time to stop talking and start writing so you folks can start experimenting.
Most importantly, there's a comments section at the bottom of this piece, so I encourage you to ask questions, make observations, demand clarification, whatever you want. This is meant to be interactive, so by all means, interact. And there is one more point to be made.
This is the first of four free kernel programming lessons that will lead into my proposed online kernel programming course that you can read about here. In short, after this and the next three lessons, you'll know whether you want to enroll for the rest of the course. All I can say is, I will do my best to make it worth your time. And if you want to keep up with breaking developments, feel free to follow me on Twitter.
Good question and, later, I'll come back and fill this in. For now, I'll simply answer -- because you can. And for a lot of people, that's a good enough reason right there.
Ah, an excellent question, given the variety of Linux distros out there. Building and booting to a new kernel should really be distro-agnostic -- the process should be (with subtle differences) pretty much the same from distro to distro and I'll try to document the differences you need to care about. But to maximize your chance of success, I'm going to pick a particular distro and version and demonstrate the process based on that. Which means that if you're running the same combination, what follows should just work. And I don't make that claim lightly. Let me explain.
Perhaps the most frustrating aspect of a lot of documentation is the occasional waffling, when the author writes something like, "And if that doesn't work, try ...". Yes, that can be annoying because, if you're trying to follow along, it can be disheartening to realize that the instructions you're reading aren't guaranteed to work. And that's where this lesson (and all others to follow) are going to try to be different.
If you're using the same distro I am, and you install the same packages, and you follow the instructions to the letter, then what you're doing should just work. With no exceptions. Guaranteed. If it doesn't, and you've done everything correctly, then I'm the one who screwed up. Let me know, and I'll fix it. Because this shouldn't end up being guesswork or trial-and-error on your part.
And on that note, the distro I've settled on for this is Ubuntu 10.04, for no particular reason. I've written the same material for Fedora and all of this should work equally well for Fedora, but I wanted to give readers the opportunity to match my working environment exactly. So if you have the freedom, install Ubuntu 10.04 on your favourite system, and bring it up to date. And we'll take it from there.
I'm going to assume that you at least have the background to
install the standard development environment on your system. You know,
make and so on. If you're missing an essential package, you'll know quickly enough. In addition, you'll want to install the
git utility for version control and checkout. On Ubuntu, it's:
$ sudo apt-get install git-core
In addition, to use any of the kernel configuration targets you'll see in the next lesson, you'll need the Curses development library:
$ sudo apt-get install libncurses5-dev
Beyond that, I can't think of anything else that won't already be on a standard Ubuntu install, but feel free to correct me in the comments section.
Before we go any further, let me introduce you to the existing kernel components on your system so you can see what goes into the bootable kernel bits, and also so you can see what's going to change after you build and boot your new kernel, because it's immensely valuable to appreciate the differences before and after you do certain steps, so let's check out the parts one by one.
uname command tells you about your running kernel. It has various options, but the one I normally care about is just
-a, for all information. My system:
$ uname -a Linux lynx 2.6.35-kwlug+ #1 SMP Mon Jun 7 13:13:50 EDT 2010 x86_64 GNU/Linux $
That shows that I'm running some variation of the (unreleased) 2.6.35 kernel, and one I built as a demo for local Linux users group. And it's the command you'll run after you boot your new kernel to verify that, yes, you really are running that new kernel. So that's bit number one. On to the other bits.
All the good stuff is going to be under your
/boot directory. Here's what mine looks like:
$ ls /boot abi-2.6.32-21-generic memtest86+.bin abi-2.6.32-22-generic System.map-2.6.32-21-generic config-2.6.32-21-generic System.map-2.6.32-22-generic config-2.6.32-22-generic System.map-2.6.34 config-2.6.34 System.map-2.6.34-rc7 config-2.6.34-rc7 System.map-2.6.35-kwlug+ config-2.6.35-kwlug+ vmcoreinfo-2.6.32-21-generic grub vmcoreinfo-2.6.32-22-generic initrd.img-2.6.32-21-generic vmlinuz-2.6.32-21-generic initrd.img-2.6.32-22-generic vmlinuz-2.6.32-22-generic initrd.img-2.6.34 vmlinuz-2.6.34 initrd.img-2.6.34-rc7 vmlinuz-2.6.34-rc7 initrd.img-2.6.34-rc7.new vmlinuz-2.6.35-kwlug+ initrd.img-2.6.35-kwlug+ $
Obviously, I have a number of bootable kernels in there. Some are stock Ubuntu offerings, while others are clearly hand-rolled by me. Each bootable kernel should be represented by a combination of matching files:
vmlinuz-2.6.35-kwlug+: The compressed, bootable kernel image
initrd.img-2.6.35-kwlug+: The initial ram disk -- an early root filesystem that allows your kernel to bootstrap and get essential device drivers to get to the final, official root filesystem
config-2.6.35-kwlug+: The record of the configuration
And why do you care about all that? Because once you think you've configured and built and installed a new kernel, you're going to check that directory again and make sure that you have new files corresponding to all of the above. Because if you don't, something's gone horribly wrong. So the above is going to act as a sanity check to make sure your build process worked properly before you try to reboot. So far, so good? And finally ...
Each kernel installed on your system will have a corresponding directory of loadable modules to go with it. This is what it looks like on my system at the moment:
$ ls /lib/modules 2.6.32-21-generic 2.6.32-22-generic 2.6.34 2.6.34-rc7 2.6.35-kwlug+ $
Not surprisingly, after we build and install a new kernel, we should expect to see a whole new directory of modules there.
The last essential bit to booting to a new kernel is that you need to add an entry for your new kernel into the GRUB bootloader configuration file on your system. On Ubuntu, that file is here:
$ cat /boot/grub/grub.cfg
Don't worry if it looks like gibberish at the moment. The only thing to remember is that once you've built and installed all your new kernel bits and pieces, you have to run the appropriate utility to add entries for it to your GRUB config file; otherwise, GRUB will never know about it and you'll never be able to boot it. Don't worry -- you'll be doing that later.
You can get a kernel source tree in a number of ways but let me make this easy -- let's live out on the edge and use the contents of the working Git repository, so you can brag to your friends that you're tracking the bleeding-edge developments. It's more fun that way.
Assuming you've installed whatever package gives you the fundamental
git command, simply pick a location somewhere in your normal user home directory or underneath it and:
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux-2.6
and let me emphasize something about the process so far.
You do not need root/superuser access to download a kernel source tree, or to configure it, or to build it. You will need root privilege only to install the final product(s) in
/boot and to upgrade the GRUB configuration file. But until then, you have no need for root privilege.
In any event, pick a location, start that
git command running and, depending on your bandwidth, go for coffee.
Now that you have your kernel source tree, you need to configure it, which involves selecting which features you want built into the eventual kernel image, which features you want built as loadable module, and which features you want to omit entirely.
If you are so inclined and you've done this sort of thing before, you can jump right in and run one of the configuration variations like:
$ make menuconfig [my favourite]
but that assumes you have a pretty good idea of what you want that would be appropriate for your system. If you don't have that kind of background, you'll want to take the safer route. [I may devote a future lesson to nothing but the intricacies of the kernel configuration process. Believe me, there's enough there to fill a lesson. But not right now. Onward.]
NOTE: All of the
make commands you see here must be run from the top of the kernel source tree.
The obviously easiest thing to do is to configure your source tree to match as closely as possible the configuration of your running kernel. What could be a safer choice? Build your new kernel to resemble as much as possible your current kernel.
To do that, use the existing configuration file as the basis for your configuration thusly. Move to the top of your new kernel source tree, and copy an appropriate config file:
$ cp /boot/config-2.6.32-22-generic .config
What the above does is copies what should be a reasonable config file for your system into the hidden
.config file at the top of your kernel source tree. Since the purpose of the normal configuration process is to take all your choices and generate that
.configfile, all you've effectively done is short-circuit the configuration step and grab what you want as the end result, at which point you would run the following command which tweaks that config file to bring it into line with the kernel source:
$ make oldconfig
But wait. There's a bit more to this story.
It's entirely possible that that existing
.config you used as the basis for your configuration isn't quite up to date; that is, it may have no entries representing extremely new features that have been added to the kernel. If that's the case, the above config step will stop at each one of those choices and ask you what to do. And if you're new to building a kernel, you may not know the right answer.
One solution is to just keep hitting ENTER and take the default, but that can get tedious. A faster solution is:
$ yes '' | make oldconfig [two single quotes, no space between]
Why the above works is left as an exercise for the student. In any event, what you end up with is a
.config file that is based as closely as possible on one you know that works and has also been adjusted to exactly correspond to your kernel source tree. So, at this point, you have your source and you've configured and recorded the result in that config file. Are you ready to build? Almost.
(As an aside, that
.config file I keep talking about is simply a text file, and you're welcome to look at it:
$ less .config
Wasn't that exciting? Onward.)
Not surprisingly, your new kernel needs to be somehow distinguishable from all of the other possible kernels you can boot, and that identification comes from the first few lines of the top-level
Makefile in the source tree. Behold, an example:
VERSION = 2 PATCHLEVEL = 6 SUBLEVEL = 35 EXTRAVERSION = -rc2 NAME = Sheep on Meth
If your Makefile contains the above, the resulting kernel will have the "name" 2.6.35-rc2. To give your kernel a unique identifier, simply edit the Makefile and change the value of the
EXTRAVERSION variable. For our build, let's go with:
EXTRAVERSION = -crash
You'll see what effect this has after the build. And constantly changing that
EXTRAVERSION variable lets you build as many kernels as you like, if you want to experiment with tweaking them all slightly differently.
Really pretty simple:
Go for a beer. Come back. And if the universe unfolded as it should have, you have a new kernel and loads of modules ready to be officially installed.
If you've followed along so far, it should be fairly obvious what you need to do to take the results of your build and install all of it to make it bootable. What needs to be done:
/bootdirectory so that GRUB can find it at boot time
Different distributions supply different utilities to make the above more convenient but I'll do all of that manually so you can see the process in action. One step at a time (and, again, this is being done on Ubuntu 10.04 so if you're using a different distro, your instructions will be different), you'll run the following commands and, now, you'll need root privilege since you're copying files into system locations.
$ sudo make modules_install $ sudo make install $ sudo update-initramfs -c -k 2.6.35-crash+ $ sudo update-grub
(NOTE: I just now noticed that the kernel that is built by the above is given a name with the suffix "crash+", not just "crash." I hadn't noticed this in earlier versions of the kernel build process and I'm guessing that it represents a build based on a checkout from a version control system. I'll verify that shortly; just take the time to check if you have the same result.)
As I'm sure you can guess, each of those commands will install or update some system file or directory with the result of your build. You can run all of them at once, or one at a time and check the result after each one. (As a reader challenge, I'll let you deduce what change each command produces.)
Once all of the above is done, you can verify that everything seems to have been installed properly by checking the following:
/bootdirectory have a new kernel, initrd image file and config file corresponding to your build?
If you can answer "yes" to all of the above, you're probably ready to reboot to your new kernel.
(Aside: I realize that Ubuntu has a utility called
make-kpkg that allows you to take your build results and turn it into a
dpkg-installable package, and I'll come back later and document that. But the end result would be exactly the same. So let's just go with the above for now.)
If all of the above looks good, you might as well take a shot and try to reboot to your new kernel. That simply involves rebooting your system and holding down the SHIFT key to get the GRUB bootloader's attention, which should dump you into a menu from which you can select the kernel you want to boot. On Ubuntu, you actually get two new choices -- a normal kernel and a recovery kernel. Pick the regular kernel once you get to the menu.
So ... did that work? Are you up and running under the new kernel? How can you tell?
$ uname -a Linux lynx 2.6.35-crash+ #1 SMP Fri Jun 11 08:17:50 EDT 2010 x86_64 GNU/Linux $
There's your proof. Congratulations. You've arrived.
I'll certainly be adding a few more thoughts to this piece based on reader feedback but here are some final notes to keep you busy and experimenting.
First, what if booting to your new kernel goes horribly wrong and the system simply hangs? No problem -- just power cycle, pop back into the GRUB menu and select an older, reliable kernel. Then roll up your sleeves and try to figure out what went wrong and fix it.
Next, even if the new kernel boots, that's no guarantee that every single feature that used to work still works. Test everything. Networking still there? Wireless still working? Sound still functioning properly? It's always possible that you'll have 98 per cent of your former functionality but some new kernel feature is causing trouble.
If you take a look at the loadable modules that were installed and see how much disk space, it's likely that your loadable modules directory is massively larger than the ones that were there before. That's because you neglected to strip them when you installed them. To save disk space:
$ make INSTALL_MOD_STRIP=1 modules_install
The difference is noticeable but, eh, disk space is cheap.
To start all over with a pristine source tree, you can remove every vestige of your build with:
$ make distclean
at which point you'll have to start the whole config/build process from scratch.
Finally, if you want to keep up with current kernel developments, you can constantly:
$ git pull
and, if you think it's worth it, build a new kernel. It's entirely up to you.
I'll still come back later and add a few more notes but, at this point, you know enough to give this a shot. Try it and let me know what happens in the comments section.
And for further reading on kernel configuration and build issues, you really need a copy of Greg Kroah-Hartman's book Linux Kernel in a Nutshell. Seriously. Either read it online or buy it. Just get it somewhere.
We're aware of the time and budget pressures at most companies, normally accompanied by the plaintive cry from management of, "Yes, I know we need training on that topic, but I just can't afford to send my entire team away for three (or four or five) days to get it!" And that's where we come in.
The main focus at Crashcourse is to offer a choice of intense, 1-day, hands-on courses on specific topics in Linux and open source. And given that we already have the laptops for the delivery of that training, the idea is to show up early, set up a classroom, then spend the day teaching exactly the topic you're interested in. No travel time, and no wasted classroom time.
If we don't already have a course that addresses the topic you're interested in, drop us a note and we'll see what we can do -- our content providers can almost certainly put together a course that's precisely what you're after.
While there are a variety of sources for Linux and open source training, we at Crashcourse are taking a slightly different approach. Our philosophy is simple: exactly the training you want, and no wasted time or travel to get it.