For the sake of tidying up some loose ends and pushing the envelope on what you can do in terms of building and loading your own modules, we're going to use one more lesson to cover the mechanics of compiling your own modules and show how that ties into the kernel source tree, your
git checkout and how to tweak the kernel source itself (a topic we'll cover in far more detail in a later lesson).
For the rest of this lesson, we'll assume that you've covered all earlier lessons and that they worked for you. If you haven't (or they didn't), now is a good time to go back and work through it all until you get back here.
Oh, and this lesson could be fairly lengthy so make yourself comfortable.
In order to work through the rest of this lesson, let's summarize what you should have on your system, to make sure we're all working with the same content.
First, all of these examples are being run on a fully-updated, 64-bit Ubuntu 10.04 system, but you're welcome to use some other Linux distro as long as you're prepared to translate things like the package names appropriately.
To build your module against the kernel tree for the current kernel, you should have installed the corresponding kernel headers package for that kernel.
In addition, to allow us to play out on the cutting edge of kernel development, you should also have used
git to clone the main kernel source somewhere under your home directory, and keep it up to date with the occasional
git pull command being run from the top of that source tree.
Next, you should have the standard development packages on your system. And if you managed to compile and load your sample module in an earlier lesson, it's a safe bet you have everything you need for the rest of this lesson.
And, finally, for consistency, you should be running the stock, default kernel for your Ubuntu (or other) system, not any custom kernel you might have built. So if necessary, reboot your system to ensure you're running the
2.6.32-22-generic and we'll take it from there.
Given that you have a choice of kernels you might run, and a variety of possible source trees, let's cover the possible combinations for building and loading your own modules, based on the
crash1.c example from an earlier lesson, and this portion of the Makefile you used:
ifeq ($(KERNELRELEASE),) KERNELDIR ?= /lib/modules/$(shell uname -r)/build PWD := $(shell pwd) .PHONY: build clean build: $(MAKE) -C $(KERNELDIR) M=$(PWD) modules
In case you'd forgotten, the purpose of the snippet above is to determine which kernel tree to build against, and that's where your choices come in. And here are your choices.
This is the simplest combination, and doesn't require you to create and reboot a new kernel. You simply write your module, at which point the Makefile above uses as the necessary kernel source tree the symlink that takes you to the source tree that came as an Ubuntu package. Everything matches, and everything works fine, and after you compile your module, you can see clearly that it was built against the kernel that is currently running:
$ modinfo crash1.ko filename: crash1.ko ... snip ... vermagic: 2.6.32-22-generic SMP mod_unload modversions
Feel free to build that module again and make sure it loads and unloads properly. This variation of module building is ideal for someone who has root privilege, but doesn't have the freedom to build and boot to a new kernel. So what's next?
You don't need to verify this section, but it should be obvious that if you have the freedom to build your own kernel and install it as described in earlier sections, you can build and load your modules exactly the same way since, once you install that new kernel and reboot, the symlink above will refer to the source tree against which you built your module, so the versions will match and, again, everything will work.
You can verify this if you want but we covered this in an earlier lesson so you can just take my word for it. But let's talk about a slight different variation now.
This is possibly the most interesting variation. Assuming you're running the stock Ubuntu
2.6.32-22-generic, you can still build your module against a different version of the kernel source tree, let's say in preparation for booting to a new kernel shortly. By building your module against a different source tree, that means you won't be able to load it until you're running the matching kernel version, but it's still interesting to see how the process works because it contains a couple traps.
Let's spend some time on this variation since it will explain a few more issues related to modules and kernel source trees.
Let's assume you've done a
git checkout of the latest kernel source (into, say, the directory
~/k/git) and that it's reasonably up to date. In short, the version of that source tree clearly doesn't match the version of the running kernel. What effect is that going to have on your module building?
First, go over to the git checkout and run
make distclean to clean out any remnants of previous configurations or builds you might have run. And now, back in your module directory, examine this line from your Makefile:
KERNELDIR ?= /lib/modules/$(shell uname -r)/build
If you're familiar with Makefiles, you'll know that the above is a conditional assignment, in that it will assign the standard kernel source location to the
KERNELDIR variable only if it hasn't been set already. And, normally, that's just fine because that default value is what you want.
But if you're running the default kernel and yet want to build your module against the cutting edge kernel source, that's exactly the variable that you'll want to override to do that. In short, what you would do is clean out your module directory with:
$ make clean
then rebuild the module while overriding the kernel source tree location with:
$ make KERNELDIR=~/k/git
Theoretically, that will build a module against the newer kernel and loadable only when running a kernel with the same version number. It seems simple, but there's still a catch.
This is the catch:
$ make KERNELDIR=~/k/git make -C /home/rpjday/k/git M=/home/rpjday/courses/crash/lkp/crash1 modules make: Entering directory `/home/rpjday/k/git' ERROR: Kernel configuration is invalid. include/generated/autoconf.h or include/config/auto.conf are missing. Run 'make oldconfig && make prepare' on kernel src to fix it. ... snip ...
Oops, what just happened there? This is what happened.
You can't build a module against an absolutely sterile, pristine,
make distcleaned kernel tree. At the very least, the kernel tree being used must be configured because there are some versioning files that are created by the configuration process that are essential to the module build process. But that's not enough.
You also need to perform at least the first part of the kernel build process since that will generate a few more files that the module build process needs. However, unlike what some documentation tells you, you don't need to perform an entire build. All that's required is, in the kernel source tree, to run the subsequent command:
$ make modules_prepare
Without getting into massive detail, that make invocation will run just enough of the kernel build process that the tree is now ready to let modules build against it. And once you do that in your kernel source tree, your module build should now work:
$ make KERNELDIR=~/k/git make -C /home/rpjday/k/git M=/home/rpjday/courses/crash/lkp/crash1 modules make: Entering directory `/home/rpjday/k/git' WARNING: Symbol version dump /home/rpjday/k/git/Module.symvers is missing; modules will have no dependencies and modversions. Building with KERNELRELEASE = 2.6.35-rc3 CC [M] /home/rpjday/courses/crash/lkp/crash1/crash1.o Building modules, stage 2. Building with KERNELRELEASE = 2.6.35-rc3 MODPOST 1 modules CC /home/rpjday/courses/crash/lkp/crash1/crash1.mod.o LD [M] /home/rpjday/courses/crash/lkp/crash1/crash1.ko make: Leaving directory `/home/rpjday/k/git' $
and I can verify the kernel version of the created module file with:
$ modinfo crash1.ko filename: crash1.ko ... snip ... vermagic: 2.6.35-rc3 SMP mod_unload ^^^^^^^^^^
and there we go. However, as I've already warned you, since you built that module against a different kernel version, you can't load it under the current kernel:
$ sudo insmod crash1.ko insmod: error inserting 'crash1.ko': -1 Invalid module format $
And all of the above should let you conclude one more obvious fact -- that the distro-specific kernel headers package installed on your system that lets you compile modules against it clearly represents at least that much of a configured kernel source tree; it can't just be pure source or you would have run into exactly the same failure as you just saw.
I'm glad you asked. As we've already established, some people will not have the freedom to build and run their own custom kernels, as much as they would like to, so they're restricted to simply building modules against the default kernel source and loading and unloading their modules. And for a lot of people, that's sufficient.
If, on the other hand, you have the flexibility to check out the git repository, then configure, build and reboot a custom, state-of-the-art kernel, not only can you build and load your modules against that new kernel source, but you can tweak the kernel source itself, and
git will always let you revert all your changes back to the repository contents after you've finished playing.
To see how this works, move to the top directory of your
git kernel source tree, and verify that nothing's different from the repository contents with:
$git diff $
See? No output, no difference. Now let's pretend we wanted to tweak the source, and rebuild a kernel to see the difference. If you're feeling ambitious, examine this snippet of code from around line 396 of the kernel source file
printk(KERN_INFO "Brought up %ld CPUs\n", (long)num_online_cpus());
If you've ever watched your system boot, you might recognize that line of output embedded in the midst of all the other boot messages. And if you have root privilege, you can see the record of all those boot messages in the log file
$ sudo grep "Brought up" /var/log/messages ... snip ... Jun 19 06:36:52 lynx kernel: [ 0.320009] Brought up 2 CPUs Jun 19 08:00:09 lynx kernel: [ 0.330014] Brought up 2 CPUs Jun 19 11:22:49 lynx kernel: [ 0.330015] Brought up 2 CPUs
And why do you care?
Because if you have the freedom to build your own kernel, you certainly have the freedom to start messing with the source, as in changing that boot-time diagnostic to something more entertaining, as in:
printk(KERN_INFO "Oh, my, there are %ld CPUs\n", (long)num_online_cpus());
And if you make that simple editing change and save it,
git will be happy to tell you how your current checked-out source differs from the HEAD revision, as in:
$ git diff diff --git a/init/main.c b/init/main.c index 3bdb152..cd13d5b 100644 --- a/init/main.c +++ b/init/main.c @@ -393,7 +393,7 @@ static void __init smp_init(void) } /* Any cleanup work */ - printk(KERN_INFO "Brought up %ld CPUs\n", (long)num_online_cpus()); + printk(KERN_INFO "Oh, my, there are %ld CPUs\n", (long)num_online_cpus()); smp_cpus_done(setup_max_cpus); }
If you're feeling ambitious, you can make that kind of innocuous change, save it, rebuild your kernel, install it, reboot and verify that, yes, your boot-time log message is now different. Which is something we'll be doing later in the course when we start modifying the kernel source in more meaningful ways.
And when you're tired of your local checkout changes, you can always simply revert everything in the entire tree back to the original source with:
$ git reset --hard HEAD HEAD is now at 7e27d6e Linux 2.6.35-rc3 $ git diff $
at which point, as you can see,
git diff now verifies that you no longer have any local changes.
How much of the above you want to test is entirely up to you. And in the next lesson, we get back to writing and extending our loadable modules.
As they used to say on the TV series Sports Night, we'll be back so stick around. And by all means, leave questions and other feedback in the comments section. That's what it's there for.