codemonkey.org.uk » Fedora

An annoying kernel packaging bug.

davej — Thu, 10 Dec 2009 23:32:25 +0000

The kernel rpm package creates an initramfs file that gets dropped in /boot at install time. Creation of this file is the responsibility of dracut (or mkinitrd in older releases), called from the kernels %post script. Because we want removal of a kernel to also remove its associated initrd file (or else /boot would fill up), we used to list the initramfs file in rpms database as a %ghost file owned by the kernel.

For a long time, this hasn’t been a problem. Now that dracut creates more feature-heavy initramfs files, people were noticing that they were running out of space in /boot during the installation of the kernel rpm. People were asking “why doesn’t rpm check for sufficient diskspace before it tries to install ?” Well, it does. The problem is that for %ghost files, it has no way of knowing how big they are going to be.

So for Fedora 12, we ended up with an ugly hack. Instead of listing the file as a ghost, we create a 20MB empty file, owned by the kernel, which gets overwritten during %post by dracut with the real initramfs.

The big problem with this, is that it destroys the possibility for deltarpms to work. (It also guarantees that signature verification using rpm’s database will fail)

Something better needs to happen, but at this stage, I’m not entirely sure what that will be.

An annoying kernel packaging bug. is a post from: codemonkey.org.uk

No related posts.

exploded Fedora devel tree.

davej — Wed, 25 Nov 2009 17:56:32 +0000

I did a make prep of every package in the devel/ branch of Fedora CVS again.
The last time I did this (a year or so ago) it took up 80G of disk space. Today, it takes up 100G.

That’s a lot of code. Some of it even does what it’s supposed to.

exploded Fedora devel tree. is a post from: codemonkey.org.uk

x86info processor topology parsing. Today was my last working day of the year, as...
An annoying kernel packaging bug. The kernel rpm package creates an initramfs file that gets...

Increasing testing of unreleased kernels.

davej — Mon, 29 Jun 2009 21:58:59 +0000

This past weekend I’ve been thinking of reviving an idea that has come up countless times. Producing RPM builds of the rawhide kernel for our already released Fedoras. The reason for not doing them so far has come down to bandwidth. (in terms of build system throughput, disk space, mirroring, and people bandwidth).

What I’m toying with doing is some devel kernels for Fedora 11 that are built outside of the Fedora build system. The Fedora kernel team now has enough build bandwidth for x86-[64] that we can actually get builds for those architectures done faster than koji.

Disk space – I’m thinking of just keeping the last 2-3 builds available.

Mirroring – Instead of having these be part of Fedora proper, I think an external repo on something like fedorapeople.org will suffice.

Which just leaves people bandwidth. For the most part the work is going to be just regularly syncing the devel/ branch with a CVS branch of F-11/ For some of this work, some scripting could be done to alleviate some of the pain. Also the frequency at which we push out these builds will determine the pain point. Perhaps every -git isn’t particularly valuable anyway. One build every handful of -git’s should be sufficient for bisecting.

There does remain one additional barrier. Occasionally we introduce something in rawhide builds which just won’t work on F11. For example, the kernel modesetting patches are tied closely to Xorg packages. Sometimes upstream changes require changes in mkinitrd or udev or some other ‘plumbing’. Some of these are regressions, and hopefully by identifying them sooner we can get them reverted/fixed upstream. Sometimes however, things get deprecated, and we need to change these packages. I’m not sure how to cope with this yet in a devel-for-F11 scenario.

One other thing that might be fun to throw into this would be the generation of -vanilla packages. The only reasons we don’t do these as part of the regular kernel builds is the various bandwidth concerns above. The specfile copes with spitting out RPMs with very little work needed. Josh Boyer has been occasionally doing these builds, though there hasn’t been a huge uptake. It’s unclear if this is due to lack of interest, or just a lack of publicity.

Another question to be answered is whether we go the route of enabling debugging in all builds as we do in rawhide, or do separate -debug builds. I’m leaning towards the latter.

I’m not committing anything to this for sure just yet, but it’s something I’ve been giving quite a bit of thought. There are still a bunch of unanswered questions.

Increasing testing of unreleased kernels. is a post from: codemonkey.org.uk

No related posts.

Fedora 11 released, onwards to F12.

davej — Fri, 19 Jun 2009 15:36:49 +0000

With the release of Fedora 11 recently, we are now back in the position of looking onwards to the next release. F11 released with a 2.6.29 kernel, and we’re already looking at doing a .30 rebase for it soon. (I was hesitant to type that, because the last time I blogged about doing a rebase, we hit some troubles and ended up skipping the 2.6.28 release for F10). While F11 was stabilising before release, devel/ had continued on, being rebased up to 2.6.30. Yesterday Kyle committed the the first rebase to get us back up to Linus’ tree of the day.

So we’re looking at 2.6.31 for F12. (With various conferences coming up over the next few months, it seems infeasible that .32 will land in time for F12′s release).

Other planned changes? We’ve talked about dropping the exec shield patch that we’ve been carrying since Fedora Core 1.
It’s a pain to have to keep carrying it, and rebasing it, and occasionally fixing it, just to add a poor emulation of a feature that has been in all CPUs for the last five years. The decision isn’t final yet, but it’s something that’s being considered.

Some of the other patches we’ve been carrying (like modesetting drm) are now starting to find their way upstream too, which is obviously a good thing. The only really big thing we’re still carrying that struggles to get upstream is utrace.

Fedora 11 released, onwards to F12. is a post from: codemonkey.org.uk

exploded Fedora devel tree. I did a make prep of every package in the...

execshield split-up.

davej — Wed, 11 Mar 2009 19:28:13 +0000

One of the longest living patchsets we’ve carried in the Fedora kernel is that of execshield. Over time, bits of it have gone upstream (in particular, some of the randomisation bits). In F11, it’s still a 1000 line 30K diff, touching all manner of core kernel functionality. To try and get more of it pushed upstream, I’ve been working on splitting it up into its component parts.

The current state of the diffs is at http://www.codemonkey.org.uk/projects/execshield/.

The emulate-NX-with-segment-limits chunk is unlikely to ever go upstream. A bit of a shame given it’s the largest part of execshield remaining. Linus wasn’t thrilled by it, and it is a pretty nasty hack.
Also, with modern CPUs having hardware-NX, it becomes less useful over time. (Though we still need to carry it judging by the number of old-school 686 users we still have).

So if we do have to keep execshield, we should at least try to make it cleaner and smaller. Every time I poke at it, I manage to shave off another hundred lines or so.

execshield split-up. is a post from: codemonkey.org.uk

No related posts.

Fedora kernel packaging changes

davej — Mon, 09 Feb 2009 18:50:49 +0000

We recently made a change in Fedora kernel packaging, to drop the regular 686 kernels. As of Fedora 11, the only 32-bit kernels built are ’586′ and ’686-PAE’ (and their -debug variants).

In F11, at install-time, Anaconda will install the PAE kernel if the CPU is capable of using it, and otherwise, will fall back to the 586 kernel. (Users upgrading with yum will see a 686->586 transition unless they are already using the -PAE flavour)

Both kernels actually have instruction scheduling optimised for 686 (Using gcc’s -mtune=generic, which optimises for the popular CPUs of the time).

The actual differences between the two builds:

586:

Uses only up to the i586 ISA
Supports up to 4GB of RAM.
Will use segment limits to approximate NX.

686-PAE:

Uses the 686 ISA including CMOV.
Uses 3-level page-tables allowing >4GB RAM.
Where possible, uses hardware NX instead of segment limits. (faster)
Has Xen virtualisation enabled.

Processors that will need to use the 586 kernel include..

all AMD CPUs prior to the Athlon
Intel Pentiums prior to the Pentium Pro
Some subset of Intel Pentium M’s (These are the only ‘modern’ CPUs)
All Cyrix and Transmeta
All VIA prior to Esther

Fedora kernel packaging changes is a post from: codemonkey.org.uk

An annoying kernel packaging bug. The kernel rpm package creates an initramfs file that gets...
exploded Fedora devel tree. I did a make prep of every package in the...

FUDCon redux.

davej — Mon, 12 Jan 2009 15:33:43 +0000

The last couple of days have been those sort of ‘awake too early / asleep too late’ sorts of days that are typical of a Linux conference. FUDCon has been kind of awesome. Friday, I sat in on a hackfest/brainstorming session on automated QA, where a whole bunch of interesting ideas were thrown around. I got to talk a little about the work I used to do grepping across whole checked out source trees, and why it isn’t practical any more, and we came up with some ideas of how we could do some of the checks at check-in/build-terim instead. Because I’m a life novice, I had something of a panic attack mid-morning, when I thought I had lost my checkbook. (I had it with me the previous day, but couldn’t remember what I did with it, and it was no longer where I remembered it last). So I bailed for an hour or two to run home and put my mind at rest. (It was of course, safe at home). It also meant I got to grab my phone which I had forgotten in the rush to get out that morning. When I got back, I hung out in various sessions, mostly ‘the hallway track’, and had various interesting conversations throughout the afternoon. I pitched a hackfest session on dracut, but by the time I had got back, it was pretty clear it wasn’t really going to happen.

Saturday was the barcamp day, with a lot of interesting presentations. This FUDCon, I didn’t do ‘the kernel report’. Instead Kyle did it, and people seemed to like it. I followed up with a session providing an overview of what we hope to accomplish with dracut, and the current state of it. Lots of good questions. Just before Paul Frields wrap-up session, I discovered a neat ‘turn a photo into an obama photo’ generator, and came up with a fitting backdrop for his talk. It turned out well for a rush-job started five minutes before he started talking.

In the evening, was the ‘FUDpub’ session at flat top Johnnies. By this time, I was starting to feel a bit run down, and spaced out. I sat on one of the sofas and just chatted the whole night. A bunch of people I know I’ve met before came up and talked to me, but I was so out of it, that I didn’t recognise them. (Apologies if this was you!). It wasn’t until later in the evening I realised who various people were who I had talked to. We had another dumping of snow just in time for us to head home.

I bailed on FUDcon for Sunday, and stayed home hacking with Kyle. We dug into a few things plaguing us, like the ‘huge modules on x86-64′ problem. Roland McGrath finally got to the bottom of that one. Other than that, and the inevitable ‘catch up with email’, it was kind of a winding-down day for FUDcon.

FUDCon redux. is a post from: codemonkey.org.uk

Not attending FUDCon Toronto. Many of my colleagues and lots of Fedora community folks...

Last post on leap seconds.

davej — Thu, 01 Jan 2009 23:03:03 +0000

I thought I was done with this. Then, today I saw this. To the best of my knowledge, Fedora 8 didn’t suffer from the bug I originally described several posts ago. I think this one happening at nearly midnight UTC is coincidence.

There’s a “me too” in the comments, but it seems odd that two people on slashdot saw it, but we never heard a peep on the Fedora mailing lists, or in bugzilla. Or even in upstream kernel.org. It could just be coincidence, the story is unsurprisingly short on details. I guess slashdot stories are easier to write than bug reports. But without additional debugging info we won’t ever know. Bear in mind that last time we saw a crash of this nature it didn’t affect everyone then either.

It was only by chance I managed to catch the backtrace in the `06 crash. I actually had two locked up machines, but one had its screen blanked, and wouldn’t unblank. The other machine had blanking disabled (setterm -blank 0) and thankfully, had also been set up to use a VGA screen resolution so had plenty of lines to display the whole backtrace.

Update: a problem has been found, and fixed.

Last post on leap seconds. is a post from: codemonkey.org.uk

No related posts.

Linux kernel 2.6.28

davej — Thu, 25 Dec 2008 01:23:01 +0000

Linus just released the 2.6.28 kernel. It’s already compiling for tomorrows rawhide. Fedora 9 & 10 will probably move to it in a few weeks. Typically, we wait until the dust settles and the first -stable release comes out. I was asked recently what bits we’re excited about in .28 for Fedora. To be honest, I didn’t give a great answer. It’s just not a “OMG, THIS RELEASE IS AWESOME” kind of release. There’s nothing in there that I was disappointed not to get into .27 for F10′s release. In fact, lots of the bits in there we were already carrying in the Fedora kernel (the DRM bits for example). Asides from that, it’s the usual churn of bug fixes, new drivers, and probably some interesting new bugs.

What about F11 ? Looking at the current schedule, we’ll get at least .29 in. I’m not sure we’ll have enough time to pull in .30 at this stage. All depends on how quickly .29 stabilises. Version numbers are so hand-wavy anyway. I wish when people asked me ‘what version is fX going to be’, they’d really ask ‘is feature xyz going to be merged by fX’. But people sure are hung up on numbers.

People tend not to notice kernel features these days for the most part. Which in a way is a good thing. (means it’s working). Unless it’s something that gets a lot of press like “unified x86 architecture” “tickless kernel” “modesetting”. There are dozens of features every release, but people don’t really get excited about a lot of them, and for good reason. They’re mostly dull from a userspace programmer/end-user perspective.

Linux kernel 2.6.28 is a post from: codemonkey.org.uk

An annoying kernel packaging bug. The kernel rpm package creates an initramfs file that gets...