Tomorrow, I should have been getting on a plane, and flying for 19 hours to Japan for the Japanese Linux Symposium, and Kernel Summit. Unfortunately, I’ve had to cancel those plans, and won’t be attending either event due to some unexpected health problems.

A few weeks ago, for no obvious reason, I went blind for 30 minutes. During this time, what I saw was a very distorted version of the world. The stripes on the crosswalk in front of me appeared as animated swirling zig-zag shapes for example. Needless to say, it freaked me the hell out. I stood still in downtown Boston for a half hour, too afraid to move, in case I stepped out in front of a bus or something. Slowly, my vision began to return, followed by the worst headache I’ve ever had in my life. Some reading later made me realise the symptoms were not entirely unlike a migraine. I thought I’d had migraines in the past, but compared to this, they were just ‘bad headaches’.

I went home, took some painkillers, and slept. For nearly 18 hours. That headache has been with me almost constantly the last month. Most days I wake up and the whole side of my head is numb and tingling. Constant nausea. Tinnitus. Dizzyness. Photosensitivity. Excessive tiredness. I’ve actually lost track of all the various symptoms I’ve experienced the last few weeks.

I hadn’t had an eye test in nearly 10 years, so I figured that would be a good place to start. I had one last week, and was told my vision was almost perfect, and I didn’t need anything corrective. The only thing of concern, was that apparently my right pupil is larger than my left. Which can be a sign of absolutely nothing to worry about, or it could be a sign of neurological issues.

I had also scheduled a visit to my doctor, which happened two days later. After telling him the various symptoms and what the eye doctor had told me, he was sufficiently concerned that he decided I needed my head checked out. So later this week, I have a CT scan to look forward to.

So that’s why I won’t be in Japan. Right now, I’m hoping it does just turn out to be ‘just migraines’, but I’m really bummed to be missing kernel summit. It’ll be the first one I’ve missed since 2000.

I’m MC’ing the boot/init miniconf next week at the Linux plumbers conference in Portland, and a slot has become available that I don’t have anyone to fill.

If you’re going to be there anyway, and you have something to talk about that may be relevant (other talks lined up include dracut & upstart) then let me know, and I’ll get you added to the speakers list.

My current 10 year british passport was due to expire next February. Given that I’ll be needing to do another US Visa renewal at some point in the next 6 months, I figured I’d get a new one a little early. (They add on the remaining time to the new passport).
If I was still living in the UK, the whole process would have been fairly trivial. For people living outside the UK however, the experience was frustrating to say the least.

My options were..

1. Fly back to the UK, apply for new passport, wait weeks for them to get around to it. (Suboptimal due to the ‘being away from home for a long time’ thing).

2. Mail my passport to some facility in Washington, DC along with an extortionate fee, where it would be kept for ‘a few weeks’, and then returned to me along with a new passport. (This option just gave me the creeps for unexplained reasons).
3. Fly back to the UK, and do a ’same day’ visit to the passport office in London, and pay extortionate fees. (This is the option I ended up taking).

So the same day passport thing.. First you need to make an appointment. If you’re living outside the UK, this is the first hurdle you fall at. Someone in their wisdom decided to make this a free number that can’t be called from overseas. Thanks to an intermediary, several round-trips, I got an appointment made. Well, sort of. You can’t make an appointment more than 2 weeks ahead of time, and I had a specific set of dates in mind, to coincide with a trip back to the UK for other reasons. So my intermediary called back 2 weeks ahead of time, and was told to pass on a list of things I would need to bring.

• A reference number.

• An invitation letter
• 2 bills as proof of my address in the US
• A letter from my intermediary saying that I was staying with him while I was in the UK
• 2 of his bills to prove he lives at that address.
• A filled out web form available at http blahblahblah

All straight forward enough I guess. I hit up the web form. “If you are traveling in the next 4 weeks, don’t use this form, go see the British consulate”. Sigh.

So I made my way to the British consulate in Boston. Only to find they’d already closed for the day. At 12.30. Turns out they’re only open for 3 hours each day. So after the weekend, I went back again, earlier this time. I enter the building, and the place is deserted. There’s no receptionist. Or any means to indicate “HEY THERE’S SOMEONE HERE WHO WANTS TO SEE SOMEONE”. So you pace around waiting, hoping that someone will eventually come out of one of the locked doors. 20 minutes pass, and someone finally appears. I explain my predicament, and I get a blank look that just tells me how the rest of the conversation is going to go.
“We don’t do anything with passports. We can’t help you” was the short version. I left there with the helpful advice of “just turn up at the interview and explain”. Having experienced dealing with such institutions when I lived in the UK, I wasn’t particularly hopeful with this strategy.

So I arrived in London a day early, and hit up a post office, on the off-chance I could get a form from them, and it would be the right form for my situation. (Short answer: It was). The next day I make my way to the passport office. I arrive a good 30 minutes early just in case. Only to find that they won’t even let you into the building more than 10 minutes before your appointment. So I waited outside. (Tip for anyone who stumbles across this who might also want to get a new passport: Get an afternoon appointment, there’s a pub across the road, which will at least make the waiting more bearable. There’s not a great deal else to do nearby).

Once I’d been let in, I went through an xray scanner, and then saw someone at a desk to get issued a number to wait for. The conversation went something like this ..

“Appointment letter”
“I didn’t get sent one. But it’s ok, I have the reference number”
“But you need a letter”
“Yes, but you didn’t send me one”
“Without a letter, you can’t see anyone”
“But the letter won’t tell you anything other than my reference number”
(variations on the above until finally he reluctantly agreed to take my reference number)
“ok, my number is 4, 8, 7, ..”
“whoa, whoa, it’s supposed to be 4, 8, 9, ..”
“well it isn’t, and this is the number I was told”
“But it’s supposed to have a 9 in it”
“Well, mine doesn’t. It’s 4, 8, 7, …”
“it’s not going to work without a 9 in it, it’s the wrong number”
“well try it anyway”
“It won’t work”
“Please, try it.”
“*sigh*”
“4, 8, 7, …”
(My name appears on the screen as if by magic)

Once I had my magic ticket, I waited upstairs for a few minutes before being called, where I was interviewed by someone who never asked for any of the bills or other paperwork that was requested. Instead, she was more concerned with knowing “what do I have to do to live in america?”. She then spent a while staring at my photographs for my new passport, looking at me, squinting at the photographs, looking at me, and coming up with lines like “I don’t think the background is white enough” “your eyes don’t look right” “how long ago were these photographs taken?” “I think the computer is going to reject them”. She seemed to be pushing to try and get me to use their photo booth to go take more photographs, without even having tried the ones I submitted. “try them anyway. if they don’t work, THEN I’ll get new photographs” “*sigh*”.

I made a point about them not slicing through the visa pages on my old passport. They showed complete indifference, which made me worried. At no point was I asked for the bills, or proof of address documentation.

I paid my extortionate fee, and was told to come back in four hours. After getting some lunch, and wandering around for a few hours, I returned, and my old passport still had its visa intact and I had a new passport waiting for collection, complete with biometric bullshit.
They had a machine that you can use to scan the rfid chip with. It has this blurb about how it doesn’t contain anything other than what the computer shows you. Just how trusting do they expect people to be? (Or maybe I’m overly cynical).

Since my earlier post on why VAIO’s suck, I’ve noticed a number of hits from people googling for how to enable hardware virtualization on Sony machines.
I’ve not tried this myself yet, but this guy seems to have had some success using an EFI application on a usb stick.

Create Digital Music has an interesting post up today by composer/musician Kim Cascone in which he details how he switched both his studio setup and his live performance machine from a Mac to a netbook running Ubuntu.

It’s always good to see success stories, though there are a number of takeaway points from his write-up. One obvious thing is that basically no distro gets JACK integration ‘just working’ out of the box. I’m not advocating it be started always (except maybe in a ‘multimedia production’ spin of Fedora maybe. After all, not every user is going to be an audio content creator. Regardless, a big step forward would be having it start on demand when programs that need it get launched for example.

Then there’s the real time kernel. I’ve back and forth’d in email with Peter Kirn CDM’s editor quite a bit recently about the rt kernel. It’s still a fairly large diff outside of the mainline kernel. This has made it hard to integrate into Fedora. We try to avoid packaging additional flavour of the kernel where we can avoid it, just due to the increased workload each build causes (increased buildtime, increased space on mirrors, increased bugs…) Also, it occasionally lags behind mainline by a few weeks, which means it’s not really feasible for rawhide.
Hopefully the -rt maintainers have a strategy for eventually getting all the remaining parts merged, but I don’t think it’s going to happen any time soon. Until then, people who need -rt will find themselves either patching it in themselves, and foregoing kernel support from their distro, or choosing a distro which chooses to package it.

Anyway, I found it an interesting read, and was expecting it to be a lot more “linux fails” than it was, and ended up being pleasantly surprised.

I’ve a number of other thoughts on Linux as a music creation platform, which I’ll go into another time, as this post is starting to ramble.

The majority of modern CPUs that support CPU scaling now use a common driver (acpi-cpufreq). Judging by the search queries that hit my blog, and the amount of mail I get on the subject, there is a failure mode of this driver that many people are hitting, that there isn’t a great deal of information on.

The failure mode looks like this:

$ modprobe acpi-cpufreq
FATAL: Error inserting (/lib/modules/…/kernel/arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.ko): No such device

Not particularly informative. We don’t spit out anything helpful to dmesg either. So what is the cause of this problem?
In many cases, /proc/cpuinfo shows the cpu supports speedstep (the ‘est’ flag). The answer in nearly all of these cases is.. The BIOS. The ACPI tables in the BIOS list which P-states a particular CPU supports. If your CPU was manufactured after your BIOS was written, you’re probably going to be out of luck. Sometimes, there are BIOS updates on the motherboard manufacturers website that will add support for newer processors. Sometimes we aren’t so lucky. In these cases, we’re out of luck, there’s nothing we can do.

There is another possibility for the error message above: kernel bugs. We have introduced bugs in the ACPI interpretor in the past which have broken parsing of the P-states on some platforms. These kinds of bugs tend to get noticed very quickly, and fixed in equally short time, but it’s worth making a point that it’s important to be running on the last kernel version before reporting bugs.

The x86-32 Fedora kernel comes in two flavours. ‘normal’ and ‘PAE’. (There are -debug flavours of both of these two, but ignore those for the purpose of this blog post). The common wisdom surrounding ‘which kernel do I use’ has for the longest time been “How much RAM do you have?”

The kernel offers several configuration options to satisy this question. If you have less than 1G of ram, you don’t need anything special. After 1G, you need to enable CONFIG_HIGHMEM4G. (In actuality, it’s something like 960MB). As the name suggests, this allows you to run kernels that can see up to 4G of RAM. It does this by mapping in/out ‘highmem’ pages (those after the 1G mark) as they are needed. The ‘normal’ kernel for Fedora is HIGHMEM4G.

This sounds like all you’d ever need. Who has more than 4G of RAM ? Even today, typically only servers. However, that mapping/unmapping comes with some overhead, and if you’re using a lot of highmem, you may be better off with the third choice..

Enter PAE (Page Address Extensions). This option is enabled by the kernel config option CONFIG_HIGHMEM64G. Its name tellingly indicates it can support up to 64G of RAM in theory. I say in theory, because there are still a number of really nasty cases that need to be taken care of (Like: the pagetables need to fit in non-highmem memory, and by filling up lowmem with pagetables, you don’t leave much for other purposes). After a point, it just makes more sense to go to x86-64. PAE also gives us the ability to do hardware NX due to it having a wider PTE (page table entry) format.

But there’s another case where PAE is desirable which seems to catch out some people.
If you have exactly 4G of RAM, you may not be able to address it all, even with a CONFIG_HIGHMEM4G kernel.

PCI devices need some physical address space. How much, depends on what PCI devices are present, and how they are configured. This address space needs to live in the lower 4G of memory. So the BIOS carves out a hole in the memory map and says “PCI lives here”. If you had 4G of RAM, you may find the memory map now looks like..

3G RAM : 1G PCI ‘hole’ : 1G RAM

See the problem? To address that top 1G of RAM, we need to use addresses past 4G. And to use those, we need PAE.

This past weekend I’ve been thinking of reviving an idea that has come up countless times. Producing RPM builds of the rawhide kernel for our already released Fedoras. The reason for not doing them so far has come down to bandwidth. (in terms of build system throughput, disk space, mirroring, and people bandwidth).

What I’m toying with doing is some devel kernels for Fedora 11 that are built outside of the Fedora build system. The Fedora kernel team now has enough build bandwidth for x86-[64] that we can actually get builds for those architectures done faster than koji.

Disk space – I’m thinking of just keeping the last 2-3 builds available.

Mirroring – Instead of having these be part of Fedora proper, I think an external repo on something like fedorapeople.org will suffice.

Which just leaves people bandwidth. For the most part the work is going to be just regularly syncing the devel/ branch with a CVS branch of F-11/ For some of this work, some scripting could be done to alleviate some of the pain. Also the frequency at which we push out these builds will determine the pain point. Perhaps every -git isn’t particularly valuable anyway. One build every handful of -git’s should be sufficient for bisecting.

There does remain one additional barrier. Occasionally we introduce something in rawhide builds which just won’t work on F11. For example, the kernel modesetting patches are tied closely to Xorg packages. Sometimes upstream changes require changes in mkinitrd or udev or some other ‘plumbing’. Some of these are regressions, and hopefully by identifying them sooner we can get them reverted/fixed upstream. Sometimes however, things get deprecated, and we need to change these packages. I’m not sure how to cope with this yet in a devel-for-F11 scenario.

One other thing that might be fun to throw into this would be the generation of -vanilla packages. The only reasons we don’t do these as part of the regular kernel builds is the various bandwidth concerns above. The specfile copes with spitting out RPMs with very little work needed. Josh Boyer has been occasionally doing these builds, though there hasn’t been a huge uptake. It’s unclear if this is due to lack of interest, or just a lack of publicity.

Another question to be answered is whether we go the route of enabling debugging in all builds as we do in rawhide, or do separate -debug builds. I’m leaning towards the latter.

I’m not committing anything to this for sure just yet, but it’s something I’ve been giving quite a bit of thought. There are still a bunch of unanswered questions.

I’m not much a book reader. I often tend to start books, and never finish them. Occasionally I’ll pick them back up with the aim of finishing them, but end up starting over, and then abandoning them again at usually the same point. It’s actually been quite a while since I found a book that I couldn’t stop reading until I finished it. Recently, I found one such book: Malcolm Gladwell’s “The tipping point”.

The book explains why some ideas ‘tip’ or become really successful, while others never get off the ground. A lot of the book explained psychological reasons for why people behave the way we do in certain scenarios. From well known theories such as group think, to defining types of people, and the roles they play in the dissemination of ideas. I found Gladwells categorisation of people into connectors/mavens/salesmen intriguing, and spent a while thinking of various people I know and trying to categorise them accordingly. (It’s possible for someone to be in more than one category).

There’s also a lot of anecdotes in there to back up most of his points, coming from some wide and varied scenarios (The studies done by the Sesame Street researchers in order to create the perfect ’sticky’ educational TV show for example). In some cases they do drag on a bit. In some cases there are multiple examples where one would have sufficed, but overall, it wasn’t tedious whilst hammering home the point.

My only gripe with the book was that there was no mention of the opposite scenario. There’s lots of examples of successful ideas that ‘tipped’, and some ideas that didn’t, but there are no examples or dissection of “Why don’t bad ideas die?”. Some ideas no matter how many times they get shot down seem to bubble back up to the surface every so often. I have some of my own theories why these zombie ideas never go away, but I would have liked to have read the authors take on it..

Anyway, rambling… Good book. Recommended. Wikipedia also has a pretty decent summary of all the points covered in the book.

With the release of Fedora 11 recently, we are now back in the position of looking onwards to the next release. F11 released with a 2.6.29 kernel, and we’re already looking at doing a .30 rebase for it soon. (I was hesitant to type that, because the last time I blogged about doing a rebase, we hit some troubles and ended up skipping the 2.6.28 release for F10). While F11 was stabilising before release, devel/ had continued on, being rebased up to 2.6.30. Yesterday Kyle committed the the first rebase to get us back up to Linus’ tree of the day.

So we’re looking at 2.6.31 for F12. (With various conferences coming up over the next few months, it seems infeasible that .32 will land in time for F12’s release).

Other planned changes? We’ve talked about dropping the exec shield patch that we’ve been carrying since Fedora Core 1.
It’s a pain to have to keep carrying it, and rebasing it, and occasionally fixing it, just to add a poor emulation of a feature that has been in all CPUs for the last five years. The decision isn’t final yet, but it’s something that’s being considered.

Some of the other patches we’ve been carrying (like modesetting drm) are now starting to find their way upstream too, which is obviously a good thing. The only really big thing we’re still carrying that struggles to get upstream is utrace.