Fear and Loathing in Les Rambles
Steve's blog about stuff nowhere near Les Rambles
This weekend, we held a combined Debian Bug Squashing Party and Ubuntu Local Jam in Portland, OR. A big thank you to PuppetLabs for hosting!
Thanks to a brilliant insight from Kees Cook, we were able to give everyone access to their own pre-configured build environment as soon as they walked in the door by deploying schroot/sbuild instances in "the cloud" (in this case, Amazon EC2). Small blips with the mirrors notwithstanding, this worked out pretty well, and let people start to get their hands dirty as soon as they walked in the door instead of spending a lot of time up front doing the boring work of setting up a build environment. This was a big win for people who had never done a package build before, and I highly recommend it for future BSPs. You can read about the build environment setup in the Debian wiki, and details on setting up your own BSP cloud in Kees's blog.
(And the cloud instances were running Ubuntu 11.10 guests, with Debian unstable chroots - a perfect pairing for our joint Debian/Ubuntu event!)
So how did this curious foray into a combined Ubuntu/Debian event go? Not too shabby:
- Roughly 25 people participated in the event - a pretty good turnout considering the short notice we gave. Thanks to everyone who turned up!
- Multiarch patches were submitted for 14 library packages by 9 distinct contributors
- Four of these people submitted their first patch to Debian!
- Three more contributors worked on patches that were not submitted to Debian by the end of the event, but we will stalk them and see to it that their patches make it in ;)
- 8 Ubuntu Stable Release Updates were looked at for verification of fixes
- 7 of these fixes were successfully verified (one bug was not reproducible)
- 6 of those packages have already been moved to the -updates pocket, where all of Ubuntu's users can now benefit from them
When all was said and done, we didn't get a chance to tackle any wheezy release critical bugs like we'd hoped. That's ok, that leaves us something to do for our next event, which will be bigger and even better than this one. Maybe even big enough to rival one of those crazy, all-weekend BSPs that they have in Germany...
Posted Mon Dec 5 18:41:57 2011Raphaël Hertzog recently announced
a new dpkg-buildflags interface in dpkg that at long last gives the
distribution, the package maintainers, and users the control they want over
the build flags used when building packages.
The announcement mail gives all the gory details about how to invoke
dpkg-buildflags in your build to be compliant; but the nice thing is, if
you're using dh(1) with debian/compat=9, debhelper does it for you
automatically so long as you're using a build system that it knows how to
pass compiler flags to.
So for the first time, /usr/share/doc/debhelper/examples/rules.tiny can now
be used as-is to provide a policy-compliant package by default (setting
-g -O2 or -g -O0 for your build regardless of how debian/rules is
invoked).
Of course, none of my packages actually work that way; among other things I
have a habit of liberally sprinkling DEB_MAINT_CFLAGS_APPEND := -Wall
in my rules, and sometimes DEB_LDFLAGS_MAINT_APPEND := -Wl,-z,defs and
DEB_CFLAGS_MAINT_APPEND := $(shell getconf LFS_CFLAGS) as well. And my
upstreams' build systems rarely work 100% out of the box with dhauto*
without one override or another somewhere. So in practice, the shortest
debian/rules file in any of my packages seems to be 13 lines currently.
But that's 13 lines of almost 100% signal, unlike the bad old days of cut'n'pasted dh_* command lists.
The biggest benefit, though, isn't in making it shorter to write a rules file with the old, standard build options. The biggest benefit is that dpkg-buildflags now also outputs build-hardening compiler and linker flags by default on Debian. Specifically, using the new interface lets you pick up all of these hardening flags for free:
-fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security -Wl,-z,relro
It also lets you get -fPIE and -Wl,-z,now by adding this one line to
your debian/rules (assuming you're using dh(1) and compat 9):
export DEB_BUILD_MAINT_OPTIONS := hardening=+pie,+bindnow
Converting all my packages to use dh(1) has always been a long-term goal,
but some packages are easier to convert than others. This was the tipping
point for me, though. Even though debhelper compat level 9 isn't yet frozen,
meaning there might still be other behavior changes to it that will make more
work for me between now and release, over the past couple of weekends I've
been systematically converting all my packages to use it with dh. In
particular, pam and samba have been rebuilt to use the default hardening
flags, and openldap uses these flags plus PIE support. (Samba already
builds with PIE by default courtesy of upstream.)
You can't really make samba and openldap out on the graph, but they're there (with their rules files reduced by 50% or more).
I cannot overstate the significance of proactive hardening. There have been
a number of vulnerabilities over the past few years that have been thwarted
on Ubuntu because Ubuntu is using -fstack-protector by default. Debian has
a great security team that responds quickly to these issues as soon as
they're revealed, but we don't always get to find out about them before
they're already being exploited in the wild. In this respect, Debian has
lagged behind other distros.
With dpkg-buildflags, we now have the tools to correct this. It's just a matter of getting packages to use the new interfaces. If you're a maintainer of a security sensitive package (such as a network-facing daemon or a setuid application), please enable dpkg-buildflags in your package for wheezy! (Preferably with PIE as well.) And if you don't maintain security sensitive packages, you can still help out with the hardening release goal.
Posted Wed Oct 19 01:45:47 2011Ubuntu has had an explicit goal for several cycles now to achieve flicker-free booting: smooth graphical boot all the way through from bootloader to the desktop, with no messy modeswitches or cuts between text and graphics mode on the console. This is no small task, to be sure; even getting flicker-free transitions between plymouth and X for the KMS-capable video cards was a handful for 10.04 LTS.
So I was suitably impressed when in the course of testing for the upcoming Ubuntu 11.10 release I found that on a laptop with the nouveau driver, we had achieved 100% flicker-free boot all the way from grub to X. And then I was astonished to find the grub-kernel transition was flicker-free when using the binary nvidia driver too!
This is in large part thanks to Colin Watson's work upstream implementing VBE support in GRUB2, and Andy Whitcroft's adding proper support for a vesafb fallback when no KMS video driver is found. The architecture of how this all hangs together is now documented in the Ubuntu wiki.
The irony is that, while nouveau gives a perfect flicker-free boot, and the nvidia binary drivers give a boot with just one modeswitch, both of my Intel systems show two modeswitches on boot - one between grub and the kernel, another between the kernel and X. Something to work on cleaning up next month, I think...
Posted Sat Oct 8 14:31:31 2011On the bus heading to the airport to go to DebConf 11, I chanced to sit down beside an older Croatian gentleman. If I've ever met another Croatian living in Portland before now, they did not have occasion to make themselves known to me.
He spoke highly of Tito at some length, and of Tito's success at holding the Russians at bay in the 1960s when the rest of the Communist block was being held there by the thumb of the Russian military. If I've ever met another person living anywhere who approved of Tito, they've not had occasion to make that opinion known to me either.
Wacky experiences like that are why I love Portland's mass transit network.
Posted Tue Jul 19 13:27:20 2011So the other day, I was able to do this in an Ubuntu natty amd64 chroot for the first time.
# cat > /etc/apt/apt.conf.d/multiarch-me
APT::Architectures { "amd64"; "i386"; };
^D
# cat >> /etc/dpkg/dpkg.cfg
foreign-architecture i386
^D
# apt-get update
# apt-get install flashplugin-installer:i386
Reading package lists... Done
Building dependency tree
Reading state information... Done
[...]
The following NEW packages will be installed:
flashplugin-installer:i386 flashplugin-nonfree:i386 gcc-4.5-base:i386
libatk1.0-0:i386 libavahi-client3:i386 libavahi-common-data:i386
libavahi-common3:i386 libc6:i386 libcairo2:i386 libcomerr2:i386
libcups2:i386 libdatrie1:i386 libdbus-1-3:i386 libdrm2:i386
libegl1-mesa:i386 libexpat1:i386 libfontconfig1:i386 libfreetype6:i386
libgcc1:i386 libgcrypt11:i386 libgdk-pixbuf2.0-0:i386 libgl1-mesa-glx:i386
libglib2.0-0:i386 libgnutls26:i386 libgpg-error0:i386 libgssapi-krb5-2:i386
libgtk2.0-0:i386 libgtk2.0-common libice6:i386 libjasper1:i386
libjpeg62:i386 libk5crypto3:i386 libkeyutils1:i386 libkrb5-3:i386
libkrb5support0:i386 libnspr4:i386 libnspr4-0d:i386 libnss3:i386
libnss3-1d:i386 libpango1.0-0:i386 libpcre3:i386 libpixman-1-0:i386
libpng12-0:i386 libselinux1:i386 libsm6:i386 libsqlite3-0:i386
libstdc++6:i386 libtasn1-3:i386 libthai-data libthai0:i386 libtiff4:i386
libudev0:i386 libuuid1:i386 libx11-6:i386 libx11-data libx11-xcb1:i386
libxau6:i386 libxcb-dri2-0:i386 libxcb-render0:i386 libxcb-shape0:i386
libxcb-shm0:i386 libxcb-xfixes0:i386 libxcb1:i386 libxcomposite1:i386
libxcursor1:i386 libxdamage1:i386 libxdmcp6:i386 libxext6:i386
libxfixes3:i386 libxft2:i386 libxi6:i386 libxinerama1:i386 libxrandr2:i386
libxrender1:i386 libxt6:i386 libxxf86vm1:i386 x11-common zlib1g:i386
0 upgraded, 78 newly installed, 0 to remove and 3 not upgraded.
Need to get 15.1 MB/15.6 MB of archives.
After this operation, 48.9 MB of additional disk space will be used.
Do you want to continue [Y/n]?
It is a truly heady experience, after so many years of talking about the need to properly support multiarch in Debian and Ubuntu, to see support for cross-installation of packages come to fruition. If you've talked to me any time in the past couple of weeks and noticed it's a little hard to get me to change the subject, well, that's why.
Many who have grown accustomed to Debian and Ubuntu's lack of support for installing i386 packages on amd64 (or vice versa) may wonder what the fuss is about. (Whereas others who are well versed in distributions such as Red Hat and SuSE may laugh and wonder what took us so long...) So maybe a few words of explanation are in order.
If you've ever installed ia32-libs on an amd64 machine anywhere; if you've ever noticed a bug where ia32-libs didn't work right because of wrong system paths, or had to file a request for another library to be added to ia32-libs because it wasn't included in the set of libraries Debian decided to package up in this grotesque, all-in-one 32-bit compatibility bundle; if you've ever decided not to install a 64-bit OS on your perfectly 64-bit-capable hardware because of concern that you wouldn't be able to run $random32-bitonly_application; multiarch is for you.
If you've gotten stuck maintaining a lib32foo "biarch" package in Debian due to popular demand, multiarch is definitely for you. :)
If you've ever cross-compiled software for a different Debian architecture than the one you were running, multiarch is for you.
If you've ever wanted to run binaries for a different architecture under emulation, and found it awkward to set up the library dependencies, multiarch is for you, too.
Because although the .deb world may be a little late to the party, we're also naturally taking things much further than anyone's done with rpm. Multiarch won't just give you the ability to install 32-bit libs on 64-bit systems; it'll give you the ability to install libs for any known architecture on any system. And a whole lot of pain just falls out of the equation in the process. A cross-compiling environment looks the same as a native-compiling environment. An emulated system looks the same as a native system. We can start to seriously consider cross-grading systems from one architecture to another.
And all this is happening now. The groundwork is there in Ubuntu natty. Wheezy will be the release that brings multiarch to Debian. When dpkg 1.16.0 is uploaded to unstable real soon now, the bootstrapping will begin.
I am immensely grateful to everyone who's helped make multiarch a reality - to Tollef, Matt and others for seeding the vision; Aurelien, Matthias and Arthur for their work to ready the toolchain; David and Michael for the apt implementation; Guillem and Raphael for the dpkg implementation, and Linaro's support to help make this possible; and the many other developers who've helped to refine this design over the years in numerous other BoFs, sessions and mailing list threads.
I'm excited to find out what the Debian community will do with multiarch now that it's upon us. Christian, maybe you should start a pool for how long it will take before all the libraries shipped in ia32-libs have been converted to multiarch and we can drop ia32-libs from the archive?
Posted Tue Mar 29 00:28:39 2011Paul Wise called attention recently to the fact that there's renewed activity around multiarch in the face of the squeeze release. Although I fervently wish we could have gotten multiarch support into squeeze, it's better late than never; and real progress is being made now, thanks mostly to the efforts of wonderful package management experts like Raphael Hertzog, Guillem Jover, and David Kalnischkies.
Don't believe it's finally coming? Here's some output from my amd64 multiarch chroot:
$ dpkg -l '*:i386'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-==============-==============-============================================
ii gcc-4.5-base 4.5.2-3ubuntu2 The GNU Compiler Collection (base package)
ii libc6 2.13-0ubuntu1+ Embedded GNU C Library: Shared libraries
ii libdb4.8 4.8.30-5multia Berkeley v4.8 Database Libraries [runtime]
ii libgcc1 1:4.5.2-3ubunt GCC support library
in libpam-modules <none> (no description available)
ii libpam0g 1.1.2-2ubuntu1 Pluggable Authentication Modules library
ii libselinux1 2.0.96-1multia SELinux runtime shared libraries
ii libstdc++6 4.5.2-3ubuntu2 The GNU Standard C++ Library v3
iU libuuid1 2.17.2-9.1ubun Universally Unique ID library
ii selinux-utils 2.0.96-1multia SELinux utility programs
ii zlib1g 1:1.2.3.4.dfsg compression library - runtime
If you're as excited about this making its way into Debian as I am, you might be wondering what you can do to help. That's great, because I'd love to tell you! Perhaps you happen to maintain a package that provides executables; and perhaps other packages depend on your package exclusively to use those executables; and perhaps some of those packages depending on your package are shared libraries.
If all of the above is true, please consider declaring this package to be
Multi-Arch: foreign as defined
here.
Note: it is very important that you not use this on any package that will provide architecture-dependent interfaces to its reverse-dependencies. So if you're shipping both shared libraries and executables in the same binary package, don't do this.
But if your package does fit this description, this is one part of multiarch that you can safely start implementing today. This will go a long way towards getting the system ready to handle multiarch libraries when they land.
My own efforts to help with getting Multi-Arch: foreign packages documented in the archive are trackable here.
Posted Sun Feb 20 18:37:12 2011Buying a ThinkPad?
Don't buy one with this wireless chipset. "ThinkPad wireless" is apparently code for "poorly supported RealTek chip".
If you do end up with one of these chips (perhaps you failed to notice the addition of a new option on the website before the time you spec'ed out your new laptop and the time you ordered it; perhaps you assumed it was iwlagn by default; who can say?), you may be treated to such issues as a lack of rfkill interface support, a driver that needs to be hard reset about once an hour to continue to be able to see APs, a kernel panic if you try to run powertop... or maybe a mysterious bug that prevents the wireless card from transmitting packets to a VoIP server, with 100% reliability.
And after you wear yourself out from laughing at the Lenovo sales rep's suggestion that you could pay a 15% restocking fee and reorder another laptop that costs $40 more to get the Intel Centrino option when that same Centrino card can be found on-line for $40 by itself, you might decide to order an Intel Centrino Ultimate-N 6300 from a third party, the same card that's advertised on Lenovo's website as an option for the X201, and install it yourself.
And that's when the fun would start.
Did you know that it's common practice for laptop OEMs to hard-code a whitelist of authorized wifi (and 3G) cards in the BIOS, and refuse to boot with an unrecognized card - nominally to ensure no one can combine a wireless adapter with their hardware antennas to create a transmission device that violates FCC regulations? And that these same OEMs may have different PCI subsystem IDs for the cards they ship than are used for the ones you can buy retail from third parties? I did not know that. Apparently some people knew that because they had tried this before, but I missed that memo, having never before needed to combine a laptop and a wifi card from different vendors.
Here, gentle reader, is a memo.
You're welcome.
Hacking the ThinkPad BIOS
So although this feature was news to me, it was not news to the Internet at large. A quick search on the BIOS error message quickly turned up a relevant description of the problem, as well as pointers to some options for fixing (in the veterinary sense) the whitelist checking in the ThinkPad BIOS. It's great to be part of such a large community of ThinkPad-using Free Software hackers looking out for each other (and not taking "no" for an answer from their BIOS)!
But wait, none of the methods described on that page for modifying your BIOS (or bypassing the check by moving the card to a different slot, or taping over one of the pins, or...) work on the X201! As Matthew summarized it, things are more difficult for "anything with three digits or starting with a six". Fortunately, another hint from Matthew took me to the website of a guy who's been providing fixes for this BIOS issue for the past couple of years.
Hmm, but the X201 is a bleeding-edge new model and isn't on his list of models that he's patched the BIOS for. Well, fortunately there's a step-by-step howto here that I can follow here to do it myself... grab the latest BIOS image from the vendor's website; decompress the image with phcomp.exe (oh Wine, you pair well with more than cheese); unpack the individual BIOS modules with phnxsplit; run phnxpatch to apply the binary patches, finding that the only one that applies is "wwan1a" when the issue is not with a WWAN card at all, so hand-hack the whitelist in the binary instead; use the fi.exe and fp.exe tools to add the necessary header back onto the BIOS module (after deciphering the example batch script accompanying the BIOS update and translating it into a reusable script in a real scripting language); use phnxmod to inject the newly-packaged BIOS module into the original image; call phnxcksm to update the BIOS's internal checksum; ...
Uh? Phnxcksum fails, saying that there's no extended checksum field. Just how deep is this rabbit hole?
How much was that restocking fee again, and how much is my time worth?
But no; now it's a matter of principle. Besides, I'd rather spend my time fighting with a BIOS than having to reinstall the OS on another new machine. Which is to say that I'd rather beg the local reverse engineering experts for their time fighting with a BIOS.
So now we have a BIOS image whose code we're confident has been correctly patched to bypass the whitelist, but we can't find the checksum field that our patching tools expect. Maybe we can flash this to the BIOS as is? No such luck; sure enough, we get a checksum error. Ah, but WinPhlash gives us a logfile, and shows that the failure is a per-module checksum mismatch. So where is that checksum?
Oh, there it is; it's the byte labeled "data_checksum" in the module header description in phnxsplit.h, which phnxmod doesn't copy when it splices an updated module into the BIOS image. Well, could it really be as easy as updating a single byte? Yes, the BIOS utility takes it and flashes it without complaint... and the system boots afterwards!
Thanks to Paul and Matthew for their great work documenting this problem in general; Zender, for tools that provide a 99% solution; and Kees and Jesse for a dazzling display of impromptu reverse-engineering.
Now, to figure out how to get these tools packaged and into Debian and Ubuntu.
Posted from my Centrino-wireless-using Lenovo X201 laptop.
Posted Tue Oct 19 16:37:02 2010Robert Collins did a very nice writeup not so long ago of creating a Debian package with bzr-builddeb, but he starts from the assumption that upstream also uses bzr. As we all know, there are many projects that have their upstream sources in a VCS, but don't use bzr. Can we still get reasonable results from bzr-builddeb, including the ability to merge new upstream releases direct from the VCS, if, say, upstream is using git?
Yes, we can!
Of course, if you're happy with using git as the VCS for your Debian packaging already, there's probably no compelling reason for you to switch to bzr. But if you prefer to use bzr and have been frustrated at having to choose between being able to use the VCS client of your choice for your packages and being able to use your VCS client to access upstream revision history, read on.
Thanks to the fine work of Jelmer Vernooij and friends, bzr-git has been
usable for a while if you just want to track the default git branch. But
what if you care about tracking multiple upstream branches?
Enter bzr git-import, which we'll use to set up a bzr repo from scratch for
our cifs-utils package, with a full import of all the branches of upstream's
git tree.
First we do some work to set up a shared bzr repository. As long as we're going to the effort of mirroring all the git branches, we probably want to publish them for other people to use, so we make sure our bzr repository is reasonably configured for sharing data between branches:
$ bzr init-repo --no-trees cifs-utils.server
Shared repository (format: 2a)
Location:
shared repository: cifs-utils.server
$ cd cifs-utils.server
Then we use bzr git-import, provided by the bzr-git plugin, to do a full
import of the upstream branches into a cleverly named subdirectory:
$ bzr git-import git://git.samba.org/cifs-utils.git upstream
[/ ] Counting objects: 181, done.
Now we create our own local 'trunk' for the package, by branching from the upstream tag matching the release version we're going to package.
$ bzr branch -r tag:cifs-utils-4.0rc1 upstream/HEAD trunk
Branched 42 revision(s).
Do a little more prep work for future branches...
$ mkdir branches
And that's it. Now we have a bzr repo locally that we can push out to our
hosting server of choice (N.B.: we could do this all directly on the server,
but alioth doesn't currently have bzr-git installed):
$ rsync -az . alioth.debian.org:/bzr/pkg-samba/cifs-utils/
$
Now that we have a bzr repository, it's time to get ourselves a working directory and do some packaging. We're probably going to work with multiple branches locally as well, so we create another shared repository for local use, and get a copy of the trunk branch we created before.
$ bzr init-repo cifs-utils
Shared repository with trees (format: 2a)
Location:
shared repository: cifs-utils
$ cd cifs-utils
$ bzr co bzr+ssh://bzr.debian.org/bzr/pkg-samba/cifs-utils/trunk
Now we grab the upstream tarball, and whip up some quick packaging with (what else?) debhelper 7:
$ wget ftp://ftp.samba.org/pub/samba/cifs-utils/cifs-utils-4.0rc1.tar.bz2
$ ln -s cifs-utils-4.0rc1.tar.bz2 cifs-utils_4.0~rc1.orig.tar.bz2
$ cd trunk
$ mkdir -p debian/source
$ echo '3.0 (quilt)' > debian/source/format
$ echo 7 > debian/compat
$ cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
$ dch --package cifs-utils --versio 4.0~rc1-1 --create 'Initial package'
Create debian/control by hand, generate an initial source package, and import
it into bzr with bzr-builddeb.
$ debuild -uc -us -S -i
$ rm -r debian
$ bzr import-dsc ../*.dsc
$
Since we want to continue tracking upstream development, we need to periodically sync our bzr import of the git tree.
$ bzr git-import git://git.samba.org/cifs-utils.git bzr+ssh://bzr.debian.org/bzr/pkg-samba/cifs-utils/upstream
$
And when we find a new upstream version in that import that we want to
package, bzr-builddeb makes this a snap, too.
$ cd cifs-utils
$ wget ftp://ftp.samba.org/pub/samba/cifs-utils/cifs-utils-$version.tar.bz2
$ cd trunk
$ bzr merge-upstream --v3 --version $version_with_epoch \
../cifs-utils-$version.tar.bz2 -r tag:cifs-utils-$version \
bzr+ssh://bzr.debian.org/bzr/pkg-samba/cifs-utils/upstream/HEAD
No distribution specified, and no changelog, assuming 'debian'
Committing to: /tmp/tmpbzOVMs/upstream/
Committed revision 2.
All changes applied successfully.
The new upstream version has been imported.
You should now review the changes and then commit.
$ bzr diff
<review changes>
$ bzr commit -m "merge upstream $version"
Committing to: bzr+ssh://bzr.debian.org/bzr/pkg-samba/cifs-utils/trunk
modified debian/changelog
Committed revision 50.
$
And that's it! It's exciting to see signs of real interoperability between DVCSes at last. Thanks to everyone who's helped make it possible!
Posted Tue Jun 1 13:39:56 2010I am very glad to know that the Debian train was not hit by a bus on its way back to Madrid.
Heat + a head cold are not a good mix. But after a night of fitful sleep, I seem to be doing better and may actually make it out to see some things in the city today (instead of wandering out for dinner, whining at the heat and immediately returning to my room). I'm sorry to not have been able to meet up with people yesterday - apparently Madrid is swarming with DDs right now - but there's always next year in New York!
Tentative plans for the day: this, then this.
Posted Sat Aug 1 01:53:04 2009Am I going to DebConf9? Yes.
Do I know how I'm getting there? No.
$ TZ=Europe/Madrid date
Tue Jun 16 23:50:43 UTC 2009
FF CC HH de PP.
Posted Tue Jun 16 17:11:31 2009