Fear and Loathing in Les Rambles

Steve's blog about stuff nowhere near Les Rambles

RSS

Like Jaldhar, I too am thrilled at the news that McCain has chosen Palin as his running mate. Finally, a candidate who embodies the truly American spirit of the Republican party!

our future vice-president!

Haven't we had enough political circus in Washington? Isn't it time to give Flying Circus a chance?

Posted Tue Sep 2 15:43:31 2008

I'm back (for several days now) from Argentina and DebConf 8 to Portland. Too long a journey and too short a trip, as always. It was great to see everybody again as well as to meet a number of new folks from the local community. Here's hoping that DebConf 8 inspires more involvement in Debian from Argentina!

It was my first time to this wonderful South American country, which made it even more melancholy to have to leave after such a short time and to be making this trip without Patty.

Random observations about Argentina, for posterity:

The 6-hour bus ride from Buenos Aires to Mar del Plata looks an awful lot like Iowa. Except for the occasional palm tree and the fact that it's 20°C in winter.
Well, ok... Iowa also tends not to have quite so many cows. Or country grills (parrillas) right next to pastures...
All the towns along the route are named after generals or colonels?
All the police on the highway wear OSHA green vests.
Following the German with the voluminous guidebook is an effective way to acquire generous portions of bife de chorizo.
People from Bahía Blanca seem to have a somewhat more paranoid opinion of what constitutes a "dangerous neighborhood" than I do. :)
"Buenos Aires" appears, I'm sad to say, to be a marketing lie. There was nothing bueno about the aire on the autopista on the ride to the airport. But the air in Mar del Plata was quite nice, and the people were the best!

As always it was a pleasure to take part in another annual DebConf, an event which is such a positive force for rejuvenating the Debian community.

And BTW, Biella, NYC is not the only US city entertaining a bid for DebConf 10.

Posted Sat Aug 23 17:21:53 2008

Pierre,

"Why on earth is https with an untrusted certificate less secure than http ?."

Apparently you thought this was a rhetorical question. I disagree. https with an untrusted certificate is less secure than http, because whether or not most users understand the meaning of https, there are a significant number of users who do understand what it means, or at least think they do. You can't seriously tell me you that you think all users always look for the lock icon instead of looking at the URL to know whether they're protected, can you? Users, on the whole, are very good at cargo-culting where technology is concerned. Just because many users look for the icon does /not/ mean that there aren't other users who know just enough of the definition of "protocol" to be a danger to themselves.

An https URL is a declaration on the part of the server (or the party linking to it) that the resource in question should be secure. This creates an expectation on the part of those users who know what https is that if the connection succeeds, it is secure, which means they'll think it's ok to do the kinds of things that they normally do over secure connections but won't do over unsecured ones.

It therefore certainly is important to give users feedback that an https connection has failed. If you try to connect to an https resource, and your browser can't verify the certificate, something is wrong. Either the server operator is a stooge for Intel trying to drive up the client CPU requirements for web connectivity so that they can sell more chips, or the server operator has failed to establish a chain of trust to you in the appropriate manner, or someone really has compromised the connection with a man-in-the-middle attack. Any of these conditions are something that ought to be brought to the user's attention, not ignored.

If you don't like seeing cumbersome security warnings for insecure https connections, how about not using https when what you really want is http in the first place?

Posted Thu Jun 26 23:08:32 2008

One of the reasons I tend not to blog very much is that I don't have much to say to the Planet Debian audience (the only place I know my blog is syndicated) that hasn't been said a dozen times before. But the world's best release party (the one good enough to cross an ocean for) gave me food for thought, and for once I'm inspired in the true spirit of blogging to write even if nobody cares what I have to say. Even if it has taken me until two months after the release to write it...

Folks within the Debian community have been asking for years whether it's really still worthwhile for Debian to do stable releases, or if we're better off focusing on our "core competency" of testing and unstable and leaving stable releases to derivatives. For me, the answer to this question is an unqualified: yes, it's worthwhile.

Of course that's the answer you'd expect from a release manager. Who would want to believe after committing as much time as the release team does that their efforts had been wasted? But the investment I've made in the etch release wasn't done without reflection; I recognize that the level of involvement required to be a release manager has come at some personal cost to me, as it does for each member of the release team, and I have made the investment anyway because I believe in Debian's philosophy, its goals, and its community.

Even outside of a stable release, Debian packages really are great; in spite of the bugs and library transitions that we as release managers spend our time on in unstable and testing, and figures showing that within a month of the etch release we already had some 500 RC bugs identified for lenny, both unstable and testing do a great job of meeting the needs of many of our users. And package maintainers are constantly improving their packages in unstable, fixing bugs and adding new features. Unstable is a flowing river, vibrant and vital, never the same twice -- and only rarely are people dragged to their deaths by the undertow. ;)

Maintainers often don't get to choose the version of their package that goes into stable. When it comes to a stable release, the bug you know is quite often better than the bug you don't. But although the release team is charged with chasing out the release-critical issues in preparation of a stable release, it isn't just the release team who deserves credit for etch. Etch is a monument to the work of all Debian maintainers, who have each helped shape the release with their contributions. "Stable" doesn't mean "perfect", but it does mean "lasting": in etch we have not only created what I think is the highest-quality OS I've ever had the privilege to be involved in, we have created something permanent that will be burned to thousands or millions of disks, something that users will continue to use and install for years to come. Far from standing in opposition to unstable, it is a perfect complement, letting us share everything that's great about Debian with users for whom tracking a constantly-changing testing or unstable simply isn't an option.

I had decided well before etch was finished that it was time for me to move on from serving as release manager, but I still look forward to great things from Debian stable for lenny and releases beyond. With last weekend's release team meeting in Juelich, and the wonderful collaboration that's sure to take place this week at DebConf 7, I think we're off to a great start. The grumbling on IRC about broken packages and missing builds and uncoordinated library transitions is already in full swing for lenny -- and I'm no small contributor to that myself even now -- but we shouldn't let that blind us to what a great achievement etch has been for the Debian community, and what a great achievement lenny will be when it's released exactly on schedule next year.

Posted Sat Jun 16 08:49:47 2007

This is the first of my two formal reports to the Dunc-Tank board on the release management funding experiment. Although submitted to the board some time earlier, I've delayed publishing it to the wider community until a release update could be sent out with more concrete information on timeline adjustments as Debian developers are wont to demand.

I'm supposing that I don't need to give you any more specific details on what I've been working on than what's already been posted in my blog. There has certainly been plenty to keep me busy the past three weeks, including a lot of triage work on the state of bug reports in the BTS, managing the last of the major package transitions into testing for etch, removing packages and NMUing them, and it has indeed kept me busy. Busy enough that I've been lax when it comes to talking about it, either here or in my blog. :)

Has the hard work paid off? Well, http://bugs.debian.org/release-critical/ shows a sharp decline in release critical bugs affecting testing over the same time period, following the peak resulting from Dunc Bank's successful campaign of identifying latent release-critical bugs in etch. Unfortunately, we have no statistics that track the number of RC bugs closed or opened over time, which makes it hard to judge how large of a dent my own work has made in the overall bug count. It's also totally guesswork to try to judge whether Dunc-Tank has encouraged others to help with the release overall, or discouraged them from doing so. What I can say is that, even though the past month coincides both with the tail-end of the Dunc Bank mass bug filings and multiple archive-wide rebuild efforts, the bug count is moving in the right direction, and I am seeing many maintainers being responsive to bugs in their packages and a number of active NMUers. If not for the buzz-kill of Lucas Nussbaum's latest round of build failure reports :), we would be down from over 300 RC bugs to less than 200 right now, which probably represents at least twice that many RC bugs closed.

In terms of how the time I'm putting into this release compares with the sarge release, I'd say it's pretty comparable. That's actually quite a positive statement about Dunc Tank's role, IMHO; getting a Debian release out is uphill all the way, and whereas the sarge release came at a time when I was in a position to work more or less full time on it without compensation, my availability this time around would have been greatly reduced, and progress would have been much more heavily dependent on individual maintainers and semi-periodic bugsquashing parties, which I'm not sure is any longer enough to offset the rate at which new RC bugs are created in the distribution by uploaders.

As it stands, I do find myself trying to consciously avoid a hero complex of needing to fix every bug personally, but there are after all only so many options for getting bugs fixed for release. A lot of maintainers respond positively to gentle nudging about their RC bugs, but there are still bugs that the maintainers can't or don't fix and that no NMUers take on, which does leave it more or less up to the release managers to address -- which is why one of my tasks for this week is to try get the count of RC bugs over 20 days old down from 90 to around 20-30, as these are the bugs that really do need more attention to get them resolved.

So yes, Dunc-Tank is making a positive difference. Andi and I have reviewed the status, and while we don't think we'll be able to hit the Dec 4 deadline (even if we get the bug count down immediately we're waiting on the debian-installer to release, which needs at least one more release candidate with the 2.6.18 kernel), we do believe that a release before the end of the year is very possible and I'm looking forward to starting the full freeze in the next few weeks.

Posted Tue Nov 21 06:19:10 2006

While NMUing plays an important part in getting us to a release, as shown by the thousand or so fixed-in-NMU RC bugs that we recently cleaned up the BTS state on, it's evident after doing this whole release-managing thing for a while that there is a very high (non-linear) correlation between the age of an open RC bug and the likelihood that a release manager is going to have to touch it at some point in order to get it fixed. By and large, maintainers really do a good job of fixing their own packages, which is probably something we don't say enough - maybe 75% of all RC bugs, I'd guess, will (or would) get fixed by maintainers in the first week without anyone needing to poke them about it, and of the remainder, probably another 80-90% get taken care of by BSPers and serial-NMUers like bug-a-day mascot Steinar Gunderson. That's why I'm not bothered overly much by the idea of full-archive searches for hidden RC bugs, like Lucas's piuparts tests, affecting the release timeline, because I know that 95-98% of those bugs are going to be gone two weeks after they're filed anyway, especially when they're no-brainer fixes to maintainer scripts or dependencies.

But then that leaves the other 2-5%, which are the ones nobody else has fixed. And why hasn't anyone fixed them? Because they're the hard ones, of course...

So although I follow the debian-bugs-rc list and occasionally swat a few non-bugs closed to save the maintainers time, and also provide some nudges when it looks like a package is causing RC bugs to be filed on other packages, my goal for the last full week of my Dunc-Tank contract was to get the count of etch RC bugs unfixed in unstable that are 20 days old or older down from around 50 to under 30. This seemed a reasonable goal, since it had been at 80 the week before, and it would go a long way towards helping the real blockers on the release. Initially I was aiming to bring down the count of RC bugs unfixed in testing that are 20 days old or older to this number, but quickly realized that this wasn't going to be realistic because it depended too much on the autobuilders all operating optimally and there wasn't time to both help fix 20 of the hardest RC bugs and handhold 60 RC bugfixes into testing in one week.

In the end, we did miss this lesser target by a day or two. Still not a bad week's work, given that I'm pretty sure I fixed about 20 of these bugs myself in that time period (through NMUs, removals, and analysis+downgrades) and they seem to average 3 hours of work for a fix. And we've been holding at or below 30 on this list of bugs since then, so we're successfully keeping up with new bugs aging past the 20-day mark, which is also a good sign.

Unfortunately, I seem to have left myself to fix the alpha Xorg 7.1 bugs on my own time. :)

By now the announcement has made the rounds that the release team doesn't think we can hit the Dec. 4 target date because more time is needed for a final release candidate of the installer. Does that mean that RC bugs were the wrong thing to be focusing on for the past weeks? Well, no; if you want one place in Debian where Brooks' Law holds, I think you'll find that getting a d-i release out is it, and we still have a hard road ahead to make sure we do get our other RC issues sorted out in the same time frame. The installer just happens to put a lower limit on a release date, we still have an opportunity to screw it up from there. ;)

At least the statistics suggest that we're going in the right direction.

Only 13% of RC bugs fall into the "20 days or older and unfixed" category. Half of all RC bugs over 10 days old have a fix of some kind in unstable, and 64% of all reported RC bugs are less than 10 days old.

We'll see whether this graph backs me up on this in a couple more days.

Posted Fri Nov 17 02:33:03 2006

According to turmzimmer, there are 53 RC bugs remaining in testing that are not fixed in unstable and are over 20 days old. This is great, because that number is down from about 80 a week or so ago; but that leaves a good number of harder bugs open that could probably use some attention from NMUers given that these are the bugs that are specifically not seeing turnaround from maintainers. Just think, if every developer fixed just one of these bugs, we would have 19 NMUs for each package, the ftp team would be pissed, and we could release just two weeks from now once ftp-master.d.o was able to clear the backlog!

And remember, any bug not found in a standard 52-bug deck is a spare portabella mushroom.

Posted Thu Nov 9 21:53:31 2006

As a matter of principle, I try to avoid any conflict of interest between my release manager and alpha porter hats, by e.g. deferring to my co-RM when there's a question of whether a particular compromise should be made to benefit alpha's releasability. Where Dunc Tank is concerned, that means some changes in priorities for bugs to work on, since I'm being paid to try to get etch out in a timely manner, not to keep the alpha port afloat. OTOH, alpha is not the only arch that produces RC bugs needing release team attention, and Dunc Tank only dictates how I spend 40-50 hours of my week -- which means another 20-30 hours a week of Debian work that I can set my own priorities for, right?

So the theme ingredient over the weekend was: problems on the alpha port.

NMUed qdbm with a workaround for the long-standing build failure, helpfully filed by Aurélien Gérôme as bug #394626; this in theory makes it possible to get the fix for bug #367374 into testing by the normal route, but now hyperestraier and qdbm both have a very strange build failure on alpha and mips...
It took quite a bit of time to track down RC bug #378346 in gail, what with all the libraries that needed to be rebuilt from experimental. Turns out to be a garbage collection issue fixed upstream. Who knows why this only showed up on alpha?
I also poked a bit at the kernel for alpha, for the possibility of supporting installation on TITAN and TSUNAMI class systems. It looks promising to have this in for etch, which would be nice because these are higher-end alphas that could extend the lifetime of the alpha port overall if we can manage to turn that support into something like a new porter machine...
Then I spent some more time poking at the kernel thanks to 397139, which showed that the fixes to get the kernel building under gcc-4.1 on alpha didn't quite do the trick.

I'll give myself 4/5 points for presentation, 5/5 points for use of the theme ingredient, and 6/10 points for taste (this plastic computer case needs more salt).

And of course, waiting for Debian kernel packages to compile on an LX164 alpha gives one time to do other things too, like following through on the RC bugs that the BTS has wrong version-tracking information for, and preparing an NMU here and there. Less than 200 RC bugs left now, and dropping daily!

Posted Tue Nov 7 07:12:52 2006

Daniel, I fervently hope that this never finds its way into a Debian package without an explicit --verbose switch. There's nothing more frustrating than being unable to reproduce a build failure from an autobuilder, and not having enough information to know if this is because it was a one-off failure, an already-fixed bug in the environment, or an incorrect reproduction of the buildd environment.

And libtool --silent sucks too.

While I understand the desire for less verbose build output, a good postprocessor for build logs would IMHO help more than a tool that just suppresses output -- once it's suppressed, you can't get that information back...

Posted Thu Nov 2 17:37:02 2006

So over the last week, the various items of release happiness have included:

getting Xorg 7.1 into testing
getting apache 2.2 into testing
approving about a half-dozen freeze exceptions for base packages fixing RC or important bugs
removing about a dozen RC-buggy packages with no activity or explicitly scheduled for removal
bumping the urgency of some 20 packages that fixed RC bugs but were uploaded at low-priority despite the recommendations of the release team
getting all the udebs into testing that are wanted for the debian-installer RC1 release, which is now going forward
tagging all "fixed" RC bugs in the BTS with version information, so that we now have a definitive view of RC bugs left in testing
prodding the autobuilders for retries of uncounted numbers of RC-bug-fixing packages, failed for one reason or another
more great bugsquashing with the Cambridge BSP -- kudos to all those who helped out!

The bad news is that in spite of the time put in on whipping testing into shape, the count of bugs currently affecting testing according to bts.turmzimmer.net is still around 320, with about 210 of these unfixed in unstable; and the other 110 bugs fixed in unstable can't be ignored either, because there are many pitfalls along the way that can prevent a bugfix from reaching testing (such as a transient build failure on a single architecture that doesn't get looked at for a week or more) that require the shepherding of the release team to overcome.

Of course, some of these bugs are still false positives resulting from no one ever having checked the accuracy of closing versions on "fixed" bugs before now, but cleaning up false positives also takes time and care. Add in what looks like an average of 20 new RC bug reports a day, and it's no wonder at all that Debian can keep a release team of 4-6 busy full-time trying to get a release out.

Also on the horizon now are the ghc 6.6 transition (just 4 packages left to be fixed) and gaim 2.0 (two packages need fixed). The linux-2.6 update to 2.6.18 isn't quite ready to go yet due to some typical, new-upstream-kernel rockiness, but the kernel team are making good progress on addressing those bugs and we should see something happening here in the next week or so to let us close out our final RC kernel bugs for the release.

Posted Thu Nov 2 16:11:37 2006