PATH Service Shutdown on 33rd Street Line

Well, there was apparently a small manhole fire just east of the Christopher Street station in Manhattan today. Depending on which news report you believe, it supposedly started underneath a PATH train! All of the passengers were evacuated, and there were no injuries. Anyhow, what does this have to do with me? I take the PATH in order to commute to Manhattan when the job requires it. Today was such a day that I had to work in our lower Manhattan office, nowhere near to where this commotion occurred. However, there was a NYLUG meeting this evening, which happens in midtown. It's easiest (I think) for me to take the PATH from midtown when I'm in midtown. Makes sense. Good thing that I knew the service was shut down ahead of the meeting in midtown, so I knew that was a no-go. Everyone was saying "Wow, we feel really sorry for the folks that live in Jersey, they're gonna have one heck of a commute tonight". Fearing the worst, I took the NYC subway down to the WTC PATH station, and hopped on the train there. I think that they had increased service frequency to that station as much as they could, since there was a train waiting for me at 10PM, and it left shortly thereafter (with the train operator telling everyone that was getting on the train that there were plenty of seats in the rear of the train - problem being everyone needed to be towards the front because of where they were going). The train got quickly underway, and I think that I got into the subway at 9:45PM or something similar, and I got out of my station in NJ at 10:20PM. 35 minutes is not a bad commute for NYC, especially since I took a massive detour from where I was in order to get home. There are people that would kill for that commute, and that's what I get when everything's messed up. So I guess this PATH commuter had some good luck! It's unclear whether or not this will be repaired for the morning rush, so I'm planning to leave a little early since the route that I take is the only one operating between NJ and NY, and I'm expecting more crowding on the already jam-packed train. Good thing the customer meeting wars relocated (for unrelated reasons) from midtown to downtown! :)

Bugzilla cleanup stats

Well, I figured that I would post some statistics on the Bugzilla cleanup that we undertook a month ago. I'm in the process (hopefully to be done tomorrow) of writing queries to perform phase 2 of this operation: UPDATE - note that because of this, 466 bugs were closed, mostly by the reporter that realized they no longer applied. 2337 have received no response - that's fully 77% that were not actionable due to either MIA reporter or they simply didn't apply anymore. Not a bad exercise.

ASSIGNED	454
CLOSED	466
MODIFIED	28
NEEDINFO	2337
NEW	331
ON_DEV	1
ON_QA	2
RELEASE_PENDING	3
Total	3622

RHEL and VMWare time skew problems

Well, I've gotten quite the number of questions over the last few days about time skew problems in RHEL4 and VMWare, so figured that I would write this. In RHEL4.7, there will be a kernel feature that allows you to pass 'divider=' on the command line in grub. The number must be a divisor of 1000, and the most common value is 10 (yielding an effective timer interrupt rate of 100Hz). I'm expecting the 4.7 beta to become available Real Soon Now(TM), though I have no information as to when that will occur. In the meantime, this change is available as a hotfix from Red Hat's Global Support Services. What this does is take the kernel tick rate (which defaults to 1000Hz) and divides it by the value of divider, to come up with an effective interrupt rate. You can verify that this is working by: Before making the change, use 'watch --interval=1 cat /proc/interrupts' and you'll notice that the timer interrupt is incrementing by 1000 every second. After you make the change, you you notice with the same command that it is incrementing via 1000/ every second. The vlaue of HZ that is exposed to modules is still 1000, thus not breaking module ABI. This same change is already present in RHEL5.1 today. In recent Fedora's, this is unnecessary due to the advent of the tickless kernel (which is useful for other reasons, including reduced power consumption on laptops). It is likely that RHEL6 will also be based on a tickless kernel.

Yes, the counter went backwards :(

I've got the F9 counter on my sidebar, and yes, the counter went backwards today :(. The release slipped, mainly due to other slippage within the development cycle, and the fact that we had some major technical difficulties with rawhide this past week, thus forcing time to be focused elsewhere. Have I yet mentioned in this blog that time is a finite resource? But the preview release is now done, and open for business! Testers go forth!

New York Linux Meetup

Well, there's a LUG-type activity (smaller than NYLUG, and not really in competiton to them, more on that later) in NYC called the NYC Linux meetup which I am involved in. We're the largest Linux meetup in the US, however, we'd like to get the word out that we're around :) We generally meet once a month in person at Think Coffee on Mercer for our regular, topical meeting. This meeting is not really in competition with NYLUG, since what we generally do is more of a roundtable discussion than someone presenting something - try to get everyone involved in the disucssion, rather than a traditional LUG-type presentation The members come from varying backgrounds, being system administrators, developers, embedded systems folks, and all levels of Linux experience are welcome. All distros are also welcomed, but I'm biased towards Fedora, and the organizer is biased towards Ubuntu (I'm trying to convert him...really :) ) I'm planning a Fedora 9 release party with the group, and the organizer is planning an Ubuntu release party (which I will actively boycott :) ). We also have a projects meetup that meets once a month generally, at varying locations (I'm gonna go scope out a new one tonight); Last time, we met at the Humanties and Social Sciences library in midtown Manhattan. We have a channel on freenode, #nylinux as well. If you're in the NYC area, feel free to drop by, we're a great bunch!

Shell history meme

Well, now that I'm back from St. Louis, I was catching up on Planet. Here's mine: $ history | awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head 252 ls 232 cd 49 cat 38 tail 38 ssh 37 rsync 29 ps 26 sudo 23 vi 17 ll

Some thoughts on bug reports and free and open source software

Following from experience with the recent Fedora Bugzilla mass-triage, I figured that I would write a few words about the state of bugs in open source projects, and where people's perception can tend to fall short of reality. People assume that because OSS is not hidden within the walls of a particular company, that there are infinite resources to bring to bear on a problem. Instead, just as with propietary software, the resources are finite, the amount of hours in a day are finite, and the fact is that most of these individuals are contributing to an OSS community in their spare time, not being paid to do it full-time. Given these resource limitations, open source projects have to be selective in what bugs they can action. The incoming stream of bugs is literally more than they can handle. In order to handle the workload, they filter ruthlessly. If you file a request to, for example, change the background from green to red, that's unlikely to get as much attention as 'this sets my computer on fire'. I'm not saying that this is good, bad, or indifferent, but simply a fact of life in the open source world. Now that I've made general comments on the state of the FOSS community, lets analyze these with Fedora, with some real metrics. Fedora is a large project, consisting of 6,265 individual components in Bugzilla as of this writing. Most of these components relate directly to a piece of software in Fedora, though a very few are for administrative overhead of the project. As of this writing, there are 590 members of 'cvsextras', the group of people that have commit access to the Fedora package collection. Of these, 293 people have e-mail addresses @redhat.com, the primary sponsor of Fedora. Of those 293, most of them have other responsibilities other than Fedora. Let's make a liberal guesstimate and say that 10% of those 293 work on Fedora full-time. Let's make another liberal estimate, and say that 10% of ALL contributors are paid or somehow able to work on the project full-time, and that all of those contributors put in a 40 hour work-week. Let's also assume that the non-fulltime contributors each put in 3 hours per week. Let us also assume that it takes 30 minutes to deal with a bug (a very low estimate). We'll also assume that all these people do all day long is tend Bugzilla (a ludicrous assumption) So, given that number of contributors, which has grown significantly in the recent past, that means that they own, on average, 10.6 components each. The total number of bugs currently open in Bugzilla is 11,966. Using these assumptions, it would take 5976 man-hours to deal with all of these bugs. That is obviously quite an investment of time. Assuming that we had the 59 people that we assume work on Fedora full-time working 40 hours a week, and the rest of the 531 community members putting in 3 hours, it would still take a week and a half to get through all of this. That is a *huge* investment of time and resources. Thus far, we've been talking entirely hypothetical numbers. and assuming that everyone was qualified to work on all types of bugs. etc. Lets now look into the real world of the kernel. There are currently 1086 open bugs (~10% of the total!) against the kernel. These bugs have 53 assignees. Let's discount 21 of these, since they only have one bug assigned to them, leaving us with 32. Let's take our earlier assumption, that 10% of the assignees work full-time, doing nothing but tending to Bugzilla, and keep to the assumption that every bug takes 30 minutes. This gives us 2.5 weeks just to deal with the kernel! In conclusion, the open source community is very happy for people to report bugs. However, the developers that produce this code are very much inundated with them. What does this mean to you, the bug reporter? That we'd like for you to understand that some bugs slip through the cracks, and simply reporting a bug is not a guarantee that it can be fixed. I'd also like to point to some resources on how to write an effective bug report: http://www.chiark.greenend.org.uk/~sgtatham/bugs.html http://bugzilla.abisource.com/page.cgi?id=bug-writing.html http://www.songbirdnest.com/bugzilla_howto_writeabug

Failure to build from source bugzilla statistics

Matt Domsch occasionally rebuilds rawhide in mock, in order to catch packages that have bitrot that won't compile any longer. He just recently started mass filing bugs for them, and here's the status of those bugs.

ASSIGNED 11 CLOSED 63 MODIFIED 1 NEW 163 Total 238

Good, but not wonderful. Matt tries really hard to make Fedora a self-hosting distribution, so make sure to get those bugs fixed so the packages can compile! NOTE - the links above, due to bugs in bugzilla's reporting interface, may lead to the desired list, or it could lead to all of the bugs :/

And it's off....

The mass bugzilla cleanup that we had targeted for F9 has begun, in case anyone was living under a rock today :). Highly unfortunately, folks are getting several e-mails per bug. We have tracked this down with Red Hat Engineering Operations, they are apparently using separate XMLRPC calls to bugzilla to effect each of the three changes that we were doing to each bug. Sorry for the spam. There were three tickets filed with Red Hat Engineering Operations for EOL bugs, "relevant rawhide" bugs, and stale rawhide bugs. Only one of them is partially done being executed. For the remaining tickets, we've requested that e-mail be suppressed for two out of three changes - you'll only receive mail for the actual comment, not the status change or whiteboard change. Once again, sorry for the spam!

Yet another bike update

Got a call from the bike shop just now, and they told me that they're just going to give me a new bike. So it'll be in tomorrow, and I can pick it up on Friday. Yipee, I can hardly wait!