cat /dev/random

Exception transparency in Java

2010-06-07T15:32:00.001-07:00

Recently posted on my Oracle blog: http://blogs.sun.com/briangoetz/entry/exception_transparency_in_java

Memtest86+ rules!

2010-05-20T14:01:00.000-07:00

About two months ago I upgraded my system from Windows XP to Windows 7 64-bit, and at the same time from 4G to 8G RAM. As always happens, I was amazed how much faster a new Windows installation was than the old one on the same hardware -- it is insidious how "Windows Decay" chips away at performance.

About a week ago, the system started behaving badly -- IE crashing, Thunderbird crashing, and starting yesterday, the whole thing blue-screening. After wasting a lot of time trying to figure out "what software was updated recently", I started to suspect memory errors. So I ran the Windows memory test program that shows up on the boot screen -- nothing.

After more dorking around, I downloaded and ran MemTest86+ (www.memtest.org), burned it to a USB drive, and ran it. It immediately found thousands of memory errors; by trying various combinations and moving modules from slot to slot, I was able to identify the bad modules. I had bought Crucial's top of the line (Ballistix Tracer LED) from Newegg; the Crucial folks immediately shipped out a replacement.

Given how many errors MemTest found, its amazing that the Windows test found nothing.

Thumbs up for MemTest86+ and Crucial customer service. Thumbs down for Windows Memory Test.

Registration is open for the 2010 JVM Language Summit

2010-05-18T09:52:00.000-07:00

We've just opened registration for the 3rd annual JVM Language Summit, to be held at Oracle's facility in Santa Clara CA on July 26-28. See http://www.jvmlangsummit.com/ for details.

Book review: Fermat's Enigma

2010-01-31T06:59:00.000-08:00

This is a nice little book about the history of mathematics and the 350-year quest for the proof to Fermat's Last Theorem. It was written by the fellow who wrote the BBC / Nova TV special on Andrew Wiles, but includes a lot more information than a one-hour show could. It does a nice job at hitting many of the high points of mathematical development from Pythagoras to modern day, including the "discovery" of zero, then negative numbers, then imaginary numbers, techniques for grappling with infinity, Turing-computability, and Godel's incompleteness theorem. It doesn't attack any of these in great depth, but it does provide a nice historical perspective while remaining about as accurate as a lay book can do. It also does a nice job of illustrating the near-hubris required for Wiles to lock himself in a closet for eight years in order to solve a problem that had eluded mathematicians for centuries. Mathematicians will enjoy the panorama; non-mathematicians will likely find the introduction to some of these obscure concepts accessible and enjoyable. Also by this author: The Code Book: The Science of Secrecy from Ancient Egypt to Quantum Cryptography .

(Recommended to me by: Stuart Marks.)

Our government, protecting us

2010-01-31T06:36:00.000-08:00

We've recently gone on an "energy efficiency" rampage at the house, replacing bulbs with CFLs, identifying devices that are unnecessarily left on all the time, wrestling with Windows to stay asleep during periods of inactivity, etc. We also recently just installed a "continuous" or "on demand" hot water heater, replacing the 50G direct-vent tank heater we had (it was getting to the end of its lifetime and it was easier to replace it preemptively.)

Unfortunately, the state requires all newly install water heaters to have a thermostatic mixing valve that limits the water temperature to 120 degrees. (For tank systems, it is recommended to keep the tank water at 140, to prevent the bacteria that causes Legionnaire's disease, but 140 is hot enough to scald. But continuous systems have a control system for the output temperature, so can be safely kept at whatever temperature you program in.) And its probably not even working right, since the output temperature is even less than 120. The valve adds cost to the system and to the installation (probably a dozen additional welds in addition to the valve), and while we now have an infinite supply of hot water, generated more efficiently, its not as hot as we like it.

Reputable plumbers are not able to remove or bypass the valve, which means we need to either find a disreputable plumber or I need to do it myself (read: find an incompetent plumber.)

Note to lawmakers: in my many years of successful shower use, I've learned a secret trick to avoid getting scalded: put your hand under the water first -- if its too hot, turn down the water temperature before getting in!

Thanks, elected officials, for making my house systems both more expensive and less useful.

e-mail packrat

2010-01-23T14:23:00.000-08:00

I've long tried to keep all the e-mail I've ever sent or received; I've got an archive going back to 1985 or so, when I first realized that keeping e-mail might be a good idea. Trouble is, keeping such an archive in one place requires a fair amount of maintenance, because formats and protocols change. Is it worth the effort?

Until 1987, I primarily used a VMS system.
In 1987, I switched to a Unix machine at MIT. I was able to import my old VMS mail into whatever the mailbox format of the day was (mbox, probably) by a script I found somewhere.
In 1992, I switched to using POP through the client program Eudora. Eudora stored the mail locally, in a folder format that was something like 'mbox', but not exactly. (For example, attachments were not stored inlined, but instead in external files.) I managed to import my old Unix mbox files into folders.
In 2004, I switched from POP to IMAP. I went through an extensive process to convert my existing mail base into real mbox files that my IMAP server could read. I spent several days writing scripts to convert the Eudora pseudo-mbox files to something imapd could handle.
In 2006, I left Quiotix, and switched my primary mail over to Tuffmail. I took my mail archive (in mbox file format) and put it up on my server machine, and serve that up with imapd. So I now have my mail split between two servers, but Thunderbird can deal with multiple servers just fine. I tried to move the archived mail to Tuffmail as well with several different tools (imapsync, offlineimap, Thunderbird bulk-copy) but I could never get a clean copy -- I suspect that the combination of crappy old multiply-converted mbox files and the old UW-IMAPD server is to blame.

Right now its still fragmented across a number of formats and servers. Yuck.

zTunes released

2010-01-09T04:17:00.000-08:00

As promised in yesterday's entry, I'm releasing my digital media management software (ztunes) to the world. It's hosted at github: http://github.com/briangoetz/ztunes. It is written in Ruby and based on "rake", the Ruby equivalent of "make". It currently has a long way to go but already does a lot.

You can download the Ruby gem here: http://github.com/briangoetz/ztunes/downloads. (It is not currently in any sort of gem repository.) It defines its gem dependencies, but you'll also need the Unix tools ffmpeg, flac, and lame. It will run on Linux and Mac but currently has some trouble on Windows since it is dependent on symbolic links for some of its functionality, which Windows doesn't support.

My motivation for writing this was that iTunes is really inadequate for managing a media library unless (a) you only want to play on iPod (or other Apple) devices and (b) you are willing to let iTunes be in control of ripping and encoding. This didn't work for us for two reasons: we have Squeezeboxes on all the stereos, and I want to rip my CDs to a non-proprietary, lossless format (that means flac, which iTunew doesn't support.) We also have music that has been aquired in various other forms (MP3s from Amazon, AAC from iTunes, WMA from Rhapsody) and want to be able to play all the music on all the devices, without transcoding it all down to a least-common-denominator. (In other words, if Squeezebox supports WMA but iPod doesn't, let Squeezebox play off the original WMA but let iPod play the transcoded version.) And this should be transparent to the rest of the family.

There are several basic tasks in managing the media library that zTunes automates:

Content ingestion. I've got a "drop" folder, into which I want to drop the originals of my media, in whatever form, and have them be analyzed, metadata extracted, and filed into a unified library based on its metadata. My metaphor here is the gas tank of an M1 tank: you can pour anything combustible (gasoline, jet fuel, diesel, used cooking oil) into the tank and it figures out how to burn it. Currently it maps a media file to a filename by using the author/album/title tags for audio or the title tag for video; audio files are named like "The Who/Who Are You/Squeezebox.flac".
Transcoding. Not all devices play all device types. So ingested content also needs to be transcoded into alternate formats, which are maintained as parallel directory trees. The transcoded trees are transient; they are merely shadows of the "authoritative" tree. Some files may need be transcoded to multiple formats; for example, video files ripped from DVD or transferred from TiVo might be transcoded to 480 x 320 video for iPhone but 320 x 240 for the older video iPods.
Syncing. I use the Windows program "Tag&Rename" to edit the metadata tags on my media files, to normalize genres, naming details like "The Cars" vs "Cars, The", "Vol 1" vs "Disk Two", etc. When I edit the metadata on an "original" file, I'd like the file to be renamed accordingly, and metadata changes to be reflected in the transcoded copies. When I delete an original, I want the transcoded copy to go away. Etc.
Device management. I would like to have a single directory for each device type, that I can point device-specific library management software (iTunes, Squeezecenter, Creative Explorer) at, and it will see the right view of the media library for that device (will only see files it can play; will see them in the "best" format available for that device.)

One thing it does not do yet is manage the integration of your external media library into iTunes (iTunes is particularly bad at dealing with files you didn't acquire through iTunes.)

See more in the README file here: http://github.com/briangoetz/ztunes/blob/master/README

I'm currently using this to manage a library of ~8,000 media files in half a dozen formats. I'd love to get some more users -- drop me a note if you're interested!

A busman's holiday

2010-01-08T14:46:00.000-08:00

Since I've been working way too hard, of course I decided to spend my XMas break...programming. (http://www.answers.com/topic/busman-s-holiday). I had two goals: rewrite my digital-media handling software, and learn Ruby. I'm pretty happy with what I accomplished on both counts.

The motivation to rewrite my digital-media scripts came from having too many conversations like the one below with Stuart Marks:

SM: Hey, you wrote a bunch of scripts to manage audio and video files, are you willing to share them?
BG: Well, in theory, yes. But I'm kind of embarassed to show them to anyone...
SM: Let me guess. Perl?
BG: Yep.
SM: I have a Perl story...
BG: Don't bother -- all Perl stories end the same way.

I'll post the full details soon -- including links to the software on github -- but for now I'll just outline the problem I was trying to solve:

Ingest digital media files in any format (MP3, AAC, WMA, WAV, FLAC, M4A, M4V, WMV, MP4, etc)
File them into a library based on their metadata
Additionally transcode them down to one or more "compressed" formats (MP3 for audio, iPhone-sized Mp4 for video) for memory-constrained devices, without letting go of the original
Organize them so that each device (iPod, Squeezebox, non-iPod MP3 player) can play all the media, in the best format that the device can recognize natively (Squeezebox supports MP3, WMA, and FLAC; iPod supports MP3 and AAC; Zen supports MP3 and WMA) or a transcoded form if it can't. For example, for a given track whose source form is WMA, Squeezebox and Zen should see the WMA but iPod should see the MP3; for a track in FLAC, Squeezebox should see the FLAC but iPod/Zen should see the transcoded MP3.

Is this a joke?

2009-10-18T07:13:00.000-07:00

We decided to upgrade the hard drive on our Tivo Series3, since the stock 250G drive only holds about 30 hours of HD video. I studied the various Tivo upgrade boards and selected a hard drive that had been recommended by some, the WD 1.5TB "EADS" Green drive. The upgrade process is simple: crack the tivo, extract the drive, move the data from the old drive to the new, "expand" the new drive (updating the partition table so it uses the extra space), and replace the drive.

Attempt 1: 1.5TB drive, using trusty Unix tools -- put the drives into a Unix box, copy data with dd, then do the expansion with 'mfstools'. This is the approach I've used several times in the past, with good results. Put the new drive in, turn it on, and it gets stuck forever in the "Wecome, powering up" screen. Back to Google.

Turns out that the S3 can't see a partition bigger than 1TB, and mfstools expands the partition to the whole rest of the drive, yielding a too-big partition. Turns out mfstools doesn't support limiting the size of the partition, but the Windows version (winmfs) does, so I'll use that instead. (Its good to have lots of spare computers around when attempting any sort of upgrade.)

Attempt 2: 1.5TB drive, using winmfs. Put the drives in the windows box, run winmfs to copy the data, and let winmfs expand the partition. It asks me "should I limit the partition to 1TB", I say yes, good. Put the drives back -- same problem. More Googling.

So I discover that "some versions of the drive I was using (WD15EADS) are 'not compatible' with Tivo Series3." Its been years since I've heard about incompatible (system, disk) pairs, and this is a standard SATA drive, but OK, I guess I bought the wrong drive. RMA time. Sorry, NewEgg. The Tivo Upgrade FAQ (http://www.tivocommunity.com/tivo-vb/showthread.php?t=370784) is telling me I should favor the WD EVVS drives instead, so I buy a 1TB drive (WD10EVVS) from Amazon.

Attempt 3: 1TB drive, winmfs. I repeat the process, copying the 250G drive to the new 1TB drive, and put the drive back in the Tivo. (At this point I've learned to try it before I fasten all the screws.) Same deal -- stuck on the "Welcome, Powering Up" screen. More Googling.

I found this update, which was added after I'd bought my drive:

The WD10EVVS was removed from the list on October 10, because there is a new
batch of that drive, manufactured on September 20, that is not compatible
with the TiVo. These incompatible drives are labeled as follows:

MDL: WD10EVVS - 63M5B0
Product of Thailand
DATE: 20 SEP 2009
DCM: [b]HAxxxxxxxx
R/N: 701640
LBA: 1953525168

I looked at my drive, and sure enough, I had one.

Is this an elaborate joke?

Next up: RMA redux, ordered a WD 10EVDS drive. Stay tuned.

Update: installed the WD10EVDS, worked fine. Fourth time's the charm!

You've been scammed!

2009-10-15T22:28:00.000-07:00

I got an odd e-mail from PayPal the other day, telling me I'd paid EU250 to something called "Skype Business Panel." My first thought was that it was a phish, but careful examination suggested it was real. I logged on to my PayPal account and indeed I'd been charged EU250. What the hell is "Skype Business Panel", anyway? Turns out it is a skype feature where businesses can allocate credit to the skype accounts of their employees and thereby manage their telephone spending.

I have spent about $10 with skype per year, recharging my skype account from PayPal when it ran low. Somehow (don't remember) I had authorized skype to charge my PayPal account when my balance got low. And this was the vector through which I was scammed. Someone must have gotten a hold of my skype password (don't know how), logged on, and billed EU250 to my PP account (which didn't require a PayPal webflow), and then allocated it to some bogus accounts.

First stop: dispute the charge with PayPal. They were completely unhelpful, pointing me to the authorization and told me to work it out with skype. Fortunately skype was more helpful, and they reversed the charge immediately.

I then logged on to my Skype Business Control Panel (now that I know such a thing exists), and found several bogus accounts linked to mine, which I deleted. After all was said and done, including the refund, I still somehow had a EU100 balance on my BCP, meaning somehow the scammers gave me EU100.

To see if you have any such preapprovals on file: login into your paypal account, click "Profile", and click "Preapproved Payments." You can delete them from there.

Laptop upgrade annoyances

2009-08-29T08:28:00.000-07:00

We've got an old Dell C400 laptop. Its seven or eight years old, but its still going strong, and it just fine for an around-the-house laptop (similar in performance to a modern netbook, but with bigger screen/keyboard, and still nice and light). The limiting performance factor right now seems to be the hard drive; many Windows operations (booting, shutdown, sleep, wake) are disk-seek-bound, so I bought an IDE SSD to replace the existing IDE drive. Hopefully that will also improve battery life and thermal characteristics.

What I'd like is a simple way to move all the data from the existing drive to the new drive, and then just toss the old drive. But this isn't as simple as it might appear. Laptop IDE cables generally only support one drive, so I can't use (say) PartitionMagic to do a partition copy the way I would on a desktop system.

A lot of people have suggested various tricks, like:

Get an IDE-USB adapter, put the old disk on that, put the new disk in the machine, boot from a Linux CD, and use dd to copy the data;

Get a pair of 40 pin to 44 pin IDE adapters, put them in a desktop system, and copy using PartitionMagic (Windows) or dd (Linux);

Find a dual-drive 44 pin IDE cable, plug both drives in, and hope that the OS / BIOS recognizes both disks;

Just reinstall Windows and whatever apps I have on the new drive (including chasing down all the device drivers, such as the touch pad, speakers, etc)

Why is this so difficult? A hard-drive-swap should be a simple, common upgrade operation, that shouldn't require using tools from another operating system, transplanting the drives into another system, or rebuilding the world from scratch.

On a similar note, I just bought a Samsung NC10 netbook, and was going to wipe the disk and reinstall OSes. I have all the software I want ripped to ISO images, many of them bootable. Why is it so hard to take a bootable ISO and turn it into a bootable USB key? (I tried "unetbootin" but it didn't work on the PartitionMagic ISO, which is usually my first step in installing onto a new PC.)

WiFi prices finally come down

2009-06-12T04:38:00.000-07:00

I've always been annoyed by just how expensive WiFi access is for such spotty coverage. I've had the T-Mobile plan ($30/mo) for a few years, which provides coverage at Starbucks (until they switched to AT&T), and many airports and hotels. Its been a better deal than not having it (they have coverage at the hotel I stay at most frequently, and the airports I transit through most frequently), but it always felt like too much money for too little service, given that there is not always a TMobile hotspot available. TMobile has roaming deals with many of the other big providers (Boingo, AT&T), but the only real benefit there is the convenience of the billing arrangement, as the roaming fees are not nominal. (Though I do like that TMobile also provides convenient pay-by-the-minute roaming access at many hotspots in Europe.)

Finally there seem to be some better alternatives. Boingo now seems to have an unlimited $10/month plan, so I switched to that. Boingo claims I also get free roaming on many TMobile, AT&T, and other hotspots -- I'll report on that once I get my first bill. I downgraded my TMobile account to the "Pay as you go" plan, which has no monthly fee, and is $3 for the first hour, which seems like a good option to have.

Starbucks also has a reasonably priced plan (Starbucks Gold Card) if you spend a lot of time in or near Starbucks (they are in the process of switching their hotspots from TMobile to AT&T.) For $25/yr, you get two hours per visit of WiFi time (not sure if this is enforced or not), plus 10% discount on most Starbucks purchases.

Initial iPhone experience -- disappointing

2008-11-19T10:51:00.000-08:00

I live in an AT&T-free state, so I have not had access to the cult that is iPhone. But recently, in preparation for AT&T moving into the state (through an asset swap that involves AT&T acquiring the VT GSM assets that Verizon bought in acquiring Rural Cellular), they are now willing to open accounts with VT addresses. So when in CA this week, I went to an AT&T store to plunk down my money so I could be cool like all my friends. I purchased a 16GB iPhone 3G.

I got out of the store and into my car, and noticed that the edge where the front metal rim meets the plastic case was extremely rough -- almost sharp enough to cut. This was not the seamless tactile experience I was expecting from Apple. So I went back in the store, and asked for an exchange. I was told that "Apple prevents AT&T from making exchanges" and was sent to the Apple Store. When I arrived at the Apple Store, the rep informed me that they could make an exchange, but it would be a refurb unit, not a new one, even though mine was clearly new, because I'd bought it at an AT&T store and not an apple store.

So I went back to the AT&T store and argued with the manager. He tried to send me back to Apple. He ended up calling the Apple store, who must have told him to take the exchange, so in the end I got a new, non-defective phone. All was made right, but the experience was none too pleasant, involving three store visits.

While in the Apple store, which had many iPhones on display, I took the opportunity to do some sampling. I discovered that many iPhones had rough or sharp spots, and not all in the same places. Seems that in reducing the cost of the 3G, perhaps some quality-control corners were cut as well, since many were not very pleasing to the touch and there were significant variations in perceivable quality.

My favorite computer science book

2008-10-15T09:53:00.000-07:00

Pierce's Types and Programming Languages is a masterful introduction to the theory and practice of type systems. One of the things that makes this book so great is that it is equally accessible to both the theory-oriented and the practice-oriented. This was driven home to me in a conversation with Ola Bini, when I saw he was carrying this book, and he commented "I love this book because I can skip all the math and get what I need from the ML implementation." I answered that I liked it for the opposite reason; I was able to get everything I needed from the math and didn't have to look at the code. Its pretty impressive that a book can be that useful and successful from two such radically different reader approaches.

I found that Pierce's treatment was extremely accessible. He starts with almost no assumptions, introduces first the untyped lambda calculus, then the simply typed lambda calculus, some obvious extensions (records, references, subtyping, union types, functional objects, etc), operational semantics, and builds gradually to more useful type systems. Each section includes motivation, analysis, a formal description of the system, soundness proofs, and ML code; the impatient can skip some of these and still get what he's talking about. There is working code for each of the languages developed. (The type systems were developed in a system that the author wrote called TinkerType, which makes it possible to build type systems by "mixing and matching" features, and it generates both the ML code and TeX source for generating the figures used in the book -- most impressive!)

Not only is this book useful to anyone who is interested in the design and science of programming languages, but it is also a pleasure to read.

What's your favorite computer science book? (Unoriginality points for anyone who says TAOCP.)

David Foster Wallace, RIP

2008-09-17T00:31:00.000-07:00

I was deeply saddened at the news that David Foster Wallace committed suicide last week.

For me, the experience of reading Wallace's writing is not unlike that of watching an olympic gymnast. While the right side of the brain is being entertained by the grace and artistry, the left side is frantically marvelling at how the human body can do that at all. The tension between the two -- where your brain can't decide where to focus, not wanting to miss either part -- adds all the more to the experience.

Wallace's mastery of the language is undeniable; one could read his work simply to marvel at the construction of each sentence or his ability to move effortlessly from one writing style to another. But, unlike other authors known for their "style", the writing is merely the surface layer; Wallace actually has something to say, his arguments are compelling and challenging and beautifully constructed, and supported with relevant data drawn from disciplines ranging from literary theory to mathematics. And somewhere along the line he also manages to make you laugh out loud -- right before you have to pick up the dictionary for the seventh time.

One is, at the same time, amazed, informed, challenged, entertained, and, honestly, filled with that feeling of "I'm not worthy" on multiple levels.

I would like to be able to say "I knew him when"; he and I overlapped for a year or two at Amherst. But I never actually met him, I only heard the stories, such as his senior English thesis being published as a novel ("The Broom of the System"), or being the only student in then-recent memory to have achieved the distinction of summa cum laude for his thesis work in two separate majors (English and Philosophy.)

Harper's Magazine has graciously made the pieces he published in that magazine available for free on the web: http://www.harpers.org/archive/2008/09/hbc-90003557 . If you've not had the pleasure, I suggest you read "Tense Present" -- which probes "the seamy underbelly of US lexicography" -- and then marvel at the notion of how entertaining and actually useful a book review of a dictionary could be.

Rest in peace.

Wallboard + paint + pressure = superglue

2008-06-07T06:20:00.000-07:00

This surprised the heck out of me. We recently finished a new TV room down in the basement. We have a 50" plasma TV, mounted on the wall with an Omnimount UCL mount (quite an impressive bit of engineering -- not cheap, but highly recommended.) Since the mount weighs upwards of 40 pounds, and supports a TV that weights 100 pounds on a torque arm as long as 2ft, it needs to be anchored pretty solidly to the wall. It is held up with 6 4" long, 3/8" wide lag screws, that screw into two separate studs. It does its job well.

So, my dad and I went to remove it from the wall. I removed the six screws, and we prepared to catch the mount. It didn't fall off the wall. We tugged on it, and it still didn't come off the wall. Seemed stuck so tight we thought we'd missed a screw! But we convinced ourselves there were no more screws, and the two of us pulled hard, and eventually it came off the wall -- taking some of the wallboard with it. Apparently the pressure of being screwed up against the wallboard (and maybe the heat from the TV too over a few years) turned the painted surface into a glue not only strong enough to hold a 40lb mount to a vertical wall but resist being pulled off!

Questions from the Peanut Gallery, part I

2008-06-02T06:27:00.000-07:00

At the "Writing the Next Great Java Book" BOF at JavaOne, there were unfortunately many more questions than we had time to answer. Fortunately, Greg Doench saved the questions (presciently, they were submitted on paper), so I'll answer a few here.

Q: About how much time did you put into your book (effort, not duration)?
A: In my case, the answers for effort and duration are the same, as I had the luxury of writing the book mostly-full-time -- I made the book my foreground activity, though I still did some consulting and training while I was working on it. I spent approximately 16 months on the book -- longer than planned (but in hindsight is no surprise.)

Q: Was the financial compensation worthwhile?
A: Unless your name is Stephen King or JK Rowling, writing books is not something you do for the financial compensation. This is more true for technical books, because (a) the audience for books like Java Concurrency in Practice is not quite as large as the audience for Harry Potter, and (b) if you have the skills to write a good technical book you probably have the skills to get a well-paying technical job. Without going into the details, I'll say that the compensation is about what I expected -- but I went into it with very realistic expectations. The compensation comes in other forms.

Q: How much support and assistance was provided by the publisher?
A: I think this is a matter of how much support and assistance you ask of the publisher -- and how much the publisher thinks you need. In our case, we did everything ourselves, including typesetting and managing the review, copy editing, and index creation. These are things the publisher often does for authors (and might even have preferred to do), but we chose to do it ourselves, and the publisher agreed. Of course, this was more work, but it was work we gladly did. The A-W team was always responsive when we did ask for things. So I think the answer is "as much as you appear to need."

Q: How does the short half-life of technical topics affect the effort?
A: I deliberately chose a topic with a longer half-life. This gave me the latitude to let the book tell me when it was done, rather than the schedule. For material with a shorter shelf life, I might be inclined to choose a shorter format, so that the book is less out-of-date by the time it is published.

Q: What would you say about books that authors release chapters to the public as they write?
A: I think this presupposes a style of writing where the author sits down and writes the book linearly. I am sure some authors do this, and some topics are more amenable to this approach than others. But one of the most important freedoms in writing is the freedom to refactor continuously; very often you don't figure out the right way to present the material until you've presented it the wrong way (just as with code.) There's nothing wrong with putting the work out there early -- this is a great source of free review -- but you have to be careful that doing so doesn't cause you to settle into the belief that the structure of the book has been decided. (The same risk is true of trying to adhere to a schedule that assigns due dates to specific chapters.)

Q: How do you avoid example source code exploding without using unrealistic examples?
A: This is really hard! But its really important. In JCiP, we set a rule for ourselves of "no code example more than a page", with the target of making most of them a half page or less. This is not easy, especially in Java! (There was only one we had to break into two separate one-page listings.) We wanted the examples to each illustrate a single point, so that the reader could look at the example and easily see what it was trying to show. There are some obvious tricks; eliminating boilerplate code like constructors, getters, and setters helps a little bit. What worked for us was to pick realistic examples that the audience would immediately understand the utility of (such as a file crawler), but abstract away the irrelevant concrete details by not showing the bodies of methods that are not needed to make the point that the example is supposed to illustrate. For example, we have a set of examples in Chapter 8 where we illustrate searching for solutions to a class of puzzles such as the "sliding block puzzles." But rather than focus on a specific puzzle -- which would take lots of space and not offer all that much insight, we abstract the nature of the puzzle by defining an interface that specifies the initial position (in terms of an abstract Position class), valid moves (in terms of an abstract Move class), and the goal position. Then we can illustrates various search techniques in terms of the abstract puzzle without getting bogged down in the details.

Q: What would you say is the role of technical books in the age where the Internet is the fastest way to publish texts and technology changes so fast that one year after publishing texts become irrelevant?
A: Some technical books are simply a form of documentation; any book that has a version number in the title is likely to fall into this category. These books have a very short shelf life. Other books, those that tend to focus on concepts rather that specific technical details, tend to have a longer shelf life. In any case, the publishing industry needs to become more agile in its approach to managing the authoring and production process, and explore more seriously alternate publication vectors such as electronic publishing.

More later.

The making of JCiP; avoiding errors in code listings

2008-05-31T05:29:00.000-07:00

At JavaOne this month, my publisher at Pearson, Greg Doench, hosted a panel/BOF entitled "Writing the Next Great Java Book." Not surprisingly, when you get four authors in a room, you get seven opinions about what's important in writing a book!

One thing that we (mostly Tim, actually) did for JCiP was set up an infrastructure for the book, similar to what you'd do for a software project. Version control, issue tracking, one-step build script, continuous build -- all of these things offer the same benefits to book projects as they do for software.

One critical aspect of the build is the handling of program listings. It is incredibly tempting to cut and paste examples from the IDE into whatever source format you're writing in (Word, Frame, LaTeX, DocBook), but this is a recipe for disaster -- errors will invariably creep in as you try and make small tweaks (such as changing variable names) outside the IDE. And code examples with errors really undermine the reader's confidence (or worse, they copy the incorrect example into their code.) So, we wanted to make sure that every code example compiled (and ideally, was tested.)

Our approach was to check the code into Subversion with the rest of the book artifacts, ensure that the build process compiled the code and ran the unit tests, and then automatically extract the examples from the code in a format into which they could be directly included by the build. Some systems (LaTeX, DocBook) make this sort of inclusion easier than others.

We marked the examples up with comments for formatting (bold, italic) and also with "snip here" comments that excluded the irrelevant portions of the code from the listings that actually went into the book. The attached perl script (phragmite.pl), written by Tim Peierls (based on an approach designed by Ken Arnold), takes as input a set of input files and produces a set of LaTeX files representing the extracted listings.

As an example, here is the Counter listing from Listing 4.1 of JCiP:

// !! Counter Simple thread-safe counter using the Java monitor pattern
// vv Counter
@ThreadSafe
public final class Counter {
  /*[*/@GuardedBy("this")/*]*/ private long value = 0;
  public /*[*/synchronized/*]*/ long getValue() {
    return value;
  }
  public /*[*/synchronized/*]*/ long increment() {
    if (value == Long.MAX_VALUE)
      throw new IllegalStateException("counter overflow");
    return ++value;
  }
}
// ^^ Counter

The first line identifies the type of the code fragment (!! for a "good example", ?? for a "bad example" which would get decorated with a Mr. Yuk), the name of the fragment (Counter), and the listing caption. The lines with the ^^ and vv mean "snip from here to here", and a listing can be made of multiple such fragments. The /*[*/ and /*]*/ comments mean "bold". The following ANT target ran the script:

  <target name="listings">
    <exec dir="${bin.dir}" executable="perl">
      <arg value="${phragmite.pl}"/>
      <arg value="${listings.dir}"/>
      <arg value="${fragments.dir}/*.java"/>
      <arg value="${fragments.dir}/jcip/*.java"/>
    </exec>
  </target>

In the book's LaTeX source, we use the following LaTeX macro to pull the listing in:

\newcommand{\JavaListing}[1]{\input{listings/#1}%

Apologies for the malware warnings

2008-05-17T03:53:00.000-07:00

Apparently, WordPress is vulnerable to some script injection bugs, and this site was hit by them. And Google tagged the site as "spreading malware", so the site shows up with a warning in Google search results and FF3 users can't get to it at all. I've upgraded Wordpress, scoured the DB for injected scripts, and am in the process of begging google to let me off the blacklist.

What a pain in the butt. People suck.

Tailored, indeed

2008-04-30T14:48:00.000-07:00

I noticed this amusing typo at the trade show floor at SDWest this year. My first thought: wow, they really are tailored!

Hacker at work

2008-04-30T14:45:00.000-07:00

This photo was taken in Josh Bloch's garage on a recent trip to CA. We were installing a new fireplace grille after the fireplace had been re-faced with some beautiful vintage tiles. Even though this was an entirely analog activity, we still managed to get a little hacking in.

A day and a half in the life...

2007-12-10T09:23:00.000-08:00

This is a story about Windows, Samba, debugging, and frustration. I just spent a day and a half debugging a very annoying performance problem with a brand-new system.

Background: I just recently bought a new desktop computer, which I do about every two years. Computers should last longer than that, but I find that, even with good "hygeine", Windows systems tend to decay to the point where they exhibit weird behaviors after about two years, for which the cure is a complete reinstall of the OS and all applications. (Pause for Mac fanboys to snicker.) The "rebuild the world" process wouldn't be so bad if it weren't so hard to migrate all one's data -- even given the fact that a lot of my data is already in subversion. The real problem is that each application sprays its configuration and data randomly around your system, whether in the program install folder, registry, documents and settings folder, local settings folder, etc.

So, I bought a new computer, an extra-quiet one from www.endpcnoise.com. These guys specialize in quiet systems, and since I work in a home environment, the computer is usually the noisiest thing in the room. (Of particular annoyance is the variable speed fan on my existng Dell system, which, every time the system worked up a sweat, made a whiny noise. And we was fined $50 and had to pick up the garbage in the snow, but thats not what I came to tell you about.) Pretty happy with the new system overall.

So, I installed XP Pro on the new system, and proceeded to install all my applications, utilities, and all kinds of groovy things that we were talking about on the bench. And then I got to the part where I tried to use it; specifically, tried to fire up IDEA and build the project I'm working on.

Now, I've got a somewhat weird setup; when developing on a project, I checkout a workspace on my Linux server, which is served via SAMBA to my local network, and I run the IDE on my Windows desktop and point the IDE at my samba share. There's a measurable performance hit vs local, but I like being able to do some things from the Linux command line and other things from the IDE, so overall its a more productive setup for me.

When you ask the IDE to "make" the project, it crawls the files in the project checking their modification times. An "empty" make on a project with ~1000 files generally takes a few seconds to figure out that there's nothing to do. But when I set it up, the new (faster) system took about 30-60s to do an empty make.

OK, it's debugging time. What's different about the two systems? Same OS (XP Pro), same service pack level, both systems up to date on patches, same Java version, same IDE version, same user credentials, no host-specific information in my SAMBA config. Different hardware. Make sure my ethernet drivers are up-to-date. Test network for errors, swap cables, all that. Run IOMeter, found that both get similar throughput for large files on the same SAMBA share.

Crank up perfmon, which tells me that the new system is sending out more packets for a make than the old one. OK, crank up ethereal, get a packet capture, and find that the new system is sending/receiving 10x as many packets for the same operation:

[brian@brian-server ~]$ wc -l /tmp/*cap
258204 /tmp/new-cap
17719 /tmp/old-cap

So, what's the difference? Let's look at the packet capture. In the old trace, for each file being probed, it did something like this:

0.467895 192.168.1.104 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Basic Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
0.468041 192.168.1.107 -> 192.168.1.104 SMB Trans2 Response, QUERY_PATH_INFO
0.468283 192.168.1.104 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Network Open Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
0.468402 192.168.1.107 -> 192.168.1.104 SMB Trans2 Response, QUERY_PATH_INFO

Two requests, two responses per file. Seemed reasonable. On the new system, for each file:

2.010471 192.168.1.113 -> 192.168.1.107
SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree
2.010698 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2754
2.010900 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \
2.011011 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2755
2.011237 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work
2.011570 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: work
2.011752 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2755
2.011833 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.012025 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler
2.012157 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2756
2.012353 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes
2.012631 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: classes
2.012796 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2756
2.012897 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.013100 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler
2.013239 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2757
2.013445 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com
2.013894 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: com
2.014095 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2757
2.014174 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.014355 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com
2.014504 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2758
2.014962 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun
2.015169 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: sun
2.015339 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2758
2.015428 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.015633 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun
2.015764 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2759
2.015980 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx
2.016221 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: javafx
2.016402 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2759
2.016493 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.016693 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx
2.016827 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x275a
2.017096 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api
2.017348 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: api
2.017520 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x275a
2.017590 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.017803 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api
2.017919 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x275b
2.018133 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree
2.018389 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: tree
2.018547 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x275b
2.018626 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.018779 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2754
2.018851 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.019157 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Basic Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.019292 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO
2.019495 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Standard Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.019613 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO
2.019832 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Internal Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.019960 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO
2.020206 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Network Open Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.020316 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO

For those of you who don't enjoy reading packet dumps (I hope that's all of you), what's going on here is that it does some sort of complicated multipacket transaction for each level of directory from the project root down to the last directory in the chain, and then does four request-responses for each file. And it repeats the directory stuff for every file, even though it just asked that.

OK, more debugging -- what could cause a system to deviate from the standard file system client behavior? Check all the network control panel settings, they're all the same. Spend several hours googling through the MS knowledge base for file sharing related problems, look at the various registry keys and file versions mentioned, nope, none of them are helpful. Google for people who have had similar problems. Many have, but no one reported a solution that works, except one person, who mentioned that their network behavior changed when they changed versions of Symantec antivirus. Well, I don't run Symantec AV, but I do run ZoneAlarm. And I do have different versions -- ZA Antivirus on the old system, ZA Suite on the new. Seems like a small difference -- they're clearly built on the same base technology -- but lets try it. Disabled ZA, reboot, make sure its not running, and run my IDE again -- no change. Still an annoying 30-60s delay before it figures out there's nothing to rebuild.

At this point, I asked my friends for help. Lots of sympathy. Lots of "check this, check that", but very little advice that actually moved me towards a solution (sorry, guys).

That post about the guy with the Symantec problem gnawed at me, though. I know security programs intercept a lot of network traffic, so the theory was perfectly plausible, and the best theory I had so far. I did the "disable ZA" thing again, rebooted, and cranked up Rootkit Hook Analyzer to see if ZA still had anything hooked, and it did, even though there were no ZA processes running and the ZA TrueVector service was stopped. So, I ran the uninstaller for ZA Suite, rebooted, checked with RHA to see that everything was unhooked (it was), and ran my IDE test again -- and this time, sweet success!

So, the conclusion is that the ZA Suite interferes with file sharing client behavior in a rather fundamental way (but one which only has a noticeable affect when dealing with lots of small files).

So, my system is temporarily defenseless against malware while I decide what to do. Why on earth would ZA rewrite the file system client packet stream like that? I want to send them a bill for that day and a half.

Remove checked exceptions?

2007-06-19T02:07:00.000-07:00

Recently, Neal Gafter mused about whether we should consider removing checked exceptions from Java. The motivation from this was not what you might expect, but rather an observation that checked exceptions interacts heavily with a lot of other language features, and that evolving the language might be easier if we were willing to consider removing some features. (Neal knows this won't ever happen, he's just trying to get us thinking about Life After Java.) Not surprisingly, it generated a storm of comments, ranging from "hell yeah!" to "hell no!".

This isn't a new topic; it comes around every few years. A few years back I wrote about the debate surrounding checked exceptions, and the debate continues to rage. My problem is that I think most of the vocal opponents of checked exceptions are objecting for the wrong reasons (back then, I wrote: "My opinion is that, while properly using exceptions certainly has its challenges and that bad examples of exception usage abound, most of the people who agree [ that checked exceptions are a bad idea ] are doing so for the wrong reason, in the same way that a politician who ran on a platform of universal subsidized access to chocolate would get a lot of votes from 10-year-olds").

Reading through the against-checked-exceptions commenters on Neal's blog, we can divide them into three primary groups:

"I don't like checked exceptions because they're too much work."

"Checked exceptions were a nice idea in theory, but using them correctly makes your code really ugly, and I'm left with a choice of ugly code or wrong code, and that seems a bad choice."

"Checked exceptions are a good idea, but the world isn't ready for them." (Frequent refrain from this group: "Man, have you looked at some of the code out there?")

To the people in camp (1), I say: engineering is hard -- get over it. Error handling is one of the hardest things to get right, and one of the easiest things to be lazy about. If you're writing code that's supposed to work more than "most of the time", you're supposed to be spending time thinking about error handling. And, pay for your own damn chocolate.

To the people in camp (2), I have more sympathy. Exceptions do make your code ugly, and proper exception handling can make your code really ugly. This is a shame, because exceptions were intended to reduce the amount of error-handling code that developers have to write. (Ever try to properly close a JDBC Connection, Statement, and ResultSet? It requires three finally blocks. Ugly if you do it right. But, almost no one ever does it right. (The real culprit here is that close() throws an exception -- what are you supposed to do with that exception? But that's fish under the bridge.))

But perhaps there's a way to not throw the baby out with the bathwater, by providing better exception handling mechanisms that are less ugly. Dependency injection frameworks did a lot of that for us already, for a large class of applications -- and the code got a lot prettier, easier to write, and easier to read. AFAICS, the two biggest removable uglinesses of exceptions are repeated identical catch clauses and exception chaining.

The repeated catch clause problem is when you call a method that might throw exceptions A, B, C, and D, which do not have a common parent other than Exception, but you handle them all the same way. (Reflection is a major offender here.)

public void addInstance(String className) {
    try {
        Class clazz = Class.forName(className);
        objectSet.add(clazz.newInstance());
    }
   catch (IllegalAccessException e) {
        logger.log("Exception in addInstance", e);
    }
    catch (InstantiationException e) {
        logger.log("Exception in addInstance", e);
    }
    catch (ClassNotFoundException e) {
        logger.log("Exception in addInstance", e);
    }
}

You'd like to fold the catch clauses together, because duplicated code is bad. Some people simply catch Exception, but this has a different meaning -- because RuntimeException extends Exception, you're also sweeping up unchecked exceptions accidentally. You can explicitly catch and rethrow RuntimeException before catching Exception -- but its easy to forget to do that.

public void addInstance(String className) {
    try {
        Class clazz = Class.forName(className);
        objectSet.add(clazz.newInstance());
    }
    catch (RuntimeException e) {
        throw e;
    }
    catch (Exception e) {
        logger.log("Exception in newInstance", e);
    }
}

My proposal for this problem is to allow disjunctive type bounds on catch clauses:

public void addInstance(String className) {
    try {
        Class clazz = Class.forName(className);
        objectSet.add(clazz.newInstance());
    }
   catch (IllegalAccessException | InstantiationException | ClassNotFoundException  e) {
        logger.log("Exception in addInstance", e);
    }
}

My compiler friends tell me that this isn't too hard.

The other big ugliness with exceptions is wrapping and rethrowing:

public void findFoo(String className) throws NoSuchFooException {
    try {
        lookupFooInDatabase(name);
    }
   catch (SQLException e) {
        throw new NoSuchFooException("Cannot find foo " + name, e);
    }
}

Now, the wrap-and-rethrow technique is very effective -- it allows methods to throw exceptions that are at an abstraction level commensurate with what the method is supposed to do, not how it is implemented, and it allows you to reimplement without destabilizing method signatures. But it adds a lot of bulk to the code. Since this is such a common pattern, couldn't it be solved with some sort of declarative "rethrows" clause:

public void findFoo(String className) throws NoSuchFooException
rethrows SQLException as NoSuchFooException {
    lookupFooInDatabase(name);
}

The rethrows clause is part of the implementation, not the signature, so maybe it goes somewhere else, but the idea is clear: if someone tries to throw an X out of this, wrap it with a Y and rethrow it.

An alternate approach to this would be possible with closures and reified generics; it would be possible to write a pseudo-control construct that said "execute this closure but if it throws X, wrap it with a Y and rethrow it." Unfortunately, with the current state of generics, we can't write such a generic method, we'd have to write a separate one for each exception type we want to wrap.

These approaches focus on the symptom -- because the arguments in group (2) are about symptoms. If we could alleviate the symptoms, people might grumble less.

The people in camp (3) are saying something slightly different. I don't really have an answer for them, because what they seem to be saying is that no matter what mechanism you give people for dealing with failure, they won't follow it. Checked exceptions were a reaction, in part, to the fact that it was too easy to ignore an error return code in C, so the language made it harder to ignore. This works on a lot of programmers who are slightly lazy but know that ignoring exceptions is unacceptable, but apparently is worse than nothing for some parts of the population. (We'd like to take away their coding rights, but we can't.)

Checked exceptions are a pain, and in some frameworks (like EJB before dependency injection), can be really painful. Once the ratio of "real code" to "error handling code" rises above some threshold, readability suffers greatly, and readability is a fundamental value in the Java language design. Even if the IDE generates the boilerplate for you, you still have to look at it, and there's a lot of noise.

On the other hand, my experiences using third party C++ libraries was even more painful than anything Java exceptions have ever subjected me to. Virtually no packages ever documented what might be thrown, so you end up playing "whack a mole" when exceptions did pop up -- and usually at your customers's site. If people are not forced to document what errors their code throws, they won't -- especially the people that the people in camp (3) are afraid of. As long as those folks are allowed to code, the value we get from checked exceptions forcing developers to document failure modes overwhelms the annoyances.

But, as I said above, I think many of the annoyances can be removed by adding a small number of exception streamlining constructs. This doesn't help Neal with simplifying closures, but it does help us get our job done with a little less pain.

Finally, a meta-note -- its really easy to misinterpret the volume of support for "removing checked exceptions" as any sort of gauge for community consensus. We're intimately familiar with the pain that checked exceptions cause; we're substantially less familiar with the pain that they free us from. (Obviously, neither approach is perfect, otherwise there'd be no debate.)

Living in the information age

2007-06-18T12:31:00.000-07:00

While reviewing my household budget recently, I realized that we had truly crossed into the information age -- we pay more for bits than we do for energy. (By bits, I mean both the infrastructure by which information is delivered to us in electronic form, and the content we purchase; by energy, I'm including only my home utility bills, not gasoline, but since I work at home, I'm guessing my gasoline consumption is lower than average.)

Home telephone (basic line + unlimited long distance): $45

Cell phone (mine, including business use): $60

Cell phone (rest of family, 4-line family plan): $110

NetFlix: $20

DirecTV (including TiVo data fee): $75

Rhapsody To Go (music subscription service): $15

DSL: $30

T-Mobile WiFi access plan (reasonable coverage at cafes, hotels, airports): $30

Total: $385/month for bits.

As to fossil fuels, our combined electric and gas bill average out to around 260/mo.

Tivo + HD -- no good choices

2007-06-17T06:38:00.000-07:00

I've been a Tivo addict since the first DirecTivo boxes came out. After getting a big screen TV, the standard definition picture looks pretty bad (especially as it seems that DirecTV compresses the hell out of their signals to make room for more pay-per-view channels.) So we wanted to upgrade to some sort of HD service, but of course Tivo is a must (no third-party DVRs -- no one who has had both a third-party DVR and Tivo has ever said anything good about the third-party DVRs.)

Option 1: DirecTV's HD Tivo. This was released a few years back, but is going to be incompatible with the HD locals that DTV is rolling out, which will be using a different encoding. (Some people have combined this solution with OTA HD, but I have no interest in playing games with antennas.) And DTV has yet to roll out HD locals in this area anyway, and there's little sign they're coming soon. So even if there was HD local channels here, there's no Tivo solution that can record them, only the DirecTV DVR.

Option 2: Digital cable + Series3 Tivo. This seems like the obvious choice, except for the high cost of the Tivo box (600+, plus the increased cost of the Tivo data service which I'd been insulated from since DTV customers were grandfathered in at the low rates). But...no TivoToGo or MRV on the Series3 yet, which means you can't transfer videos to the video iPod. This has to do with content restrictions surrounding their CableCard certification, but annoying it applies not only to protected HD but also to unprotected SD content. Ugh. (As to MRV, if you have two Series3 Tivos with CableCards, why is there a problem moving the content from one Tivo to another?)

Option 3: Comcast DVR with Tivo. Comcast did a deal with Tivo where you can get their Motorola DVR and upgrade to Tivo software as it is rolled out. But its going to be a long time before the rollout reaches here.

MythTV is not an option; you can't put a CableCard in a MythTV box.

So far, no hacks have appeared for the Series3 (other than those that involve reprogramming the PROM and resoldering it) that get around these restrictions.

We're going to bite the bullet and go with Option 2, and hope that eventually Tivo resolves its dispute with CableLabs and reinstates some form of TivoToGo and/or MRV.

Flying has its upsides too

2007-06-04T02:37:00.000-07:00

I was in New York yesterday for my great-aunt's 90th birthday party. The party was over at six, and our flight wasn't until 11, but we figured that New York was a pretty good place to pass a few hours, so we weren't worried. But as the party was ending, it started to rain pretty hard, so we decided to just pack it in and head to the airport, even though JFK isn't the most fun place to kill a few hours.

Wandering around, I passed a half-asleep fellow camped out behind his laptop and thought "That looks an awful lot like Doug Lea". And it was -- he was on a six hour layover on the way home from Amsterdam (yuck). Usually the surprises you get at the airport are of the unpleasant variety, but not always.

Data packrat

2007-05-28T14:51:00.000-07:00

I'm finally moving forward with my plan to get all my life data into digital form. The infrastructure is there (see earlier posts), with lots of redundant disk space and subversion repositories. Now I just have to clean out the file cabinets.

All the various entities that send me monthly statements (banks, brokerages, utilities) are trying to get me onto some sort of "paperless" plan (and, IMO, going about it in a pretty stupid way) by offering some sort of online statements. That's great going forward, but what about existing statements? (And, how long should I really hold on to these records? Can I throw away those tax returns from 1987? Won't they be useful to my biographer?) They all seem to offer some set of past statements in downloadable form; some going back only a few months, some going back seven years.

Most will only let you see the older statements if you agree to let them stop mailing you statements. (You can rescind that agreement at any time, so you know what I did. I don't really _want_ the paper statements if I can have good PDFs, but e-mail is so unreliable that I hesitate to let them use e-mail to send me important notices that might be indicators of identity theft or other bad things.) So I downloaded all the statements I could find, scanned some of the others I thought were worth having, and relegated the paper copies to a box in the basement that, if it got destroyed, I wouldn't be upset.

Not one of the dozen banks, utilities, or brokerages has the statement download thing right. None of them have a "download all my statements" feature, which make downloading seven years worth pretty annoying. (And all are implemented in ways that prevent you from shortcutting around their bad UIs or scripting it yourself.) None offers any sort of scriptable interface for downloading statements, so if you want to continue to gather statements, you have to visit twelve web sites. (I'd like to have the PDFs delivered right into my Quicken; they've been talking about electronic bill presentment for years but I don't see it here yet.) Some make it easier by offering an option to e-mail you the PDF monthly in addition to the physical delivery; some only offer that as an alternative to the physical delivery. Some (Wells Fargo) won't even let you download any e-statements unless you consent to online-only delivery (and the online statements don't have the check images that the physical statements do.) Guess I'll be "consenting" for them five minutes a year to get the past year's statements, yuck.

Bulk scanning turns out to be not so easy with cheap consumer grade scanners. I bought a Visioneer RoadWarrior for receipts and such, but use the scanning features of my HP LaserJet 3050 for bulk scanning because it has a document feeder. But its still pretty slow, and the software sucks. (I'm surprised that the throughput with the RoadWarrior is bound not by the physical scanning speed, but the software that turns it into the appropriate file format and drops it into a drop folder.) So I ended up not scanning everything I thought I would, at least not in the first round. Slowly migrating...

Subterranean data center, part II

2007-05-28T14:31:00.000-07:00

Moving the server and all the network hardware to the basement was great -- it got it out of my closet -- but that presented a problem for the wireless, because the wireless router wasn't strong enough to get a signal up to the back bedroom on the second floor, where we have a squeezebox and need a steady supply of bits. I have a couple of the cheap Linksys routers (I am always surprised at how useful it turns out to have various extra computer parts lying around.) Unfortunately, the Linksys firmware doesn't do what I wanted -- which was to have one box do the gateway router stuff (NAT, DHCP, etc) and another act as a wireless access point. They want you to buy the more expensive Access Point version of the box, which is identical except for the firmware.

So, I installed the DD-WRT firmware on one of my Linksys routers, which lets it act as an access point -- among many other things. DD-WRT is a linux-based distribution for cheap hardware routers, which includes all sorts of networing software not supported by the out-of-the-box firmware (e.g., access point and access point client modes, ipv6, VPN, WPA (client and server), port forwarding, QoS management, SNMP, DMZ, etc.)

As often happens, the road was bumpy but in the end everything worked fine. There are half a dozen different versions of the popular WRTG54, so the instructions might not fit your version exactly. (I learned this on the part where it says "pull firmly to remove the bezel", and my version had screws holding the circuit board to the bezel...and pulling firmly ripped them out.) Despite following all the directions carefully, the first flashing attempt failed, and I had "bricked" my router. I followed the various "debricking" instructions, and eventually had to resort to the most extreme, where you have to short a few pins on the flash chip to restore it to its default state...and eventually I got WRT downloaded into the box. From there, it was smooth sailing, the web-based admin GUI was easy.

Once I had DD-WRT running, it was easy to configure it as an access point -- and if I need better coverage, can just add more. Also, a cheap router + DD-WRT is the cheapest way to put a wired ethernet device onto a wireless network; run it in "access point client" mode. Much cheaper than buying a device designed for this purpose...

Subterranean data center

2007-05-28T14:14:00.000-07:00

I've been getting paranoid lately about data loss. This was almost certainly prompted by a disk failure at my old business that caused some actual data loss. As seems to happen a lot, the disk failure also disclosed a failure in our otherwise sensible-seeming backup program, with the result being that I lost several months of archived e-mail, among other things. Disk failure rates seem to be on the rise; the combination of rising areal densities and the public's clear choice of "cheap" over "reliable" virtually guarantees it. (See, for example, http://australianit.news.com.au/story/0,24897,21553519-15321,00.html.) And with larger capacities, the negative consequences of a disk failure is that much greater.

So, about six months ago I embarked on a domestic data infrastructure program to reduce my risk. This includes:

Relocating my server system to the basement, where the temperature is probably more to its liking (and where additional noise was not going to bother me). This necessitated running lots of Cat 6 cable through the walls; I put a gigabit switch in the basement and ran cable runs to most of the rooms where data would be needed. (Lesson learned: no matter how many cable runs you think you need, run more. Pulling 2 wires is only marginally more expensive than pulling one...)

Attaching a RAID array to the server system.

Getting a hosting provider with reasonable storage limits where I can put some of my data so it is accessible from off my own private network.

Migrating all critical data into SVN repositories.

For the RAID system, I put a 3Ware 9500S-4LP hardware RAID card in my Linux system (about $300). This has four SATA ports. The reason I went with hardware RAID instead of motherboard RAID (sometimes called "fake hardware raid") is that the hardware solution seemed to offer more in the way of hot migration and upgrades. Building my own RAID system turned out to be more of a hassle than expected, mostly because I ended up ordering parts from mutiple vendors because no one carried all the parts I needed. I bought the RAID card and the drives (three 500G drives) for a total of $850. I bought a four-bay enclosure from Addonics. I opted to spring for "multilane SATA", which allows multiple SATA drives to be connected over a single cable; this required adapters at both the enclosure side and the system side, since both the enclosure and the system just had four regular internal SATA connectors. (Running four cables from system to enclosure seemed like it was asking for trouble.) The trickiest part turned out to be getting the right SATA multilane cable; turns out there are two different types of SATA multilane connectors (screw type and latch type), and many enclosures and adapters are vague about which kind they need. So I ended up buying the wrong cable first, and then had to buy the right kind from sataparts.com. Once I got the RAID system physically put together, it was pretty easy. My Linux distro already had the right 3Ware driver installed, and the controller had a nice web interface that let me configure the volume set. With RAID 5, the three 500G drives show up as a 1TB SCSI disk, which I partitioned using LVM.

I could have bought a NAS box, but six months ago the choices were pretty weak. (I suspect this has gotten slightly better.) Would have been less hassle to put together, and maybe cheaper, but I'm sure there would have been compromises too. I'm pretty happy with the hardware RAID solution, and I've got a choice of upgrade paths. (I could throw another 500G in, and have it rebalance the data across four drives giving me 1.5TB, or when the cheap 1TB drives come out, I can pull one 500G out, let the array run in "degraded mode", throw two 1TBs in, create a "degraded" RAID set from them, move the data, then pull the 500G drives and put the third TB drive in giving me 2TB.)

My system is on my home network, which is connected to the internet using via a consumer-grade NAT firewall. So getting out is easy, but getting in is hard. I could have gone the dynamic DNS route, but I chose instead to get a hosting provider for files that I wanted access to from outside. I set up a hosting account at www.textdrive.com, which is great. They make it really easy to set up SVN, WebDAV, etc, so I set up two SVN repositories on my hosted system for files I need roving access to (such as presentation slides, in case I get to a conference and my laptop doesn't.) I set up two because SVN doesn't have good support for actually removing things from repositories, so they tend to grow over time. So there's a "permanent" and "transient" repository; the transient repository is for short-lived projects where after some point I won't need the history any more. SVN turns out to be a reasonably nice solution for accessing the same file from multiple systems, since I tend to either be at home and use my desktop system exclusively, or be on the road and use my laptop exclusively.

I decided to get all my data into SVN, after being inspired by this article from Jason Hunter. Even for data that you don't think is ever going to change, like photos (hey, what about photoshop?), SVN turns out to be a pretty good solution. If you get a new computer, you can just do one checkout and all your data is there. Keeping an up-to-date checkout on your home and laptop systems (in addition to the server) mitigates a number of data loss scenarios. I'm not there yet -- I'm still migrating, but I'm making progress.

The big question mark now is the backup strategy -- backing up a terabyte is pretty hard.

Squeezebox update: iTunes support redux

2007-05-24T11:20:00.000-07:00

Well, the vbrfix program didn't work quite as advertised, and borked the mp3s I ran it on. Fortunately, they had all been converted from FLAC, so restoring was simply a matter of re-running the flac-to-mp3 script (which does take a while to chew on 300G of music.) But the "Fix MP3 Header" option in "foobar2000" does the trick, and now iTunes is happy with my MP3s. But since fb2k is a Windows app, it means that the incremental conversion process when new FLAC files are added has a manual step, rather than one I can script.

JCiP best seller at JavaOne 2007

2007-05-17T04:35:00.000-07:00

For the second year in a row, Java Concurrency in Practice was the best selling book at the JavaOne bookstore...thanks everyone!

At JAX in Wiesbaden this week

2007-04-23T19:41:00.000-07:00

I'm at the JAX conference, in Wiesbaden, Germany this week (www.jax.de.) I'll be speaking on Wednesday evening on (what else) concurrency and performance.

Squeezebox update: iTunes support

2007-04-15T14:25:00.000-07:00

There are lots of tools for dealing with audio files of various formats (MP3, FLAC, AAC, Ogg, WMA, etc), but most of them fall down when it comes to letting you deal with an entire library of music files. (I suppose some of them must be good, but there are so many, you get tired of trying them.)

I've mostly fallen back on scripting-based approaches. When I rip using EAC (which rips to WAV, which has no tagging features), I instruct EAC to create a file that has all the relevant tags (title, artist, genre, etc) embedded in the file name, and I've got a script that feeds these into the flac encoder, creates files using a directory hierachy of the form /artist/album/track.flac, and sets the appropriate flac tags.

I downloaded another script (http://robinbowes.com/projects/flac2mp3) that takes this directory and creates a parallel directory of MP3 files, transcoded from the flac. It uses LAME to do the encoding, using the "--alt-preset standard" settings, and I point iTunes at that directory. So far, so good.

Well, iTunes seems confused by VBR (variable bit rate) MP3s that it didn't encode itself. The symptom is that iTunes thinks the tracks are way longer than they are.

After much searching, the best alternative I came up with was to use a program called 'vbrfix' (http://www.willwap.co.uk/Programs/vbrfix.php), which rewrites the mp3 headers in a way that iTunes is happy with. (It claims to fix other problems too. The documentation sucks, and I had to download the source and build it (with no make file, just guessed, but it wasn't hard), and it does appear to render the MP3s compatible with iTunes.

Interview on Software Engineering Radio

2007-01-28T01:41:00.000-08:00

Tune in to Software Engineering Radio to hear the interview Marcus Voelter did with David Holmes and I at OOPSLA 2006.

JCiP named Jolt Award Finalist

2007-01-16T15:49:00.000-08:00

2006 Winners and Finalists

JCiP was named a finalist in the 2006 Jolt Awards in the technical books category.

Mama's got a squeezebox...

2007-01-12T07:30:00.000-08:00

A number of friends have asked about my Squeezebox home audio setup, so rather than repeat myself, I'll write it up here.

Squeezebox is a device that you attach to your home network and your stereo, streaming digital music to your stereo system. There are lots of such devices on the market; Roku's Soundbridge, Apple's AirTunes, and offerings from networking companies like Netgear and Linksys. I chose Squeezebox because it has a digital (optical) audio out, meaning that what gets piped into my expensive stereo is not going through the analog stage of a $20 sound card. The model I have has a wireless (802.11g) interface and a wired ethernet port as well; I paid about $250 each, and we have them in multiple rooms so you can listen to any music from any room without fussing with physical CDs. (You can even sync them so you can have the same music throughout the house, say for parties.)

My primary motivation for this transition was that I hate CD furniture; it's mostly ugly, and takes up an obscene amount of room in your living room if you have any reasonable size music collection. Even transferring from jewel cases to sleeves, which gives about a 2.5:1 space compression, CDs can overwhelm your living room.

The first challenge: ripping the CDs. This is the most time consuming step, so I did not want to have to do this again because I had chosen the wrong audio format. There are three components to ripping: the audio extraction, the association of metadata (artist, title, genre) with albums and tracks, and conversion to the format of choice (mp3, wma, etc.) As with many other situations, you can get all-in-one solutions like iTunes or Windows Media Player that will do all of these in a single step, or you can have more control over each step but have to deal with multiple programs.

It turns out that nearly all rippers do not take advantage of the error-correcting information present in CDs. So if you have a scratched CD, you'll get bad bits when you rip, and those bad bits will stay with your recording forever. The only Windows-based tool I know of that will use the error correction is EAC (Exact Audio Copy). It's not as slick as iTunes, but it will usually get you a perfect rip. For CDs in good condition, it can usually rip at 10x, ripping a whole CD in 6 minutes or so. If it detects bit errors, it will slow down and keep reading until it is satisfied; for one really badly scratched CD (one that wouldn't even play in my car), it chewed for 24 hours, and got all but ~1000 bits off!

EAC will rip to WAV, and has a mechanism to post-process to further launch an external converter (mp3, wma, etc), which I did not use. Instead, I saved the WAV files to disk and post-processed them separately. iTunes and WMP can access the commercial CDDB database that associates metadata (artist, album, track names) with albums and tracks; EAC uses the open-source freedb database, which is convenient but whose data quality is less than perfect. Expect to spend some time correcting titles and genres that don't match up (e.g., the first CD of a set is called Volume One, where the second is called Disc 2, or one volume lists Rock as the genre, where the other lists Pop). You can do this through EAC or using an ID3 tag editor, but in any case, expect to spend some time cleaning up the data.

For my storage format, I chose FLAC, the open-source lossless audio compression, which stores files in about 55% of the space of the WAV file. This about about three times bigger than a good VBR MP3 or AAC, but disk space is cheap -- real cheap. (As of this writing, 500G drives are going for less than $200.) And the time to re-rip is very expensive. I set up the ripper on Windows to write the output files to a drop folder on my Linux server (named using a convention that embeds the track, artist, album, and genre, since WAV doesn't support metadata tags), and have a home-grown perl-script (willing to share, just ask) that will find the files and feed them to the flac converter.

Squeezebox versions 2 and later support FLAC native, so it doesn't have to transcode to MP3 on the fly. This is nice because the transcoding interfers with fast forward / rewind functionality on the Squeezebox. So, following the chain, error-free RIP courtesy of EAC, lossless conversion to FLAC, digital transfer from server to squeezebox, lossless FLAC decompression to PCM on squeezebox, digital out to receiver -- meaning no end-to-end signal loss, and digital-to-analog conversion done by my receiver. Just as if I'd plugged the CD player's optical out into the receiver.

For the server software, the free SlimServer package is written in Perl so can run on Windows, Linux, or Mac. I chose Linux since I did not want to downgrade the reliability of my stereo to that of my Windows desktop. (I have a Linux server in the house anyway, but if you don't, you can build one fairly cheaply.)

If you want to transfer to your iPod or other device, you need to transcode from FLAC to MP3 or AAC or WMA or whatever your favorite portable format is. The best MP3 encoder is called LAME (open source); you then have to decompress from FLAC to WAV, and pipe that into LAME to get an MP3 out. (I believe iTunes for Mac has a LAME plugin, but not iTunes for Windows.) LAME encoding using VBR (variable bit rate) takes a while. Disk space is cheap enough you might consider an automated nightly script to encode all new FLAC files into a parallel tree of MP3 files for transfer to iPod, if iPod is a big enough part of your life.

Once you get all the ripping done, it's pretty nice. It took me about a week to rip ~400 CDs "in the background" while I was working. Thereafter, the only time you need to find the physical CDs is if you want to play them in the car. And the SlimServer software has a web interface that lets you create playlists and such, so you can set up playlists for parties so you don't have to be fussing with CDs.

Highly recommended. We've got two squeezeboxes now (living room and bedroom) and are considering adding more (kids room, family room). Plus there's a software player you can use on the computer.

Microsoft humor

2006-12-30T08:32:00.000-08:00

Actual ad from Battlestar Galactica home page.

Reader mail: Urban performance legends

2006-12-30T08:14:00.000-08:00

In response to Urban Performance Legends, Revisited, an anonymous reader wrote that I was "too enthusiastic" about the JVM's ability to inline virtual method calls:

Moreover, the example provided is misleading: inlining in java can be generally done for private or final methods. Non-final public and protected methods can't be automatically inlined because they can be overidden by a subclass. The actual type of the object may be unknown at compile time and at load time. The only way to inline them is to analyse the code to find every use of the class and perform type inference. This can be done sometimes with inner classes that have limited scope, but not with public classes that must be compiled indipendentely from each other.

Unfortunately, this is a common myth about optimization in Java (and other dynamically compiled languages): that if a method could be overridden by a class that has yet to be loaded, then the compiler cannot optimize away the virtual function call. This is just plain wrong (as is the rest of what this reader says), and today's JVMs can devirtualize and inline through virtual calls using a number of techniques.

One such technique is called monomorphic call transformation, where the compiler observes that for a given method foo(), there is no class loaded right now that overrides it, so calls to foo() can be compiled (speculatively) as direct calls instead of virtual calls. If a class is loaded later that makes foo() polymorphic, the compiler can invalidate the speculatively optimized code. This is covered in detail in Dynamic Compilation and Performance Measurement. There are other techniques as well, such as inline virtual caching.

The myth that final has any effect on method invocation performance for monomorphic methods was well-exploded by Cliff Click's 2003 and 2005 JavaOne presentations.

Java (as well as other managed languages, like C#) is not C. Dynamic compilers are smarter than you think.

Reader mail: making tasks noncancelable, and polling for interruption

2006-12-30T07:53:00.000-08:00

I frequently get e-mail feedback on my articles on IBM developerWorks. Unfortunately, I can rarely reply to them because no one ever leaves their e-mail address. The majority of e-mails I get are erroneous corrections (not that I don't make mistakes -- I make plenty, and those get pointed up too -- its just there are even more people out there who are really really sure they're always right, but aren't), which I'd be happy to respond to if anyone ever left their e-mail... Here's one from Dealing with InterruptedException. Listing 6 shows an example of how to make an operation noncancelable by deferring interruptions until the operation completes.

public Task getNextTask(BlockingQueue queue) { boolean interrupted = false; try { while (true) { try { return queue.take(); } catch (InterruptedException e) { interrupted = true; // fall through and retry } } } finally { if (interrupted) Thread.currentThread().interrupt(); } }
The anonymous reader asks: shouldn't this be "while (!interrupted)"? Won't the loop never terminate? The idea behind this example was to address the problem of "what if we have an operation that should not be interruptible, but is composed using interruptible steps?" We will have to ignore the interruptions when they occur and retry the interrupted operation -- but the key is we want to remember if there has been an interruption, so we can restore the interrupted status after we complete our noncancelable operation. That way, we don't throw away the information that someone requested cancellation, we just defer the cancellation request until after the noncancelable operation operation completes.

A related issue that is often asked is how often we should check for interruption. Listing 3 shows a typical interruptible operation wrapped in a Runnable. When this happens, in order to not throw away the evidence that an interruption was requested, we have to re-set the interrupted status with Thread.currentThread().interrupt().

public class TaskRunner implements Runnable { private BlockingQueue queue; public TaskRunner(BlockingQueue queue) { this.queue = queue; } public void run() { try { while (true) { Task task = queue.take(10, TimeUnit.SECONDS); task.execute(); } } catch (InterruptedException e) { // Restore the interrupted status Thread.currentThread().interrupt(); } } } It is often asked: shouldn't the loop header be while (!Thread.currentThread().isInterrupted()), instead of while(true)? This one is a little more subtle. We can look at this from several perspectives: correctness, responsiveness and performance.

From a correctness perspective, the existing code is correct, because (reasonably coded) interruptible blocking methods check if the interrupted status is set on method entry, and immediately throw InterruptedException() if it is. Similarly, the existing approach is effectively as responsive as the suggested alternative, because the first step in the while loop is to call an interruptible blocking method. (If substantial computation occurred between loop entry and the first interruptible blocking method call, responsiveness might give us a reason to test the interrupted status in the loop header as well -- and maybe other places throughout the loop too.)

What about performance? As with most performance questions of this sort, the answer is "We don't have enough information to tell". Testing the interrupted status in the loop header costs a little more (because the test is repeated in the loop header and in the blocking take() call), but in the case that the interrupted status is set, saves us the cost of instantiating and catching the exception thrown from take() when it finds the interrupted status set on entry. Which is a performance win will depend on how often the operation is actually interrupted -- and we usually don't have this information when coding general-purpose library code.

When in doubt, do the thing that makes the code simplest, cleanest, and most readable.

Farewell Quiotix, hello Sun!

2006-09-10T04:26:00.000-07:00

I've been self-employed for fifteen years; I founded Quiotix in 1992 to pursue some short-term consulting opportunities after getting laid off from my last job, and I never looked back. I like the flexibility, and I don't mind the stress of not knowing what my income is going to be in any given month. I like not having to ask permission to take a day off, or buy a new monitor, or attend a conference that looks interesting. So in the past, when offered full-time positions, my response has usually been "Why would I want that?" (This can be very perplexing to someone offering you a job; they're used to getting a very different response.)

So, those of you who know me well may be a little surprised to learn that as of September, I will be joining Sun Microsystems as a Sr. Staff Engineer / Technical Evangelist in the Java SE engineering organization. I'll be involved in a lot of things, including a lot of the same things I've been doing as an independent -- like writing technical papers on JVM internals and speaking at conferences. I'll also be involved with the Java SE engineering, QA, and education efforts. I have a feeling I'm going to be pretty busy.

To the customers, colleagues, business partners, and employees I've worked with at Quiotix, I offer thanks for fifteen good years and the best of luck in the future.

Nerdy fun

2006-08-09T11:39:00.000-07:00

Memories of iceberg metaphors past

2006-06-15T13:40:00.000-07:00

In his Dancing About Architecture blog, Eric Lunt calls JCiP a "two copy" book -- one for home, one for work. I like the way he thinks.

He describes concurrency as an "iceberg" subject:

Concurrency in Java is a very difficult, iceberg subject, and I guarantee your intution will fail you more often than not.

And he's more right than he realizes! At one point we were going to put an iceberg on the cover, and the introduction was even titled "Under the waterline" (though the metaphor does not appear in the final version of the book at all.) But then we realized that leading with something so depressing might not be the best strategy.

From an earlier draft of the introduction:

Icebergs typically have only ten to twenty percent of their volume above the waterline ??? the rest is hidden from view, lurking underwater. Concurrency is sort of like an iceberg ??? the explicit elements (synchronization, notification) are above the waterline and are clearly labeled, but in order to avoid an unpleasant collision, you must also be aware of what is below the waterline ??? and there???s a lot down there. Below the waterline lies all of the implicit policies and agreements that govern how these mechanisms are intended be used and mean in specific situations within your program. While it may sound trivial, it is not enough to simply use synchronization to ensure thread safety ??? you have to use synchronization correctly and consistently. Describing how to craft and document sensible under-the-waterline policies for the effective use of Java???s built-in concurrency constructs is the central goal of this book. Fortunately, it is not prohibitively difficult to develop a strategy that encompasses the submerged elements ??? as long as you take them into account from the beginning.

All concurrency constructs (such as synchronization, notification, or interruption) involve more than one thread. One of the reasons that writing and maintaining concurrent code in Java is so difficult is that all of the major concurrency constructs involve some degree of implicit agreement between multiple parties as to how the concurrency construct is to be used or interpreted . The concurrency constructs provided by the language are low-level mechanisms, but they are used in applications to enforce application-level concurrency policies or protocols. The tricky part is that policies are defined by the developer and can vary across programs or classes; the same mechanism can be used to enforce a variety of policies. These policies are not captured anywhere in the code (if you???re lucky, it???s captured in documentation, but more often it is not), and therefore it is difficult to automatically (or manually) verify or validate that a program is using the concurrency mechanisms correctly.

Each of the mechanisms for concurrency ??? such as synchronization, notification, and interruption ??? involves cooperation between two or more threads, and each involves an implicit and explicit component. With synchronization, the act of taking and releasing the lock is explicit, but the reason for taking and releasing the lock ??? because the lock guards a particular set of shared state variables ??? is implicit. Similarly, the act of waiting or notifying on a condition variable is explicit, but why you are waiting ??? that you are waiting for a specific condition predicate to become true ??? is implicit.

Summer backyard fun

2006-06-08T04:20:00.000-07:00

Check out http://eepybird.com/dcm1.html, for an amazing demonstration of what happens when Mentos and Diet Coke mix, along with an example of what happens when geeks have too much time on their hands.

I tried this in the backyard with the kids the other day, and it works! I found that the "crown" technique, where you punch a ring of holes around the top of the bottle, works particularly well (got a big cheer from the neighbors.) The "coke hose" technique works well too, where you don't bother with the cap and just dump some broken mentos in, but you have to do it quick...

I thought the video spoke for itself, but apparently it didn't, as I got this e-mail from someone who shall remain nameless (but whose initials are WMR) but who should have known better...

You failed to mention that this should only be done outdoors...OOPS! BTW, this really does work! Now, if you excuse me, I have a huge mess to clean up in the kitchen... :(

JCiP sells through its first printing

2006-06-08T03:43:00.000-07:00

Java Concurrency in Practice sold through its first printing (5000 copies) in less than a month. If you're waiting for your copy, be patient -- more are being printed!

Rumor has it it that Chinese and Japanese translations are underway.

JCiP best-selling book at JavaOne!

2006-05-20T13:21:00.000-07:00

Java Concurrency in Practice was the best selling book at the JavaOne bookstore!

JavaOne, day 1

2006-05-16T13:16:00.000-07:00

JavaOne seemed better attended this year than last; business conditions are picking up. Once again, they rewrote the Schedule Builder application, and once again, it sucked. How hard is it to write a data-driven web application? Shouldn't that be a slam-dunk for Sun?

In an attempt to manage class sizes better, signup for each talk was mandatory. This means they had to scan your RFID badge for every talk, which created a huge bottleneck as there were typically only four scanning stations and they wanted to move a thousand people into the room in a five to ten minute window. No one seemed happy about this.

India wrap-up

2006-05-09T13:14:00.000-07:00

Overall, I didn't get a chance to see as much as I would have liked; the already compressed schedule got further compressed by arriving a day late, and the very hot weather (42-45).

I was _very_ impressed with the caliber of students in my class. They were bright, well educated, and motivated -- this made for a great class experience, and we had a lot of fun. The food was good, and all the people I met (well, except for the shopkeepers and hawkers) were very nice. Indians seem very protective of Westerners; they expressed
constant concern that the weather or the food might be hotter than I could take.

I was truly impressed with the industriousness and resilience of the Indian people. No matter how poor, it seemed everyone was working hard to raise their situation. Everyone is a businessman; we could learn a thing or two about capitalism from them. And my god, there are a lot of people there. Everywhere you went, even outside the city, it seemed to have the population density of Grand Central Station.

The roads are mind boggling. A mix of cars, scooters, bikes, bicycle taxis, mini taxis, and cows, all going different speeds and sometimes different directions. Clearly they play this game by different rules.

I took a day trip to Agra and saw the Taj and the Agra Fort, and got one afternoon in Delhi to do some additional sightseeing and shopping. Unfortunately, I managed to get a spot of Delhi Belly on the last day, which lessened my motivation to leave the hotel for further sightseeing. And man, was it hot. 42-45C (110-115F) in the shade. Power outages were frequent; every day the paper reported the net power shortfall.

Dysentery roulette

2006-05-02T22:10:00.000-07:00

The health advisories I read before leaving all warn strongly of avoiding the local water supply. Of course, this is nearly impossible -- dishes are washed in water, food prep staff washes their hands before touching the food (you hope), etc. So while its easy to say "don't drink the water", in reality it is pretty hard not to. So every time I eat something, I am playing a game of "dysentery roulette" -- it might be yummy, it might make me sick. Kind of makes everything just a little more exciting.

Something surprising

2006-05-02T03:29:00.000-07:00

Today, I saw a man riding an elephant on a city street! I'm not in Kansas any more...

Arrived, eventually

2006-04-30T14:35:00.000-07:00

Well, the best laid plans...

I was supposed to get onto the nonstop from NY to Delhi, but the feeder flight from Burlington was delayed 2.5 hours and it was clear that I was not going to make my connection. So they rebooked me for the next day, but at that point the nonstop was full, and I had to be booked onto a connecting flight with a 4hr layover in NY and a 4h layover in Amsterdam at 7AM (and which got into Delhi several hours later.) So, instead of the 17h trip I planned, I effectively had a 44 hour trip.

Once on the ground, things went smoothly. There were about 300 drivers waiting with name placards outside of baggage claim, but I eventually found mine and got to the hotel around 1:00AM local time. The hotel facilities and the service are very nice.

Traffic here is mind boggling. The roads are a free-for-all with cars, trucks, motorcycles, bicycles, pedestrians, and cows, often going vastly different speeds and sometimes different directions.

Heading to India

2006-04-26T12:42:00.000-07:00

I'm heading to India on Friday, to teach my Java for C++ Programmers class. One of my training customers has offices all over the world, and the India office is next on the rotation. I feel like I've won the outsourcing lottery -- outsourcing is now putting money in my pocket.

Fortunately, my customer is springing for a business class ticket (which is effectively a first class ticket, as the only airline that runs a nonstop from the US to Delhi, Continental, merged their business class and first cabins into "BusinessFirst", and on the international flights BusinessFirst compares to first on other airlines.) The nonstop NY-DEL is approximately 14 hours, which shaves 5h off the "change planes in Europe" options.

A business class ticket to India costs approximately 6K; a coach ticket costs 1-1.5K. If I had the choice of buying the coach ticket and pocketing the difference, would I? Not sure. On the one hand, I wouldn't pay 4.5K out of my own pocket to upgrade, but on the other hand I might just not go if it meant flying coach. It's a long trip.

Getting a visa turned out to be somewhat of a hassle. Apparently, standard procedure for getting a visa involves sending your ORIGINAL passport to the foreign consulate, and hoping that they send it back in time (and don't lose it.) I used an expediter service to process it, which reduced the hassle factor, and they also rented me a local cell phone to take with me, which was pretty convenient.

After the training is over, I should have about three days to sightsee. That gives me a day each in Agra, Jaipur, and Delhi. I don't know what to expect, but it should be interesting.

An even better quote

2006-04-18T08:10:00.000-07:00

From the preface of Pragmatic Ajax (Gehtland, Almaer, and Galbraith), on book writing:

"Writing a book is a lot like (we imagine) flying a spaceship too close to a black hole. One second you're thinking "Hey, there's something interesting over there" and a picosecond later, everything you know and love has been sucked inside and crushed."

Indeed.

The beast is slain

2006-04-17T14:55:00.000-07:00

I turned in final PDF today for Java Concurrency in Practice. Books should be ready by JavaOne. Woohoo!

The final tally: 425 pages, and sixteen months.

Not quite sure what to do with myself now.

Churchill on writing

2006-03-03T12:49:00.000-08:00

Ted Neward pointed me to this quote the other day, which I'd not seen before, but which characterizes the writing process quite well:

Writing a book is an adventure. To begin with, it is a toy and an amusement; then it becomes a mistress, and then it becomes a master, and then a tyrant. The last phase is that just as you are about to be reconciled to your servitude, you kill the monster, and fling him out to the public.
Winston Churchill

I'm gaining the upper hand over the monster. Public humiliation of it, or me, is imminent.