Showing posts with label software. Show all posts
Showing posts with label software. Show all posts

Saturday, January 9, 2010

zTunes released

As promised in yesterday's entry, I'm releasing my digital media management software (ztunes) to the world.  It's hosted at github: http://github.com/briangoetz/ztunes.  It is written in Ruby and based on "rake", the Ruby equivalent of "make".  It currently has a long way to go but already does a lot. 

You can download the Ruby gem here: http://github.com/briangoetz/ztunes/downloads.  (It is not currently in any sort of gem repository.)  It defines its gem dependencies, but you'll also need the Unix tools ffmpeg, flac, and lame.  It will run on Linux and Mac but currently has some trouble on Windows since it is dependent on symbolic links for some of its functionality, which Windows doesn't support.

My motivation for writing this was that iTunes is really inadequate for managing a media library unless (a) you only want to play on iPod (or other Apple) devices and (b) you are willing to let iTunes be in control of ripping and encoding.  This didn't work for us for two reasons: we have Squeezeboxes on all the stereos, and I want to rip my CDs to a non-proprietary, lossless format (that means flac, which iTunew doesn't support.)  We also have music that has been aquired in various other forms (MP3s from Amazon, AAC from iTunes, WMA from Rhapsody) and want to be able to play all the music on all the devices, without transcoding it all down to a least-common-denominator.  (In other words, if Squeezebox supports WMA but iPod doesn't, let Squeezebox play off the original WMA but let iPod play the transcoded version.)  And this should be transparent to the rest of the family.

There are several basic tasks in managing the media library that zTunes automates:

  • Content ingestion.  I've got a "drop" folder, into which I want to drop the originals of my media, in whatever form, and have them be analyzed, metadata extracted, and filed into a unified library based on its metadata.  My metaphor here is the gas tank of an M1 tank: you can pour anything combustible (gasoline, jet fuel, diesel, used cooking oil) into the tank and it figures out how to burn it.  Currently it maps a media file to a filename by using the author/album/title tags for audio or the title tag for video; audio files are named like "The Who/Who Are You/Squeezebox.flac". 
  • Transcoding.  Not all devices play all device types.  So ingested content also needs to be transcoded into alternate formats, which are maintained as parallel directory trees.  The transcoded trees are transient; they are merely shadows of the "authoritative" tree.  Some files may need be transcoded to multiple formats; for example, video files ripped from DVD or transferred from TiVo might be transcoded to 480 x 320 video for iPhone but 320 x 240 for the older video iPods. 
  • Syncing.  I use the Windows program "Tag&Rename" to edit the metadata tags on my media files, to normalize genres, naming details like "The Cars" vs "Cars, The", "Vol 1" vs "Disk Two", etc.  When I edit the metadata on an "original" file, I'd like the file to be renamed accordingly, and metadata changes to be reflected in the transcoded copies.  When I delete an original, I want the transcoded copy to go away.  Etc.
  • Device management.  I would like to have a single directory for each device type, that I can point device-specific library management software (iTunes, Squeezecenter, Creative Explorer) at, and it will see the right view of the media library for that device (will only see files it can play; will see them in the "best" format available for that device.) 

One thing it does not do yet is manage the integration of your external media library into iTunes (iTunes is particularly bad at dealing with files you didn't acquire through iTunes.) 

See more in the README file here: http://github.com/briangoetz/ztunes/blob/master/README

I'm currently using this to manage a library of ~8,000 media files in half a dozen formats.  I'd love to get some more users -- drop me a note if you're interested! 

Friday, January 8, 2010

A busman's holiday

Since I've been working way too hard, of course I decided to spend my XMas break...programming.  (http://www.answers.com/topic/busman-s-holiday).  I had two goals: rewrite my digital-media handling software, and learn Ruby.  I'm pretty happy with what I accomplished on both counts.

The motivation to rewrite my digital-media scripts came from having too many conversations like the one below with Stuart Marks:

SM: Hey, you wrote a bunch of scripts to manage audio and video files, are you willing to share them?
BG: Well, in theory, yes.  But I'm kind of embarassed to show them to anyone...
SM: Let me guess.  Perl?
BG: Yep.
SM: I have a Perl story...
BG: Don't bother -- all Perl stories end the same way.

I'll post the full details soon -- including links to the software on github -- but for now I'll just outline the problem I was trying to solve:

  • Ingest digital media files in any format (MP3, AAC, WMA, WAV, FLAC, M4A, M4V, WMV, MP4, etc)
  • File them into a library based on their metadata
  • Additionally transcode them down to one or more "compressed" formats (MP3 for audio, iPhone-sized Mp4 for video) for memory-constrained devices, without letting go of the original
  • Organize them so that each device (iPod, Squeezebox, non-iPod MP3 player) can play all the media, in the best format that the device can recognize natively (Squeezebox supports MP3, WMA, and FLAC; iPod supports MP3 and AAC; Zen supports MP3 and WMA) or a transcoded form if it can't.  For example, for a given track whose source form is WMA, Squeezebox and Zen should see the WMA but iPod should see the MP3; for a track in FLAC, Squeezebox should see the FLAC but iPod/Zen should see the transcoded MP3. 

 

 

Wednesday, October 15, 2008

My favorite computer science book



Pierce's Types and Programming Languages is a masterful introduction to the theory and practice of type systems.  One of the things that makes this book so great is that it is equally accessible to both the theory-oriented and the practice-oriented.  This was driven home to me in a conversation with Ola Bini, when I saw he was carrying this book, and he commented "I love this book because I can skip all the math and get what I need from the ML implementation."  I answered that I liked it for the opposite reason; I was able to get everything I needed from the math and didn't have to look at the code.  Its pretty impressive that a book can be that useful and successful from two such radically different reader approaches.

I found that Pierce's treatment was extremely accessible.  He starts with almost no assumptions, introduces first the untyped lambda calculus, then the simply typed lambda calculus, some obvious extensions (records, references, subtyping, union types, functional objects, etc), operational semantics, and builds gradually to more useful type systems.  Each section includes motivation, analysis, a formal description of the system, soundness proofs, and ML code; the impatient can skip some of these and still get what he's talking about.  There is working code for each of the languages developed.  (The type systems were developed in a system that the author wrote called TinkerType, which makes it possible to build type systems by "mixing and matching" features, and it generates both the ML code and TeX source for generating the figures used in the book -- most impressive!)

Not only is this book useful to anyone who is interested in the design and science of programming languages, but it is also a pleasure to read.

What's your favorite computer science book?  (Unoriginality points for anyone who says TAOCP.)

Saturday, May 31, 2008

The making of JCiP; avoiding errors in code listings

At JavaOne this month, my publisher at Pearson, Greg Doench, hosted a panel/BOF entitled "Writing the Next Great Java Book." Not surprisingly, when you get four authors in a room, you get seven opinions about what's important in writing a book!

One thing that we (mostly Tim, actually) did for JCiP was set up an infrastructure for the book, similar to what you'd do for a software project. Version control, issue tracking, one-step build script, continuous build -- all of these things offer the same benefits to book projects as they do for software.

One critical aspect of the build is the handling of program listings. It is incredibly tempting to cut and paste examples from the IDE into whatever source format you're writing in (Word, Frame, LaTeX, DocBook), but this is a recipe for disaster -- errors will invariably creep in as you try and make small tweaks (such as changing variable names) outside the IDE. And code examples with errors really undermine the reader's confidence (or worse, they copy the incorrect example into their code.) So, we wanted to make sure that every code example compiled (and ideally, was tested.)

Our approach was to check the code into Subversion with the rest of the book artifacts, ensure that the build process compiled the code and ran the unit tests, and then automatically extract the examples from the code in a format into which they could be directly included by the build. Some systems (LaTeX, DocBook) make this sort of inclusion easier than others.

We marked the examples up with comments for formatting (bold, italic) and also with "snip here" comments that excluded the irrelevant portions of the code from the listings that actually went into the book. The attached perl script (phragmite.pl), written by Tim Peierls (based on an approach designed by Ken Arnold), takes as input a set of input files and produces a set of LaTeX files representing the extracted listings.

As an example, here is the Counter listing from Listing 4.1 of JCiP:
// !! Counter Simple thread-safe counter using the Java monitor pattern
// vv Counter
@ThreadSafe
public final class Counter {
/*[*/@GuardedBy("this")/*]*/ private long value = 0;
public /*[*/synchronized/*]*/ long getValue() {
return value;
}
public /*[*/synchronized/*]*/ long increment() {
if (value == Long.MAX_VALUE)
throw new IllegalStateException("counter overflow");
return ++value;
}
}
// ^^ Counter

The first line identifies the type of the code fragment (!! for a "good example", ?? for a "bad example" which would get decorated with a Mr. Yuk), the name of the fragment (Counter), and the listing caption. The lines with the ^^ and vv mean "snip from here to here", and a listing can be made of multiple such fragments. The /*[*/ and /*]*/ comments mean "bold". The following ANT target ran the script:
  <target name="listings">
<exec dir="${bin.dir}" executable="perl">
<arg value="${phragmite.pl}"/>
<arg value="${listings.dir}"/>
<arg value="${fragments.dir}/*.java"/>
<arg value="${fragments.dir}/jcip/*.java"/>
</exec>
</target>

In the book's LaTeX source, we use the following LaTeX macro to pull the listing in:

\newcommand{\JavaListing}[1]{\input{listings/#1}%

Monday, December 10, 2007

A day and a half in the life...

This is a story about Windows, Samba, debugging, and frustration. I just spent a day and a half debugging a very annoying performance problem with a brand-new system.

Background: I just recently bought a new desktop computer, which I do about every two years. Computers should last longer than that, but I find that, even with good "hygeine", Windows systems tend to decay to the point where they exhibit weird behaviors after about two years, for which the cure is a complete reinstall of the OS and all applications. (Pause for Mac fanboys to snicker.) The "rebuild the world" process wouldn't be so bad if it weren't so hard to migrate all one's data -- even given the fact that a lot of my data is already in subversion. The real problem is that each application sprays its configuration and data randomly around your system, whether in the program install folder, registry, documents and settings folder, local settings folder, etc.

So, I bought a new computer, an extra-quiet one from www.endpcnoise.com. These guys specialize in quiet systems, and since I work in a home environment, the computer is usually the noisiest thing in the room. (Of particular annoyance is the variable speed fan on my existng Dell system, which, every time the system worked up a sweat, made a whiny noise. And we was fined $50 and had to pick up the garbage in the snow, but thats not what I came to tell you about.) Pretty happy with the new system overall.

So, I installed XP Pro on the new system, and proceeded to install all my applications, utilities, and all kinds of groovy things that we were talking about on the bench. And then I got to the part where I tried to use it; specifically, tried to fire up IDEA and build the project I'm working on.

Now, I've got a somewhat weird setup; when developing on a project, I checkout a workspace on my Linux server, which is served via SAMBA to my local network, and I run the IDE on my Windows desktop and point the IDE at my samba share. There's a measurable performance hit vs local, but I like being able to do some things from the Linux command line and other things from the IDE, so overall its a more productive setup for me.

When you ask the IDE to "make" the project, it crawls the files in the project checking their modification times. An "empty" make on a project with ~1000 files generally takes a few seconds to figure out that there's nothing to do. But when I set it up, the new (faster) system took about 30-60s to do an empty make.

OK, it's debugging time. What's different about the two systems? Same OS (XP Pro), same service pack level, both systems up to date on patches, same Java version, same IDE version, same user credentials, no host-specific information in my SAMBA config. Different hardware. Make sure my ethernet drivers are up-to-date. Test network for errors, swap cables, all that. Run IOMeter, found that both get similar throughput for large files on the same SAMBA share.

Crank up perfmon, which tells me that the new system is sending out more packets for a make than the old one. OK, crank up ethereal, get a packet capture, and find that the new system is sending/receiving 10x as many packets for the same operation:

[brian@brian-server ~]$ wc -l /tmp/*cap
258204 /tmp/new-cap
17719 /tmp/old-cap


So, what's the difference? Let's look at the packet capture. In the old trace, for each file being probed, it did something like this:

0.467895 192.168.1.104 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Basic Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
0.468041 192.168.1.107 -> 192.168.1.104 SMB Trans2 Response, QUERY_PATH_INFO
0.468283 192.168.1.104 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Network Open Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
0.468402 192.168.1.107 -> 192.168.1.104 SMB Trans2 Response, QUERY_PATH_INFO


Two requests, two responses per file. Seemed reasonable. On the new system, for each file:

2.010471 192.168.1.113 -> 192.168.1.107
SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree
2.010698 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2754
2.010900 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \
2.011011 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2755
2.011237 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work
2.011570 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: work
2.011752 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2755
2.011833 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.012025 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler
2.012157 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2756
2.012353 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes
2.012631 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: classes
2.012796 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2756
2.012897 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.013100 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler
2.013239 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2757
2.013445 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com
2.013894 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: com
2.014095 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2757
2.014174 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.014355 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com
2.014504 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2758
2.014962 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun
2.015169 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: sun
2.015339 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2758
2.015428 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.015633 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun
2.015764 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2759
2.015980 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx
2.016221 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: javafx
2.016402 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2759
2.016493 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.016693 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx
2.016827 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x275a
2.017096 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api
2.017348 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: api
2.017520 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x275a
2.017590 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.017803 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api
2.017919 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x275b
2.018133 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree
2.018389 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: tree
2.018547 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x275b
2.018626 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.018779 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2754
2.018851 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.019157 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Basic Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.019292 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO
2.019495 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Standard Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.019613 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO
2.019832 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Internal Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.019960 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO
2.020206 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Network Open Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.020316 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO


For those of you who don't enjoy reading packet dumps (I hope that's all of you), what's going on here is that it does some sort of complicated multipacket transaction for each level of directory from the project root down to the last directory in the chain, and then does four request-responses for each file. And it repeats the directory stuff for every file, even though it just asked that.

OK, more debugging -- what could cause a system to deviate from the standard file system client behavior? Check all the network control panel settings, they're all the same. Spend several hours googling through the MS knowledge base for file sharing related problems, look at the various registry keys and file versions mentioned, nope, none of them are helpful. Google for people who have had similar problems. Many have, but no one reported a solution that works, except one person, who mentioned that their network behavior changed when they changed versions of Symantec antivirus. Well, I don't run Symantec AV, but I do run ZoneAlarm. And I do have different versions -- ZA Antivirus on the old system, ZA Suite on the new. Seems like a small difference -- they're clearly built on the same base technology -- but lets try it. Disabled ZA, reboot, make sure its not running, and run my IDE again -- no change. Still an annoying 30-60s delay before it figures out there's nothing to rebuild.

At this point, I asked my friends for help. Lots of sympathy. Lots of "check this, check that", but very little advice that actually moved me towards a solution (sorry, guys).

That post about the guy with the Symantec problem gnawed at me, though. I know security programs intercept a lot of network traffic, so the theory was perfectly plausible, and the best theory I had so far. I did the "disable ZA" thing again, rebooted, and cranked up Rootkit Hook Analyzer to see if ZA still had anything hooked, and it did, even though there were no ZA processes running and the ZA TrueVector service was stopped. So, I ran the uninstaller for ZA Suite, rebooted, checked with RHA to see that everything was unhooked (it was), and ran my IDE test again -- and this time, sweet success!

So, the conclusion is that the ZA Suite interferes with file sharing client behavior in a rather fundamental way (but one which only has a noticeable affect when dealing with lots of small files).

So, my system is temporarily defenseless against malware while I decide what to do. Why on earth would ZA rewrite the file system client packet stream like that? I want to send them a bill for that day and a half.

Tuesday, June 19, 2007

Remove checked exceptions?

Recently, Neal Gafter mused about whether we should consider removing checked exceptions from Java.  The motivation from this was not what you might expect, but rather an observation that checked exceptions interacts heavily with a lot of other language features, and that evolving the language might be easier if we were willing to consider removing some features.  (Neal knows this won't ever happen, he's just trying to get us thinking about Life After Java.)  Not surprisingly, it generated a storm of comments, ranging from "hell yeah!" to "hell no!".

This isn't a new topic; it comes around every few years.  A few years back I wrote about the debate surrounding checked exceptions, and the debate continues to rage.  My problem is that I think most of the vocal opponents of checked exceptions are objecting for the wrong reasons (back then, I wrote: "My opinion is that, while properly using exceptions certainly has its challenges and that bad examples of exception usage abound, most of the people who agree [ that checked exceptions are a bad idea ] are doing so for the wrong reason, in the same way that a politician who ran on a platform of universal subsidized access to chocolate would get a lot of votes from 10-year-olds").

Reading through the against-checked-exceptions commenters on Neal's blog, we can divide them into three primary groups:

  1. "I don't like checked exceptions because they're too much work." 

  2. "Checked exceptions were a nice idea in theory, but using them correctly makes your code really ugly, and I'm left with a choice of ugly code or wrong code, and that seems a bad choice." 

  3. "Checked exceptions are a good idea, but the world isn't ready for them."  (Frequent refrain from this group: "Man, have you looked at some of the code out there?")


To the people in camp (1), I say: engineering is hard -- get over it.  Error handling is one of the hardest things to get right, and one of the easiest things to be lazy about.  If you're writing code that's supposed to work more than "most of the time", you're supposed to be spending time thinking about error handling.  And, pay for your own damn chocolate. 

To the people in camp (2), I have more sympathy.  Exceptions do make your code ugly, and proper exception handling can make your code really ugly.  This is a shame, because exceptions were intended to reduce the amount of error-handling code that developers have to write.  (Ever try to properly close a JDBC Connection, Statement, and ResultSet?  It requires three finally blocks.  Ugly if you do it right.  But, almost no one ever does it right.  (The real culprit here is that close() throws an exception -- what are you supposed to do with that exception?  But that's fish under the bridge.)) 

But perhaps there's a way to not throw the baby out with the bathwater, by providing better exception handling mechanisms that are less ugly.  Dependency injection frameworks did a lot of that for us already, for a large class of applications -- and the code got a lot prettier, easier to write, and easier to read.  AFAICS, the two biggest removable uglinesses of exceptions are repeated identical catch clauses and exception chaining. 

The repeated catch clause problem is when you call a method that might throw exceptions A, B, C, and D, which do not have a common parent other than Exception, but you handle them all the same way.  (Reflection is a major offender here.) 

public void addInstance(String className) {
    try {
        Class clazz = Class.forName(className);
        objectSet.add(clazz.newInstance());
    }
   catch (IllegalAccessException e) {
        logger.log("Exception in addInstance", e);
    }
    catch (InstantiationException e) {
        logger.log("Exception in addInstance", e);
    }
    catch (ClassNotFoundException e) {
        logger.log("Exception in addInstance", e);
    }
}

You'd like to fold the catch clauses together, because duplicated code is bad.  Some people simply catch Exception, but this has a different meaning -- because RuntimeException extends Exception, you're also sweeping up unchecked exceptions accidentally.  You can explicitly catch and rethrow RuntimeException before catching Exception -- but its easy to forget to do that.

public void addInstance(String className) {
    try {
        Class clazz = Class.forName(className);
        objectSet.add(clazz.newInstance());
    }
    catch (RuntimeException e) {
        throw e;
    }
    catch (Exception e) {
        logger.log("Exception in newInstance", e);
    }
}

My proposal for this problem is to allow disjunctive type bounds on catch clauses:

public void addInstance(String className) {
    try {
        Class clazz = Class.forName(className);
        objectSet.add(clazz.newInstance());
    }
   catch (IllegalAccessException | InstantiationException | ClassNotFoundException  e) {
        logger.log("Exception in addInstance", e);
    }
}

My compiler friends tell me that this isn't too hard. 

The other big ugliness with exceptions is wrapping and rethrowing:

public void findFoo(String className) throws NoSuchFooException {
    try {
        lookupFooInDatabase(name);
    }
   catch (SQLException e) {
        throw new NoSuchFooException("Cannot find foo " + name, e);
    }
}

Now, the wrap-and-rethrow technique is very effective -- it allows methods to throw exceptions that are at an abstraction level commensurate with what the method is supposed to do, not how it is implemented, and it allows you to reimplement without destabilizing method signatures.  But it adds a lot of bulk to the code.  Since this is such a common pattern, couldn't it be solved with some sort of declarative "rethrows" clause:

public void findFoo(String className) throws NoSuchFooException
rethrows SQLException as NoSuchFooException {
    lookupFooInDatabase(name);
}

The rethrows clause is part of the implementation, not the signature, so maybe it goes somewhere else, but the idea is clear: if someone tries to throw an X out of this, wrap it with a Y and rethrow it. 

An alternate approach to this would be possible with closures and reified generics; it would be possible to write a pseudo-control construct that said "execute this closure but if it throws X, wrap it with a Y and rethrow it."  Unfortunately, with the current state of generics, we can't write such a generic method, we'd have to write a separate one for each exception type we want to wrap. 

These approaches focus on the symptom -- because the arguments in group (2) are about symptoms.  If we could alleviate the symptoms, people might grumble less.

The people in camp (3) are saying something slightly different.  I don't really have an answer for them, because what they seem to be saying is that no matter what mechanism you give people for dealing with failure, they won't follow it.  Checked exceptions were a reaction, in part, to the fact that it was too easy to ignore an error return code in C, so the language made it harder to ignore.  This works on a lot of programmers who are slightly lazy but know that ignoring exceptions is unacceptable, but apparently is worse than nothing for some parts of the population.  (We'd like to take away their coding rights, but we can't.) 

 Checked exceptions are a pain, and in some frameworks (like EJB before dependency injection), can be really painful.  Once the ratio of "real code" to "error handling code" rises above some threshold, readability suffers greatly, and readability is a fundamental value in the Java language design.  Even if the IDE generates the boilerplate for you, you still have to look at it, and there's a lot of noise. 

On the other hand, my experiences using third party C++ libraries was even more painful than anything Java exceptions have ever subjected me to.  Virtually no packages ever documented what might be thrown, so you end up playing "whack a mole" when exceptions did pop up -- and usually at your customers's site.  If people are not forced to document what errors their code throws, they won't -- especially the people that the people in camp (3) are afraid of.  As long as those folks are allowed to code, the value we get from checked exceptions forcing developers to document failure modes overwhelms the annoyances.

But, as I said above, I think many of the annoyances can be removed by adding a small number of exception streamlining constructs.  This doesn't help Neal with simplifying closures, but it does help us get our job done with a little less pain. 

Finally, a meta-note -- its really easy to misinterpret the volume of support for "removing checked exceptions" as any sort of gauge for community consensus.  We're intimately familiar with the pain that checked exceptions cause; we're substantially less familiar with the pain that they free us from.  (Obviously, neither approach is perfect, otherwise there'd be no debate.)

Thursday, May 17, 2007

JCiP best seller at JavaOne 2007

For the second year in a row, Java Concurrency in Practice was the best selling book at the JavaOne bookstore...thanks everyone!

Sunday, January 28, 2007

Interview on Software Engineering Radio

Tune in to Software Engineering Radio to hear the interview Marcus Voelter did with David Holmes and I at OOPSLA 2006.

Saturday, December 30, 2006

Reader mail: Urban performance legends

In response to Urban Performance Legends, Revisited, an anonymous reader wrote that I was "too enthusiastic" about the JVM's ability to inline virtual method calls:
Moreover, the example provided is misleading: inlining in java can be generally done for private or final methods. Non-final public and protected methods can't be automatically inlined because they can be overidden by a subclass. The actual type of the object may be unknown at compile time and at load time. The only way to inline them is to analyse the code to find every use of the class and perform type inference. This can be done sometimes with inner classes that have limited scope, but not with public classes that must be compiled indipendentely from each other.

Unfortunately, this is a common myth about optimization in Java (and other dynamically compiled languages): that if a method could be overridden by a class that has yet to be loaded, then the compiler cannot optimize away the virtual function call.  This is just plain wrong (as is the rest of what this reader says), and today's JVMs can devirtualize and inline through virtual calls using a number of techniques. 

One such technique is called monomorphic call transformation, where the compiler observes that for a given method foo(), there is no class loaded right now that overrides it, so calls to foo() can be compiled (speculatively) as direct calls instead of virtual calls.  If a class is loaded later that makes foo() polymorphic, the compiler can invalidate the speculatively optimized code.  This is covered in detail in Dynamic Compilation and Performance Measurement.  There are other techniques as well, such as inline virtual caching. 

The myth that final has any effect on method invocation performance for monomorphic methods was well-exploded by Cliff Click's 2003 and 2005 JavaOne presentations. 

Java (as well as other managed languages, like C#) is not C.  Dynamic compilers are smarter than you think.

Reader mail: making tasks noncancelable, and polling for interruption

I frequently get e-mail feedback on my articles on IBM developerWorks.  Unfortunately, I can rarely reply to them because no one ever leaves their e-mail address.  The majority of e-mails I get are erroneous corrections (not that I don't make mistakes -- I make plenty, and those get pointed up too -- its just there are even more people out there who are really really sure they're always right, but aren't), which I'd be happy to respond to if anyone ever left their e-mail... Here's one from Dealing with InterruptedException.  Listing 6 shows an example of how to make an operation noncancelable by deferring interruptions until the operation completes. 

public Task getNextTask(BlockingQueue queue) {
    boolean interrupted = false;
    try {
        while (true) {
            try {
                return queue.take();
            } catch (InterruptedException e) {
                interrupted = true;
                // fall through and retry
            }
        }
    } finally {
        if (interrupted)
            Thread.currentThread().interrupt();
    }
}

The anonymous reader asks: shouldn't this be "while (!interrupted)"?  Won't the loop never terminate?  The idea behind this example was to address the problem of "what if we have an operation that should not be interruptible, but is composed using interruptible steps?"  We will have to ignore the interruptions when they occur and retry the interrupted operation -- but the key is we want to remember if there has been an interruption, so we can restore the interrupted status after we complete our noncancelable operation.  That way, we don't throw away the information that someone requested cancellation, we just defer the cancellation request until after the noncancelable operation operation completes. 

A related issue that is often asked is how often we should check for interruption.  Listing 3 shows a typical interruptible operation wrapped in a Runnable.  When this happens, in order to not throw away the evidence that an interruption was requested, we have to re-set the interrupted status with Thread.currentThread().interrupt(). 

public class TaskRunner implements Runnable {
    private BlockingQueue queue;
    public TaskRunner(BlockingQueue queue) {
        this.queue = queue;
    }
    public void run() {
        try {
             while (true) {
                 Task task = queue.take(10, TimeUnit.SECONDS);
                 task.execute();
             }
         }
         catch (InterruptedException e) {
             // Restore the interrupted status
             Thread.currentThread().interrupt();
         }
    }
} 

It is often asked: shouldn't the loop header be while (!Thread.currentThread().isInterrupted()), instead of while(true)? This one is a little more subtle.  We can look at this from several perspectives: correctness, responsiveness and performance. 

From a correctness perspective, the existing code is correct, because (reasonably coded) interruptible blocking methods check if the interrupted status is set on method entry, and immediately throw InterruptedException() if it is.  Similarly, the existing approach is effectively as responsive as the suggested alternative, because the first step in the while loop is to call an interruptible blocking method.  (If substantial computation occurred between loop entry and the first interruptible blocking method call, responsiveness might give us a reason to test the interrupted status in the loop header as well -- and maybe other places throughout the loop too.) 

What about performance?  As with most performance questions of this sort, the answer is "We don't have enough information to tell".  Testing the interrupted status in the loop header costs a little more (because the test is repeated in the loop header and in the blocking take() call), but in the case that the interrupted status is set, saves us the cost of instantiating and catching the exception thrown from take() when it finds the interrupted status set on entry.  Which is a performance win will depend on how often the operation is actually interrupted -- and we usually don't have this information when coding general-purpose library code. 

When in doubt, do the thing that makes the code simplest, cleanest, and most readable.

Sunday, September 10, 2006

Farewell Quiotix, hello Sun!

I've been self-employed for fifteen years; I founded Quiotix in 1992 to pursue some short-term consulting opportunities after getting laid off from my last job, and I never looked back. I like the flexibility, and I don't mind the stress of not knowing what my income is going to be in any given month. I like not having to ask permission to take a day off, or buy a new monitor, or attend a conference that looks interesting. So in the past, when offered full-time positions, my response has usually been "Why would I want that?" (This can be very perplexing to someone offering you a job; they're used to getting a very different response.)

So, those of you who know me well may be a little surprised to learn that as of September, I will be joining Sun Microsystems as a Sr. Staff Engineer / Technical Evangelist in the Java SE engineering organization. I'll be involved in a lot of things, including a lot of the same things I've been doing as an independent -- like writing technical papers on JVM internals and speaking at conferences. I'll also be involved with the Java SE engineering, QA, and education efforts. I have a feeling I'm going to be pretty busy.

To the customers, colleagues, business partners, and employees I've worked with at Quiotix, I offer thanks for fifteen good years and the best of luck in the future.

Thursday, June 15, 2006

Memories of iceberg metaphors past

In his Dancing About Architecture blog, Eric Lunt calls JCiP a "two copy" book -- one for home, one for work. I like the way he thinks.

He describes concurrency as an "iceberg" subject:
Concurrency in Java is a very difficult, iceberg subject, and I guarantee your intution will fail you more often than not.

And he's more right than he realizes! At one point we were going to put an iceberg on the cover, and the introduction was even titled "Under the waterline" (though the metaphor does not appear in the final version of the book at all.) But then we realized that leading with something so depressing might not be the best strategy.

From an earlier draft of the introduction:
Icebergs typically have only ten to twenty percent of their volume above the waterline ??? the rest is hidden from view, lurking underwater. Concurrency is sort of like an iceberg ??? the explicit elements (synchronization, notification) are above the waterline and are clearly labeled, but in order to avoid an unpleasant collision, you must also be aware of what is below the waterline ??? and there???s a lot down there. Below the waterline lies all of the implicit policies and agreements that govern how these mechanisms are intended be used and mean in specific situations within your program. While it may sound trivial, it is not enough to simply use synchronization to ensure thread safety ??? you have to use synchronization correctly and consistently. Describing how to craft and document sensible under-the-waterline policies for the effective use of Java???s built-in concurrency constructs is the central goal of this book. Fortunately, it is not prohibitively difficult to develop a strategy that encompasses the submerged elements ??? as long as you take them into account from the beginning.


All concurrency constructs (such as synchronization, notification, or interruption) involve more than one thread. One of the reasons that writing and maintaining concurrent code in Java is so difficult is that all of the major concurrency constructs involve some degree of implicit agreement between multiple parties as to how the concurrency construct is to be used or interpreted . The concurrency constructs provided by the language are low-level mechanisms, but they are used in applications to enforce application-level concurrency policies or protocols. The tricky part is that policies are defined by the developer and can vary across programs or classes; the same mechanism can be used to enforce a variety of policies. These policies are not captured anywhere in the code (if you???re lucky, it???s captured in documentation, but more often it is not), and therefore it is difficult to automatically (or manually) verify or validate that a program is using the concurrency mechanisms correctly.



Each of the mechanisms for concurrency ??? such as synchronization, notification, and interruption ??? involves cooperation between two or more threads, and each involves an implicit and explicit component. With synchronization, the act of taking and releasing the lock is explicit, but the reason for taking and releasing the lock ??? because the lock guards a particular set of shared state variables ??? is implicit. Similarly, the act of waiting or notifying on a condition variable is explicit, but why you are waiting ??? that you are waiting for a specific condition predicate to become true ??? is implicit.



Thursday, June 8, 2006

JCiP sells through its first printing

Java Concurrency in Practice sold through its first printing (5000 copies) in less than a month. If you're waiting for your copy, be patient -- more are being printed!

Rumor has it it that Chinese and Japanese translations are underway.

Saturday, May 20, 2006

JCiP best-selling book at JavaOne!

Java Concurrency in Practice was the best selling book at the JavaOne bookstore!

Tuesday, May 16, 2006

JavaOne, day 1

JavaOne seemed better attended this year than last; business conditions are picking up. Once again, they rewrote the Schedule Builder application, and once again, it sucked. How hard is it to write a data-driven web application? Shouldn't that be a slam-dunk for Sun?

In an attempt to manage class sizes better, signup for each talk was mandatory. This means they had to scan your RFID badge for every talk, which created a huge bottleneck as there were typically only four scanning stations and they wanted to move a thousand people into the room in a five to ten minute window. No one seemed happy about this.

Tuesday, April 18, 2006

An even better quote

From the preface of Pragmatic Ajax (Gehtland, Almaer, and Galbraith), on book writing:

"Writing a book is a lot like (we imagine) flying a spaceship too close to a black hole. One second you're thinking "Hey, there's something interesting over there" and a picosecond later, everything you know and love has been sucked inside and crushed."

Indeed.

Monday, April 17, 2006

The beast is slain

I turned in final PDF today for Java Concurrency in Practice. Books should be ready by JavaOne. Woohoo!

The final tally: 425 pages, and sixteen months.

Not quite sure what to do with myself now.