This surprised the heck out of me. We recently finished a new TV room down in the basement. We have a 50" plasma TV, mounted on the wall with an Omnimount UCL mount (quite an impressive bit of engineering -- not cheap, but highly recommended.) Since the mount weighs upwards of 40 pounds, and supports a TV that weights 100 pounds on a torque arm as long as 2ft, it needs to be anchored pretty solidly to the wall. It is held up with 6 4" long, 3/8" wide lag screws, that screw into two separate studs. It does its job well.
So, my dad and I went to remove it from the wall. I removed the six screws, and we prepared to catch the mount. It didn't fall off the wall. We tugged on it, and it still didn't come off the wall. Seemed stuck so tight we thought we'd missed a screw! But we convinced ourselves there were no more screws, and the two of us pulled hard, and eventually it came off the wall -- taking some of the wallboard with it. Apparently the pressure of being screwed up against the wallboard (and maybe the heat from the TV too over a few years) turned the painted surface into a glue not only strong enough to hold a 40lb mount to a vertical wall but resist being pulled off!
Saturday, June 7, 2008
Monday, June 2, 2008
Questions from the Peanut Gallery, part I
At the "Writing the Next Great Java Book" BOF at JavaOne, there were unfortunately many more questions than we had time to answer. Fortunately, Greg Doench saved the questions (presciently, they were submitted on paper), so I'll answer a few here.
Q: About how much time did you put into your book (effort, not duration)?
A: In my case, the answers for effort and duration are the same, as I had the luxury of writing the book mostly-full-time -- I made the book my foreground activity, though I still did some consulting and training while I was working on it. I spent approximately 16 months on the book -- longer than planned (but in hindsight is no surprise.)
Q: Was the financial compensation worthwhile?
A: Unless your name is Stephen King or JK Rowling, writing books is not something you do for the financial compensation. This is more true for technical books, because (a) the audience for books like Java Concurrency in Practice is not quite as large as the audience for Harry Potter, and (b) if you have the skills to write a good technical book you probably have the skills to get a well-paying technical job. Without going into the details, I'll say that the compensation is about what I expected -- but I went into it with very realistic expectations. The compensation comes in other forms.
Q: How much support and assistance was provided by the publisher?
A: I think this is a matter of how much support and assistance you ask of the publisher -- and how much the publisher thinks you need. In our case, we did everything ourselves, including typesetting and managing the review, copy editing, and index creation. These are things the publisher often does for authors (and might even have preferred to do), but we chose to do it ourselves, and the publisher agreed. Of course, this was more work, but it was work we gladly did. The A-W team was always responsive when we did ask for things. So I think the answer is "as much as you appear to need."
Q: How does the short half-life of technical topics affect the effort?
A: I deliberately chose a topic with a longer half-life. This gave me the latitude to let the book tell me when it was done, rather than the schedule. For material with a shorter shelf life, I might be inclined to choose a shorter format, so that the book is less out-of-date by the time it is published.
Q: What would you say about books that authors release chapters to the public as they write?
A: I think this presupposes a style of writing where the author sits down and writes the book linearly. I am sure some authors do this, and some topics are more amenable to this approach than others. But one of the most important freedoms in writing is the freedom to refactor continuously; very often you don't figure out the right way to present the material until you've presented it the wrong way (just as with code.) There's nothing wrong with putting the work out there early -- this is a great source of free review -- but you have to be careful that doing so doesn't cause you to settle into the belief that the structure of the book has been decided. (The same risk is true of trying to adhere to a schedule that assigns due dates to specific chapters.)
Q: How do you avoid example source code exploding without using unrealistic examples?
A: This is really hard! But its really important. In JCiP, we set a rule for ourselves of "no code example more than a page", with the target of making most of them a half page or less. This is not easy, especially in Java! (There was only one we had to break into two separate one-page listings.) We wanted the examples to each illustrate a single point, so that the reader could look at the example and easily see what it was trying to show. There are some obvious tricks; eliminating boilerplate code like constructors, getters, and setters helps a little bit. What worked for us was to pick realistic examples that the audience would immediately understand the utility of (such as a file crawler), but abstract away the irrelevant concrete details by not showing the bodies of methods that are not needed to make the point that the example is supposed to illustrate. For example, we have a set of examples in Chapter 8 where we illustrate searching for solutions to a class of puzzles such as the "sliding block puzzles." But rather than focus on a specific puzzle -- which would take lots of space and not offer all that much insight, we abstract the nature of the puzzle by defining an interface that specifies the initial position (in terms of an abstract Position class), valid moves (in terms of an abstract Move class), and the goal position. Then we can illustrates various search techniques in terms of the abstract puzzle without getting bogged down in the details.
Q: What would you say is the role of technical books in the age where the Internet is the fastest way to publish texts and technology changes so fast that one year after publishing texts become irrelevant?
A: Some technical books are simply a form of documentation; any book that has a version number in the title is likely to fall into this category. These books have a very short shelf life. Other books, those that tend to focus on concepts rather that specific technical details, tend to have a longer shelf life. In any case, the publishing industry needs to become more agile in its approach to managing the authoring and production process, and explore more seriously alternate publication vectors such as electronic publishing.
More later.
Q: About how much time did you put into your book (effort, not duration)?
A: In my case, the answers for effort and duration are the same, as I had the luxury of writing the book mostly-full-time -- I made the book my foreground activity, though I still did some consulting and training while I was working on it. I spent approximately 16 months on the book -- longer than planned (but in hindsight is no surprise.)
Q: Was the financial compensation worthwhile?
A: Unless your name is Stephen King or JK Rowling, writing books is not something you do for the financial compensation. This is more true for technical books, because (a) the audience for books like Java Concurrency in Practice is not quite as large as the audience for Harry Potter, and (b) if you have the skills to write a good technical book you probably have the skills to get a well-paying technical job. Without going into the details, I'll say that the compensation is about what I expected -- but I went into it with very realistic expectations. The compensation comes in other forms.
Q: How much support and assistance was provided by the publisher?
A: I think this is a matter of how much support and assistance you ask of the publisher -- and how much the publisher thinks you need. In our case, we did everything ourselves, including typesetting and managing the review, copy editing, and index creation. These are things the publisher often does for authors (and might even have preferred to do), but we chose to do it ourselves, and the publisher agreed. Of course, this was more work, but it was work we gladly did. The A-W team was always responsive when we did ask for things. So I think the answer is "as much as you appear to need."
Q: How does the short half-life of technical topics affect the effort?
A: I deliberately chose a topic with a longer half-life. This gave me the latitude to let the book tell me when it was done, rather than the schedule. For material with a shorter shelf life, I might be inclined to choose a shorter format, so that the book is less out-of-date by the time it is published.
Q: What would you say about books that authors release chapters to the public as they write?
A: I think this presupposes a style of writing where the author sits down and writes the book linearly. I am sure some authors do this, and some topics are more amenable to this approach than others. But one of the most important freedoms in writing is the freedom to refactor continuously; very often you don't figure out the right way to present the material until you've presented it the wrong way (just as with code.) There's nothing wrong with putting the work out there early -- this is a great source of free review -- but you have to be careful that doing so doesn't cause you to settle into the belief that the structure of the book has been decided. (The same risk is true of trying to adhere to a schedule that assigns due dates to specific chapters.)
Q: How do you avoid example source code exploding without using unrealistic examples?
A: This is really hard! But its really important. In JCiP, we set a rule for ourselves of "no code example more than a page", with the target of making most of them a half page or less. This is not easy, especially in Java! (There was only one we had to break into two separate one-page listings.) We wanted the examples to each illustrate a single point, so that the reader could look at the example and easily see what it was trying to show. There are some obvious tricks; eliminating boilerplate code like constructors, getters, and setters helps a little bit. What worked for us was to pick realistic examples that the audience would immediately understand the utility of (such as a file crawler), but abstract away the irrelevant concrete details by not showing the bodies of methods that are not needed to make the point that the example is supposed to illustrate. For example, we have a set of examples in Chapter 8 where we illustrate searching for solutions to a class of puzzles such as the "sliding block puzzles." But rather than focus on a specific puzzle -- which would take lots of space and not offer all that much insight, we abstract the nature of the puzzle by defining an interface that specifies the initial position (in terms of an abstract Position class), valid moves (in terms of an abstract Move class), and the goal position. Then we can illustrates various search techniques in terms of the abstract puzzle without getting bogged down in the details.
Q: What would you say is the role of technical books in the age where the Internet is the fastest way to publish texts and technology changes so fast that one year after publishing texts become irrelevant?
A: Some technical books are simply a form of documentation; any book that has a version number in the title is likely to fall into this category. These books have a very short shelf life. Other books, those that tend to focus on concepts rather that specific technical details, tend to have a longer shelf life. In any case, the publishing industry needs to become more agile in its approach to managing the authoring and production process, and explore more seriously alternate publication vectors such as electronic publishing.
More later.
Saturday, May 31, 2008
The making of JCiP; avoiding errors in code listings
At JavaOne this month, my publisher at Pearson, Greg Doench, hosted a panel/BOF entitled "Writing the Next Great Java Book." Not surprisingly, when you get four authors in a room, you get seven opinions about what's important in writing a book!
One thing that we (mostly Tim, actually) did for JCiP was set up an infrastructure for the book, similar to what you'd do for a software project. Version control, issue tracking, one-step build script, continuous build -- all of these things offer the same benefits to book projects as they do for software.
One critical aspect of the build is the handling of program listings. It is incredibly tempting to cut and paste examples from the IDE into whatever source format you're writing in (Word, Frame, LaTeX, DocBook), but this is a recipe for disaster -- errors will invariably creep in as you try and make small tweaks (such as changing variable names) outside the IDE. And code examples with errors really undermine the reader's confidence (or worse, they copy the incorrect example into their code.) So, we wanted to make sure that every code example compiled (and ideally, was tested.)
Our approach was to check the code into Subversion with the rest of the book artifacts, ensure that the build process compiled the code and ran the unit tests, and then automatically extract the examples from the code in a format into which they could be directly included by the build. Some systems (LaTeX, DocBook) make this sort of inclusion easier than others.
We marked the examples up with comments for formatting (bold, italic) and also with "snip here" comments that excluded the irrelevant portions of the code from the listings that actually went into the book. The attached perl script (phragmite.pl), written by Tim Peierls (based on an approach designed by Ken Arnold), takes as input a set of input files and produces a set of LaTeX files representing the extracted listings.
As an example, here is the Counter listing from Listing 4.1 of JCiP:
The first line identifies the type of the code fragment (!! for a "good example", ?? for a "bad example" which would get decorated with a Mr. Yuk), the name of the fragment (Counter), and the listing caption. The lines with the ^^ and vv mean "snip from here to here", and a listing can be made of multiple such fragments. The /*[*/ and /*]*/ comments mean "bold". The following ANT target ran the script:
In the book's LaTeX source, we use the following LaTeX macro to pull the listing in:
One thing that we (mostly Tim, actually) did for JCiP was set up an infrastructure for the book, similar to what you'd do for a software project. Version control, issue tracking, one-step build script, continuous build -- all of these things offer the same benefits to book projects as they do for software.
One critical aspect of the build is the handling of program listings. It is incredibly tempting to cut and paste examples from the IDE into whatever source format you're writing in (Word, Frame, LaTeX, DocBook), but this is a recipe for disaster -- errors will invariably creep in as you try and make small tweaks (such as changing variable names) outside the IDE. And code examples with errors really undermine the reader's confidence (or worse, they copy the incorrect example into their code.) So, we wanted to make sure that every code example compiled (and ideally, was tested.)
Our approach was to check the code into Subversion with the rest of the book artifacts, ensure that the build process compiled the code and ran the unit tests, and then automatically extract the examples from the code in a format into which they could be directly included by the build. Some systems (LaTeX, DocBook) make this sort of inclusion easier than others.
We marked the examples up with comments for formatting (bold, italic) and also with "snip here" comments that excluded the irrelevant portions of the code from the listings that actually went into the book. The attached perl script (phragmite.pl), written by Tim Peierls (based on an approach designed by Ken Arnold), takes as input a set of input files and produces a set of LaTeX files representing the extracted listings.
As an example, here is the Counter listing from Listing 4.1 of JCiP:
// !! Counter Simple thread-safe counter using the Java monitor pattern
// vv Counter
@ThreadSafe
public final class Counter {
/*[*/@GuardedBy("this")/*]*/ private long value = 0;
public /*[*/synchronized/*]*/ long getValue() {
return value;
}
public /*[*/synchronized/*]*/ long increment() {
if (value == Long.MAX_VALUE)
throw new IllegalStateException("counter overflow");
return ++value;
}
}
// ^^ Counter
The first line identifies the type of the code fragment (!! for a "good example", ?? for a "bad example" which would get decorated with a Mr. Yuk), the name of the fragment (Counter), and the listing caption. The lines with the ^^ and vv mean "snip from here to here", and a listing can be made of multiple such fragments. The /*[*/ and /*]*/ comments mean "bold". The following ANT target ran the script:
<target name="listings">
<exec dir="${bin.dir}" executable="perl">
<arg value="${phragmite.pl}"/>
<arg value="${listings.dir}"/>
<arg value="${fragments.dir}/*.java"/>
<arg value="${fragments.dir}/jcip/*.java"/>
</exec>
</target>
In the book's LaTeX source, we use the following LaTeX macro to pull the listing in:
\newcommand{\JavaListing}[1]{\input{listings/#1}%
Saturday, May 17, 2008
Apologies for the malware warnings
Apparently, WordPress is vulnerable to some script injection bugs, and this site was hit by them. And Google tagged the site as "spreading malware", so the site shows up with a warning in Google search results and FF3 users can't get to it at all. I've upgraded Wordpress, scoured the DB for injected scripts, and am in the process of begging google to let me off the blacklist.
What a pain in the butt. People suck.
What a pain in the butt. People suck.
Wednesday, April 30, 2008
Tailored, indeed
I noticed this amusing typo at the trade show floor at SDWest this year. My first thought: wow, they really are tailored!
Hacker at work
Monday, December 10, 2007
A day and a half in the life...
This is a story about Windows, Samba, debugging, and frustration. I just spent a day and a half debugging a very annoying performance problem with a brand-new system.
Background: I just recently bought a new desktop computer, which I do about every two years. Computers should last longer than that, but I find that, even with good "hygeine", Windows systems tend to decay to the point where they exhibit weird behaviors after about two years, for which the cure is a complete reinstall of the OS and all applications. (Pause for Mac fanboys to snicker.) The "rebuild the world" process wouldn't be so bad if it weren't so hard to migrate all one's data -- even given the fact that a lot of my data is already in subversion. The real problem is that each application sprays its configuration and data randomly around your system, whether in the program install folder, registry, documents and settings folder, local settings folder, etc.
So, I bought a new computer, an extra-quiet one from www.endpcnoise.com. These guys specialize in quiet systems, and since I work in a home environment, the computer is usually the noisiest thing in the room. (Of particular annoyance is the variable speed fan on my existng Dell system, which, every time the system worked up a sweat, made a whiny noise. And we was fined $50 and had to pick up the garbage in the snow, but thats not what I came to tell you about.) Pretty happy with the new system overall.
So, I installed XP Pro on the new system, and proceeded to install all my applications, utilities, and all kinds of groovy things that we were talking about on the bench. And then I got to the part where I tried to use it; specifically, tried to fire up IDEA and build the project I'm working on.
Now, I've got a somewhat weird setup; when developing on a project, I checkout a workspace on my Linux server, which is served via SAMBA to my local network, and I run the IDE on my Windows desktop and point the IDE at my samba share. There's a measurable performance hit vs local, but I like being able to do some things from the Linux command line and other things from the IDE, so overall its a more productive setup for me.
When you ask the IDE to "make" the project, it crawls the files in the project checking their modification times. An "empty" make on a project with ~1000 files generally takes a few seconds to figure out that there's nothing to do. But when I set it up, the new (faster) system took about 30-60s to do an empty make.
OK, it's debugging time. What's different about the two systems? Same OS (XP Pro), same service pack level, both systems up to date on patches, same Java version, same IDE version, same user credentials, no host-specific information in my SAMBA config. Different hardware. Make sure my ethernet drivers are up-to-date. Test network for errors, swap cables, all that. Run IOMeter, found that both get similar throughput for large files on the same SAMBA share.
Crank up perfmon, which tells me that the new system is sending out more packets for a make than the old one. OK, crank up ethereal, get a packet capture, and find that the new system is sending/receiving 10x as many packets for the same operation:
[brian@brian-server ~]$ wc -l /tmp/*cap
258204 /tmp/new-cap
17719 /tmp/old-cap
So, what's the difference? Let's look at the packet capture. In the old trace, for each file being probed, it did something like this:
0.467895 192.168.1.104 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Basic Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
0.468041 192.168.1.107 -> 192.168.1.104 SMB Trans2 Response, QUERY_PATH_INFO
0.468283 192.168.1.104 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Network Open Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
0.468402 192.168.1.107 -> 192.168.1.104 SMB Trans2 Response, QUERY_PATH_INFO
Two requests, two responses per file. Seemed reasonable. On the new system, for each file:
2.010471 192.168.1.113 -> 192.168.1.107
SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree
2.010698 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2754
2.010900 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \
2.011011 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2755
2.011237 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work
2.011570 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: work
2.011752 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2755
2.011833 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.012025 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler
2.012157 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2756
2.012353 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes
2.012631 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: classes
2.012796 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2756
2.012897 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.013100 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler
2.013239 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2757
2.013445 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com
2.013894 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: com
2.014095 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2757
2.014174 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.014355 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com
2.014504 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2758
2.014962 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun
2.015169 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: sun
2.015339 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2758
2.015428 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.015633 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun
2.015764 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2759
2.015980 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx
2.016221 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: javafx
2.016402 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2759
2.016493 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.016693 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx
2.016827 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x275a
2.017096 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api
2.017348 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: api
2.017520 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x275a
2.017590 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.017803 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api
2.017919 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x275b
2.018133 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree
2.018389 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: tree
2.018547 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x275b
2.018626 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.018779 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2754
2.018851 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.019157 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Basic Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.019292 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO
2.019495 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Standard Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.019613 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO
2.019832 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Internal Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.019960 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO
2.020206 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Network Open Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.020316 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO
For those of you who don't enjoy reading packet dumps (I hope that's all of you), what's going on here is that it does some sort of complicated multipacket transaction for each level of directory from the project root down to the last directory in the chain, and then does four request-responses for each file. And it repeats the directory stuff for every file, even though it just asked that.
OK, more debugging -- what could cause a system to deviate from the standard file system client behavior? Check all the network control panel settings, they're all the same. Spend several hours googling through the MS knowledge base for file sharing related problems, look at the various registry keys and file versions mentioned, nope, none of them are helpful. Google for people who have had similar problems. Many have, but no one reported a solution that works, except one person, who mentioned that their network behavior changed when they changed versions of Symantec antivirus. Well, I don't run Symantec AV, but I do run ZoneAlarm. And I do have different versions -- ZA Antivirus on the old system, ZA Suite on the new. Seems like a small difference -- they're clearly built on the same base technology -- but lets try it. Disabled ZA, reboot, make sure its not running, and run my IDE again -- no change. Still an annoying 30-60s delay before it figures out there's nothing to rebuild.
At this point, I asked my friends for help. Lots of sympathy. Lots of "check this, check that", but very little advice that actually moved me towards a solution (sorry, guys).
That post about the guy with the Symantec problem gnawed at me, though. I know security programs intercept a lot of network traffic, so the theory was perfectly plausible, and the best theory I had so far. I did the "disable ZA" thing again, rebooted, and cranked up Rootkit Hook Analyzer to see if ZA still had anything hooked, and it did, even though there were no ZA processes running and the ZA TrueVector service was stopped. So, I ran the uninstaller for ZA Suite, rebooted, checked with RHA to see that everything was unhooked (it was), and ran my IDE test again -- and this time, sweet success!
So, the conclusion is that the ZA Suite interferes with file sharing client behavior in a rather fundamental way (but one which only has a noticeable affect when dealing with lots of small files).
So, my system is temporarily defenseless against malware while I decide what to do. Why on earth would ZA rewrite the file system client packet stream like that? I want to send them a bill for that day and a half.
Background: I just recently bought a new desktop computer, which I do about every two years. Computers should last longer than that, but I find that, even with good "hygeine", Windows systems tend to decay to the point where they exhibit weird behaviors after about two years, for which the cure is a complete reinstall of the OS and all applications. (Pause for Mac fanboys to snicker.) The "rebuild the world" process wouldn't be so bad if it weren't so hard to migrate all one's data -- even given the fact that a lot of my data is already in subversion. The real problem is that each application sprays its configuration and data randomly around your system, whether in the program install folder, registry, documents and settings folder, local settings folder, etc.
So, I bought a new computer, an extra-quiet one from www.endpcnoise.com. These guys specialize in quiet systems, and since I work in a home environment, the computer is usually the noisiest thing in the room. (Of particular annoyance is the variable speed fan on my existng Dell system, which, every time the system worked up a sweat, made a whiny noise. And we was fined $50 and had to pick up the garbage in the snow, but thats not what I came to tell you about.) Pretty happy with the new system overall.
So, I installed XP Pro on the new system, and proceeded to install all my applications, utilities, and all kinds of groovy things that we were talking about on the bench. And then I got to the part where I tried to use it; specifically, tried to fire up IDEA and build the project I'm working on.
Now, I've got a somewhat weird setup; when developing on a project, I checkout a workspace on my Linux server, which is served via SAMBA to my local network, and I run the IDE on my Windows desktop and point the IDE at my samba share. There's a measurable performance hit vs local, but I like being able to do some things from the Linux command line and other things from the IDE, so overall its a more productive setup for me.
When you ask the IDE to "make" the project, it crawls the files in the project checking their modification times. An "empty" make on a project with ~1000 files generally takes a few seconds to figure out that there's nothing to do. But when I set it up, the new (faster) system took about 30-60s to do an empty make.
OK, it's debugging time. What's different about the two systems? Same OS (XP Pro), same service pack level, both systems up to date on patches, same Java version, same IDE version, same user credentials, no host-specific information in my SAMBA config. Different hardware. Make sure my ethernet drivers are up-to-date. Test network for errors, swap cables, all that. Run IOMeter, found that both get similar throughput for large files on the same SAMBA share.
Crank up perfmon, which tells me that the new system is sending out more packets for a make than the old one. OK, crank up ethereal, get a packet capture, and find that the new system is sending/receiving 10x as many packets for the same operation:
[brian@brian-server ~]$ wc -l /tmp/*cap
258204 /tmp/new-cap
17719 /tmp/old-cap
So, what's the difference? Let's look at the packet capture. In the old trace, for each file being probed, it did something like this:
0.467895 192.168.1.104 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Basic Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
0.468041 192.168.1.107 -> 192.168.1.104 SMB Trans2 Response, QUERY_PATH_INFO
0.468283 192.168.1.104 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Network Open Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
0.468402 192.168.1.107 -> 192.168.1.104 SMB Trans2 Response, QUERY_PATH_INFO
Two requests, two responses per file. Seemed reasonable. On the new system, for each file:
2.010471 192.168.1.113 -> 192.168.1.107
SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree
2.010698 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2754
2.010900 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \
2.011011 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2755
2.011237 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work
2.011570 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: work
2.011752 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2755
2.011833 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.012025 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler
2.012157 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2756
2.012353 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes
2.012631 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: classes
2.012796 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2756
2.012897 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.013100 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler
2.013239 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2757
2.013445 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com
2.013894 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: com
2.014095 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2757
2.014174 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.014355 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com
2.014504 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2758
2.014962 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun
2.015169 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: sun
2.015339 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2758
2.015428 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.015633 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun
2.015764 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x2759
2.015980 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx
2.016221 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: javafx
2.016402 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2759
2.016493 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.016693 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx
2.016827 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x275a
2.017096 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api
2.017348 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: api
2.017520 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x275a
2.017590 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.017803 192.168.1.113 -> 192.168.1.107 SMB NT Create AndX Request, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api
2.017919 192.168.1.107 -> 192.168.1.113 SMB NT Create AndX Response, FID: 0x275b
2.018133 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, FIND_FIRST2, Pattern: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree
2.018389 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, FIND_FIRST2, Files: tree
2.018547 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x275b
2.018626 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.018779 192.168.1.113 -> 192.168.1.107 SMB Close Request, FID: 0x2754
2.018851 192.168.1.107 -> 192.168.1.113 SMB Close Response
2.019157 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Basic Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.019292 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO
2.019495 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Standard Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.019613 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO
2.019832 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Internal Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.019960 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO
2.020206 192.168.1.113 -> 192.168.1.107 SMB Trans2 Request, QUERY_PATH_INFO, Query File Network Open Info, Path: \work\openjfx-compiler\classes\production\openjfx-compiler\com\sun\javafx\api\tree\OnDeleteElementTree.class
2.020316 192.168.1.107 -> 192.168.1.113 SMB Trans2 Response, QUERY_PATH_INFO
For those of you who don't enjoy reading packet dumps (I hope that's all of you), what's going on here is that it does some sort of complicated multipacket transaction for each level of directory from the project root down to the last directory in the chain, and then does four request-responses for each file. And it repeats the directory stuff for every file, even though it just asked that.
OK, more debugging -- what could cause a system to deviate from the standard file system client behavior? Check all the network control panel settings, they're all the same. Spend several hours googling through the MS knowledge base for file sharing related problems, look at the various registry keys and file versions mentioned, nope, none of them are helpful. Google for people who have had similar problems. Many have, but no one reported a solution that works, except one person, who mentioned that their network behavior changed when they changed versions of Symantec antivirus. Well, I don't run Symantec AV, but I do run ZoneAlarm. And I do have different versions -- ZA Antivirus on the old system, ZA Suite on the new. Seems like a small difference -- they're clearly built on the same base technology -- but lets try it. Disabled ZA, reboot, make sure its not running, and run my IDE again -- no change. Still an annoying 30-60s delay before it figures out there's nothing to rebuild.
At this point, I asked my friends for help. Lots of sympathy. Lots of "check this, check that", but very little advice that actually moved me towards a solution (sorry, guys).
That post about the guy with the Symantec problem gnawed at me, though. I know security programs intercept a lot of network traffic, so the theory was perfectly plausible, and the best theory I had so far. I did the "disable ZA" thing again, rebooted, and cranked up Rootkit Hook Analyzer to see if ZA still had anything hooked, and it did, even though there were no ZA processes running and the ZA TrueVector service was stopped. So, I ran the uninstaller for ZA Suite, rebooted, checked with RHA to see that everything was unhooked (it was), and ran my IDE test again -- and this time, sweet success!
So, the conclusion is that the ZA Suite interferes with file sharing client behavior in a rather fundamental way (but one which only has a noticeable affect when dealing with lots of small files).
So, my system is temporarily defenseless against malware while I decide what to do. Why on earth would ZA rewrite the file system client packet stream like that? I want to send them a bill for that day and a half.
Subscribe to:
Posts (Atom)
