Today, (and yesterday, and a good portion of the night in-between) I ran into a nasty bug in a library that I didn’t know my code depended on. It isn’t particularly important what I was working on, but just for context: I needed to strip a lot of text content out of nodes in the complete wikipedia revision history dump, so I was using Sax to parse the xml stream, filter out the stuff I wanted filtered out, and save the stuff that, well, I wanted saved. Being that the input was all of wikipedia, there were a fair number of unicode characters in there. As it turns out, the 2.6.2 xercesImpl has some sort of bug that allows xml with certain characters to be read without throwing exceptions, but when you try to write the chars that were actually read, you end up trying to write characters that aren’t valid in xml. Even if I’d known that in advance, my response would have been something like “ok, so what? I’m not using xercesImpl, and certainly not a version that old”.

Well.

You see, in addition to using Maven, I’ve also been using the Google Collections and JSR305 libraries, so I just drop those <dependency> entries into the pom for all my new projects–I just assume that I’ll need them, and I usually do.

Unfortunately, JSR305 1.3.8 depends on jaxen 1.1.1, which depends on xercesImpl 2.6.2 (jaxen also needs this dependency via xom 1.0, for what that’s worth).

Because that dependency was already present in my build path (via mvn eclipse:eclipse) and in the generated jar (via <addClasspath> and <classpathPrefix> in the maven-jar-plugin configuration section), I never realized that my sax code actually had a direct dependency on xerces as well. This all came to a head when, 3.53gb into my 2.8tb run, these rather unhelpful exceptions started popping up:

<rant> “?” is not unicode — it fits just fine in asci tables everywhere — so please don’t tell me that it’s an invalid unicode character :) (0xd800 is an invalid unicode character, and that would have been much more helpful) </rant>

Many hours later I was able to find a sample of the actual input that was causing these problems, and I was able to reproduce the issue with an input slightly smaller than 2.8tb. Once that was done, I set out to make a minimal test case. Rather than bother with a new maven project, I just hacked it out in emacs (not using google collections, etc. because, clearly, I wanted it minimal). To my surprise, everything worked, and worked fantastically! But how? I didn’t even supply an xml api on the classpath, yet it ran just fine!

In truth, I did supply an xml api — xercesImpl.jar, and many other libraries — via my environment’s $CLASSPATH. (Figuring that out was another adventure, but I digress.) Once it became clear that I was indeed using a broken library it was simply a matter of explicitly specifying the dependency on a new version of xercesImpl, and rebuilding.

The moral?

Know your dependencies! This should come along with knowing your language’s built-in APIs well. It wasn’t clear to me that the SAX packages I was using were not part of the core java API, so it didn’t strike me as odd that I didn’t need to specify a classpath entry or a pom dependency before I could use sax.

If you suspect something strange, you can see the dependency tree in the generated html documentation you get when running mvn site.

I’ve been digging into OSGi a bit over the last week or so inorder to create some Eclipse plugins that will automatically discover eachother, and I’ve been generally impressed with the framework on the whole. The documentation is a bit lacking, but there are some good blog posts about it. (Specifically [Neil Bartlett's introduction to OSGi][intro].)

One thing that bugged me is the repetition needed when you implement the CommandProvider interface to add commands to the OSGi console. CommandProvider defines one method you must supply:

OSGi then uses reflection to extract each of the methods that starts with an ““, and supplies those methods to the command environment as new commands. (The “” is trimmed, and the name of the method becomes the command name.) General practice is to include the name of the method in the return value of getHelp(), along with a description of what the method does, eg:

This seems like a pain to maintain, so I took a quick look at annotations, and propose a new syntax:

Here we’ve extracted the getHelp() method into a new superclass, so our SampleCommandProvider now extends an abstract class instead of implementing an interface. It also makes use of an Annotation, which we need to define:

Finally, we just need to define the superclass that implements getHelp():

Note that the actual reflection on the class only happens once — all subsequent calls to getHelp() use a cached copy of the documentation.

[intro]: http://neilbartlett.name/blog/osgi-articles/

In the end, I spent a weekend and a couple evenings building a new Wizard API for Java, called CJWizard. The library is released under the Apache V.2 license, so it should work for just about anything you want to use it for. I would like to know if you’re using it, and what you’re using it for, just to sate my own curiosity :). The project is hosted on code.google.com, so please submit issues, and feel free to contribute to the project.

CJWizard provides the structure needed to quickly create simple dialogs by implementing an abstract class (WizardPage) for each page of the dialog, and adding them to a PageFactory that can generate pages on-demand, as they are required. This puts the programmer in full control of how the wizard proceeds. The CJWizard architecture also makes it easy to add a wizard to an existing application (either via an additional JDialog, or embedding in some other component), and/or insert custom wrapper widgets around the dialog pages–meaning that you can quickly add customized navigational controls aside from the standard Previous/Next/Finish/Cancel buttons.

Some aspects were taken from the Java.Net wizard API, such as auto-detecting named components, and automatically collecting the values from them, but CJWizard takes a much simpler approach (and in some ways, a less powerful one — CJWizard does not listen to every key event, only collecting values when the user navigates away from a WizardPage). In most cases, you only need to name widgets prior to adding them to the WizardPage, and their values will be collected in a settings map automatically.

CJWizard was meant to provide a flexible way to generate professional-looking multi-step dialogs very quickly.

This is a straight-forward recursive implementation of fib. When run with n=4, we see this:

9 invocations of fib(n), but only 5 unique invocations. Lets memoize the results, and try this again:

Much better — 5 invocations, 5 unique sets of parameters.

Here’s the source with memoization:

Notice that we only needed to add 4 new lines of code in order to memoize the results. When fib(n) is called, it simply checks to see if it has previously been called with n, and if so, that result is used again. If the parameter has never been seen before, the method continues as normal, storing the computed result before returning. Memoization turns this naive (and exponential) implementation of fib(n) into an efficient (linear) operation.

Memoization in the real world

So, (un?)fortunately we don’t spend all day implementing cool new ways of computing ever increasing entries of the fibinocci sequence — how can memoization be put to use? After all, many algorithms are already implemented in some fairly optimal fashion by the language APIs, and you’d be a fool not to use those implementations. What opportunity will you have to memoize functions?

It turns out that you can memoize anything, as long as the function is pure with respect to the memos (meaning: the function doesn’t depend on any thing that is not used to key the hash of memos). If the function is not pure, then you can still use memoization, but either the memo hash must key on all the state and parameters that can affect the results of the function. On the other hand, if f depends on some state that changes very rarely, then it may make more sense to simply discard all the stored memos each time that aspect of state is altered.

Memoization is extremely handy when you have very common operations that are fairly expensive. I recently needed to optimize some code that compares strings based on the case-insensitive stems of the words, with stopwords removed. So the strings “he wanted an apple” and “he wants apples” should be equal. (”an” is a stopword, and ignored)

This meant doing many, many calls to a string stemmer, each of which is a fairly expensive operation. Fortunately, hashing strings as extremely cheap (on the order of 1/4th the time it took to stem a string of the same length), and I had plenty of memory to store the parameters and the results in a Map. Adding memos to the two primary time-hoggers (the stemmer and a tokenizer) cut the execution time of the application down from 2 hours to just over 7 minutes.

Summary

You can memoize any function that only depends on it’s parameters and constant state (or near-constant state — just don’t forget to discard your memos when the state changes!). If the function is invoked multiple times you will probably see a performance improvement.

If you need to memoize a function with multiple arguments, then you just need to nest Maps, or create a unique key by combining the parameters in some way.

Memoization is an extremely easy way to improve performance under certain circumstances, particularly if you have a solid grasp on when state changes outside of your methods / functions, or program in a functional style. It can be memory intensive, however. If the results of your functions are large, or maintain references to large objects, then memoization may penalize performance if you run out of memory and have to make use of swap space.