Weapons of Choice

I do Java for a living. Apart from a two-year hiatus back in 2003-05 I’ve been a Java guy ever since I left university in 1997 (Man, sixteen years now! I didn’t realize.).

Java is… well, it’s Java. For all it warts, it’s actually a pretty decent environment to get things done in. Yeah, it’s verbose as hell. Yeah, most projects involve heaps of boilerplate XML. And yeah, Maven is still the best we’ve got. But still, when you know your way around it’s not bad. The 3rd party library support is outstanding, and most of it is Open Source. Performance is generally good. And the cross-platform compatibility is at this point something we just take for granted.

There have been other languages. I was heavily into Common Lisp for a while. I would prefer it over java were it not for the lack of standard libraries. Clojure seemed like a logical progression, but for some reason I couldn’t get into it. I love the premise: “Functional programming with full access to the Java ecosystem,” but for some reason I never stuck around.

I was on the Scala wagon, I know enough Ruby to be dangerous, and I’ve flirted with Go, Haskell, Python, and CoffeeScript. All excellent languages in their own right. None of them really stuck. I’ve read the books, the blogs, the twitters. I know these are the languages the cool kids use nowadays. Maybe it was me, maybe I wasn’t hipster enough.

But there is one language that has grown on me. That hasn’t gone away or been replaced. It’s a ugly little thing, full of warts and cruft that has accumulated over many years. It is a Lisp, but it’s awkward and eccentric even by Lisp standards. It lacks basic necessities like namespaces and, for the longest time, lexical scoping. The standard library is confused and naming conventions are all over the place. Many Common Lisp staples (like loop, map, and reduce) are missing. There is a Common Lisp compatibility package, but using it was controversial and for a long time actively discouraged.

I am, of course, talking about emacs lisp. I took stock of my github projects and realized that apart from the Java stuff, almost all of my recent activities are elisp. This was not something I had planned, it’s just that Emacs is such an awesome elisp IDE. I’m more productive in elisp than in any other environment. When I make code changes I’m not just connecting to a REPL, I’m actually changing the very environment I’m currently coding in. This is incredibly powerful.

So, Java and elisp. Not very cool, I know. But do not underestimate the power of knowing your tools and getting shit done.

Raspberry Pi

The main focus at the JFokus conference last week seemed to be the Raspberry Pi. Sure, Oracle was pushing Java 8, JEE 7 and whatnot, but everybody was talking about the Little PC That Could. There were raffles where you could win them, and according to rumors they were given away to the audience at the embedded systems keynote.

I happened to attend a technical session, mostly because there was nothing better on. Gerrit Grunwald showed off a Pi-based system he had set up to display the temperature in his home, and I was sold immediatly. As I left the room I flipped open my laptop, found a reseller and ordered myself a starter kit. A few hours later I got a notice saying it had shipped and by early afternoon the next day it had been delivered, a full two days before I got home myself.

These are my impressions after playing with it over the weekend:

  • It really is small. Really, really small. Of course, I seem to have a thing for small computers.
  • Video output is strictly HDMI. It hooked up to the TV and booted nicely, but when I tried connecting it to my old 17” monitor using a HDMI-DVI converter I couldn’t get any signal at all. It doesn’t really matter all that much, as getting a SSH server up and running was trivial, and I kind of prefer accessing it through the network anyway.
  • I had the notion that Go would be a good language to do some systems programming in, but getting the infrastructure in place was not trivial. Essentially you have to apply a small patch and rebuild the debian packages before installing, and remember to put export GOARM=5 in your .bashrc.
  • The CPU isn’t great. Building Go from scratch took me well over 30 minutes. I realize I may have been spoiled by my MacBook.

And now I have a trivial webserver running:

Disclaimer: I don't really know Go
package main
import (
"fmt"
"net/http"
"html"
"log"
)
var port = 8001
var s = &http.Server {
Addr: fmt.Sprintf(":%d", port),
}
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hullo, %q", html.EscapeString(r.URL.Path))
})
log.Printf("starting web server on port %d", port)
log.Fatal(s.ListenAndServe())
}

Asadmin Shell Completion

I’ve been juggling glassfish servers a lot lately, and found the asadmin CLI admin tool pretty nice. There was one thing lacking though: ZSH tab completion. So, being a hacker, I thought to myself: “How hard can it be?”

Pretty hard, as it turns out. The ZSH completion system is ridiculously complex and poorly documented. There is a pretty good ebook, but it is from 2003 and I’m sure a lot of things have changed since then. Other than that, then only tutorial on writing your own completion functions was an old Linux Mag article.

After a lot of trial and error, borrowing code from various oh-my-zsh plugins, and nearly giving up in frustration, I’ve now produced a plugin of my own. It’s pretty large (1147 lines), but has completions for all the 265 subcommands that I know of. I quickly realized that maintaining the argument completions for all these subcommands would be quite impractical, so I did what any hacker would do: I wrote quick-and-dirty program that goes through all the glassfish jar files on the classpath, looking for classes that implement the asadmin commands. It then looks at annotations and uses a bit of introspection to determine what arguments are available for each command, and what completions (if any) should be used to complete these. The end result is a ZSH script fragment which I then insert manually into my completion script. Overall, this has worked out pretty well. And it sure beats doing it by hand.

Anyway, if you work with asadmin and would like some hot completion action going, check out my oh-my-zsh branch on github.

Finally, an Actual Emacs-eclim Release

After hacking on and off on emacs-eclim for the better part of three years, it’s finally time to make an actual, honest to God, release! With a version number and everything!

Even better, I made two releases, both at the same time. Thing is, the eclim guys switched the format of the response messages to JSON with eclim 1.7.3. That was a rather good change, but it meant we had to do quite a bit of work on our end to get everything working properly. And that’s why it finally made sense to start talking about emacs-eclim releases. If you are stuck on an older version of eclim, you need emacs-eclim version 0.1; but for eclim 1.7.3 or later you must run emacs-eclim 0.2, or HEAD.

Let's Write an Emacs Mode for Inform 7!

So, my love affair with Inform continues. Whatever love I housed for the in-built editor however, has withered and died. The keybindings all felt wrong, I kept accidentally narrowing the scope of the code, and had to use the mouse way too much for it to be a pleasant writing experience. So I thought, Hey, why not write an emacs mode for Inform? I’ve been meaning to learn about major modes and syntax highlighting anyway, and this is as good an excuse as any. After a bit of research, it turned out that creating a new major mode was pretty straightforward.

First, since we want headings in the source code to stand out, we’ll define a new font face, based on font-lock-preprocessor-face. It could be inherited from any face, really, but let’s pick one that’s likely to have been included in most colour themes.

(defface inform7-heading-face
'((t (:inherit font-lock-preprocessor-face :weight bold :height 1.2)))
"Face for Inform 7 headings"
:group 'font-lock-highlighting-faces)

Then some keybindings for our mode. There’s not much for us to do here, but let’s make RET automatically jump to the correct indentation level; that’s always nice:

(defvar inform7-mode-map
(let ((map (make-sparse-keymap)))
(define-key map (kbd "RET") 'newline-and-indent)
(define-key map "\C-j" 'newline-and-indent)
map)
"Keymap for inform7 major mode")

The easiest way to add syntax highlighting for our mode is to use regexp-based matchers. Inform 7 hasn’t got that much of a formal syntax, but it would be nice to highlight keywords like if, then and otherwise and for section headings in the text. We should also find some way of supporting indexed text (i.e. anthing inside a string surrounded by []).

The last one is a bit tricky, as the same delimiters are also used for comments (see below), and it’s not actually possible to distinguishing between them using only regexps. The solution here, prepending a ‘.’ to the matcher, is a workaround that’s good enough for most cases though.

One thing to watch out for is that that the names of the faces must be symbols. For the built-in faces there are variables defined for each of these, but we must be careful to quote the name of the face we defined above.

(defconst inform7-font-lock-keywords
`(( ,(regexp-opt '("Include" "Use" "let" "say" "if" "otherwise") 'words) . font-lock-keyword-face)
("^\\(\\(?:Book\\|Chapter\\|Part\\|Section\\|Volume\\) - .*\\)" . 'inform7-heading-face)
(".\\(\\[.*?\\]\\)." 0 font-lock-variable-name-face t)
)
"Highlighting expressions for inform7-mode")

We now have everything we need to define our new mode. Current wisdom holds that it’s a good idea to extend an existing mode using define-derived-mode, and have someone else do all the heavy lifting. In our case, we want to extend sws-mode, which handles significant whitespace and provides functions for intelligent indentation of lines and blocks of code. It is part of jade-mode.

Defining our major mode is deceptively simple; we just pass in a mode name, the name of the mode we’re deriving from, a printable name that will be displayed in the status line, and a body of code that will be executed when the mode is initialized. define-derived-mode will take care of the rest and set up a keymap, syntax table, mode hooks and ensure that the parent mode and the hooks are called on init. In our case, we’ve already created the keymap manually, so define-derived-mode will use that one instead.

(define-derived-mode inform7-mode sws-mode "Inform7"
"Major mode for editing inform 7 story files."
(visual-line-mode)
(set (make-local-variable 'font-lock-defaults) '(inform7-font-lock-keywords nil t)))

Two things remain, though. First, the line indent function does always increases the indentation of new/empty lines by one step. For inform code, we only want that if the preceding line ends with a ‘:’, otherwise we’d like the new line to have the same indentation level as the previous one.

It turns out, that can be accomplished by adding some advice around sws-do-indent-line, which is the function in sws-mode that actually carries out the indentation:

(defadvice sws-do-indent-line (around inform7-indent-blank () activate)
"Ensure proper indentation of blank lines according to Inform7 conventions."
(if (and (eq major-mode 'inform7-mode) (sws-empty-line-p))
(if (save-excursion
(previous-line)
(looking-at "^.*:[ \t]*$"))
(indent-to (sws-max-indent))
(indent-to (sws-previous-indentation)))
ad-do-it))

The other thing we need to fix is highlighting of comments in the text. We don’t want to do this with regexps, as it’s tricky to get this working reliably for multi-line comments. Instead, we’ll take advantage of the built-in facilities for syntactic highlighting. Incidentally, you may have noticed that we didn’t have to do anything to get highlighting of strings in the code; the syntactic parser gives us that functionality for free.

These two lines modifies the syntax table, telling the parser to treat [ and ] as comment delimiters.

(modify-syntax-entry ?\[ "<]" inform7-mode-syntax-table)
(modify-syntax-entry ?\] ">[" inform7-mode-syntax-table)

So there we are, 41 lines of code all in all, not bad for a start.

JFokus 2012

Last week was JFokus, and a great time was had by all. This was the third time for me, and I’m really glad they moved the location to Stockholm Waterfront. Last time I went, we were crammed into a cinema complex with far too little floorspace, no wifi and very limited toilet facilities. This year though, the location was great and the surprisingly good coffee kept flowing all throughout the three days.

There seemed to be a good deal of interest in Scala at the event, but I’m not sure this actually reflects some kind of actual change in the Java ecosphere. Mostly, it were small, hip, consultancies doing the promotion, and I have a gnawing suspicion that they don’t have much Scala work lined up per se, but are instead using Scala to get the attention of potential recruits. Time will tell.

I’m a pretty introverted guy, and don’t really feel that comfortable around a lot of people. Most of the time during the big breaks I found myself wishing for a laptop so that I could get hacking instead of having to wander round aimlessly pretending to feel at home in the crowd. Fate, however, seated me beside Chris Richardson (of Cloud Foundry fame) during lunch the first day, and we had a pretty good discussion, even though my english was still getting up to speed. For some reason I also managed to corner Greg Luck during the tuesday bar and talk to him about Ehcache. Both were very nice guys, and I’d like to thank them both for helping me come out of my shell a bit.

The Ballad of Emacs-Eclim

I’ve been using emacs-eclim for java development fulltime for well over two years now, and I’ve been a co-maintainer for a couple of months, but I’ve never really written anything about it. I guess part of it is because I’m way too lazy to do a big write-up explaining all the moving parts and how it relates to eclim and eclipse. You know, a proper blog post.

Like most java programmers, I was a long-time eclipse victim. If you’re not from the java world yourself, it might be hard to grasp the love/hate relationship that all java programmers have with this IDE. Or hate/hate relationship, as it may be. Sure, some people turn to NetBeans or IntelliJ, but you can’t get away from the fact that eclipse has in essence steamrolled all over the java IDE space. Eclipse is everywhere.

Incidentally, ‘steamrolled’ is an appropriate word here, as it suggests huge mass and inertia bordering on inevitability, both of which are also properties of eclipse itself. It takes forever to start up, and if you actually manage to kick off a complex build operation, it’s impossible to interrupt it.

Emacs was my escape. But the story is kind of complicated. You see, there was this bunch of enterprising vim hackers that realized that even though their editor was the greatest thing since sliced bread, it wasn’t really that good at handling large java codebases, something which eclipse is atually pretty competent at. So they wrote a plugin for eclipse that turned it into a kind of server, and a vim extension that acted as a client, and then they could edit the code in vim, while leveraging the strengths of eclipse, such as command completion, compilation, and refactoring. And they called it eclim.

My setup is obscure

Then another enterprising fellow ported the client libraries to emacs, and the result was emacs-eclim. It was a bit rough at first, but it is improving steadily. In the past year, we’ve gotten auto-complete-mode support, inline error highlighting, improved handling of import statements and numerous bug fixes. Right now I’m working on making calls to the eclimd backend asynchronous, to make editing more responsive. It kind of works, but right now it’s ugly.

I actually have no idea how many users we have. There are 116 people watching the project on Github right now, and bug reports are being filed every now and then, so I’d like to think that at least someone finds it worthwhile.

Nine Months of Working Remote

After a routine software update yesterday, my work laptop suddenly refused to boot. That was a bit scary, and left me stressed out. You see, I’m working remote, and the rest of our IT department is located about 1000 km away. Which pretty much meant “You’re on your own, kid. If you can’t solve this yourself, then you won’t have a workstation for a few days.”

This is the first time, however, that I actually had to reflect on my working conditions as a remote worker. I have a home office set up, complete with a stand-up desk, a Fatboy bean bag, and multi-function printer. I do the lion part of my work on my laptop, hooked up to the compay VPN. Inter-team communication is done using Campfire and daily telephone meetings. In many ways, this is just like working at the office, except a) I can blast whatever I want on the stereo, and b) I don’t get dragged into meetings as often as I used to.

Working alone does mean that I’m not just a developer anymore; I’m also responsible for first-line support for my own equipment. If the wifi goes down, no-one will fix it except me. If the laptop bombs out, I’ll either have to solve it myself, or find some way of setting up a temporary workspace on one of the ancient machines I have lying about until the company can send me a replacement.

Apart from that, I have no complaints about working alone. When I tell people about my situation, they usually go “Aaah, but you’ve got to have such self dicipline to be working from home.” The assumption being, that as soon as you don’t have someone looking over your shoulder, you will immediately revert to spending your working time goofing off. I don’t get that. When I’m working, I’m working. Same as if I were at the office. And there’s nothing special about me. In fact, I’m a pretty lazy guy.

I got the laptop working again, after an agonizing hour of recovery mode reboots, random fiddling with the BIOS and finally reinstalling the X server and proprietary NVIDIA drivers.

Generating Java Code With Maven, QDox and StringTemplate

Introduction: why would you ever want to generate Java code?

In my current project at work we have quite a few value objects with giant constructor methods; these behemoths take 20 to 30 arguments each and it’s crucial that you get all of them in the right order, which is a pain in those cases when you’re only really interested in setting one or two of them. Just as an example, let’s just say that these classes looks something like this:

public class ValueObj {
private int value1;
private int value2;
private int value3;
...
private Object obj1;
private Object obj1;
...
public ValueObj(int value1, int value2, int value2, ... Object
obj1, Object obj2, ...) {
this.value1 = value1;
this.value2 = value2;
this.value2 = value3;
...
this.obj1 = obj1;
this.obj2 = obj2;
...
}
public int getValue1() {
return this.value1;
}
... (other getter methods follow)
}

About a year ago or so we started using the Builder pattern, creating a separate Builder class for each value object class. The builder has one setter method for each parameter, and one create() method that instantiates the new object.

public class ValueObjBuilder {
private int value1;
private int value2;
private int value3;
...
private Object obj1;
private Object obj1;
...
public ValueObjBuilder value1(int value1) {
this.value1 = value1;
return this;
}
public ValueObjBuilder value2(int value2) {
this.value2 = value2;
return this;
}
...
public ValueObj create() {
return new ValueObj(value1, value2, ...);
}
}

Since the setter methods return a reference to the builder itself it is possible to chain invocations:

ValueObj obj = new ValueObjBuilder().value1(23).value2(47).create();

However convenient these builder classes are to use, I very quickly tired of maintaining them. Every time you add or remove a property to a value object, you also need to make the corresponding change to the builder. Keeping two classes in sync doesn’t seem like too much work, but it is unnecessary, and it occurred to me that I could probably write something to generate the builders automatically.

Writing a Maven plugin

We use Maven for our builds. Ok, ok, I know. Maven is horrible, maven downloads the whole universe, maven is braindead. Yes, I know all these things. But here’s the ting: Maven is like democracy: It’s not perfect, but it’s the best we have.

One of the nice things about maven is that it has a well-defined standard build lifecycle. The build process is divided into steps, and during one of the steps, generate-sources, a maven plugin has the opportunity to generate source that will then be compiled during the compile step.

When invoked, we’d like the plugin to:

  1. Loop over our java classes, looking for constructors marked with our custom @builder javadoc comment.

  2. For each matching constructor, generate a new source file under target/generated-sources/builderbuilder/ containing the Builder class.

The plugin is invoked before the compilation step, meaning that all the generated code will be included in the final output.

As an example, this code (in src/main/java/example/ValueObj.java):

package example;
public class Cat {
private String name;
private int age;
/**
* @builder
*/
public Cat(String name, int age) {
this.name = name;
this.age = age;
}
}

Will generate the following (in target/generated-sources/builderbuilder/example/CatBuilder.java) when run through our plugin:

package example;
public class CatBuilder {
private String name;
private int age;
public CatBuilder name(String name) {
this.name = name;
return this;
}
public CatBuilder age(int age) {
this.age = age;
return this;
}
public Cat create() {
return new Cat(name, age);
}
}

For those of you who like to read along, the code for the whole plugin is on github. The finished version has some extra bells and whistles, like the ability to generate abstract builder classes and explicitly the name of the create() method.

Parsing Java code

Okay, so let’s say we want to pick apart the java class files and for each constructor found we want to generate a helper class as discussed above. Now, normally we’d use the introspection facilities already built into java, but now we have one problem: our code-generating code will run during the generate-sources step, before any compilation has actually taken places.

This is a problem. What we need is a java parser that can pull apart the relevant bits of the source files and give us enough information that so we can generate the new classes. Preferably the level of detail should be on par with with that of the java.util.reflect.* functions, and it should also be fast. I actually started writing something like that, but that’s another story. Instead, let’s check out QDox, an amazing little library used internally by maven.

Using it is pretty simple. First, create a new JavaDocBuilder object and tell it where the source code you want to parse lives:

this.docBuilder = new JavaDocBuilder();
for (String r : sources) {
docBuilder.addSourceTree(new File(r));
}

Then loop over all known classes and generate the builder:

for (JavaClass jc : docBuilder.getClasses()) {
generateBuilderFor(jc);
}

Generating the output

This is the generateBuilderFor() method:

public void generateBuilderFor(JavaClass jc) throws IOException {
for(JavaMethod m: jc.getMethods()) {
if (m.isConstructor()) {
DocletTag dc = m.getTagByName("builder");
if (dc != null) {
String builderName = dc.getNamedParameter("name");
if (builderName == null)
builderName = jc.getName() + "Builder";
String packageName = dc.getNamedParameter("package");
if (packageName == null)
packageName = jc.getPackageName();
StringTemplate st = templates.getInstanceOf("builder");
st.setAttribute("packageName", packageName);
st.setAttribute("builderName", builderName);
st.setAttribute("resultClass", jc.asType().toString());
List<Param> ps = new LinkedList<Param>();
for(JavaParameter p: m.getParameters()) {
ps.add(new Param(p.getType().toGenericString(), p.getName()));
}
st.setAttribute("parameters", ps);
File pd = new File(outputDirectory, packageName.replaceAll("\\.", "/"));
pd.mkdirs();
FileWriter out = new FileWriter(new File(pd, builderName + ".java"));
try {
out.append(st.toString());
} finally {
out.flush();
out.close();
}
}
}
}
}
public static class Param {
public final String type;
public final String name;
public Param(String type, String name) {
this.type = type;
this.name = name;
}
}

This is all pretty straightforward; we loop through the methods for the class, looking for a constructor that has the @builder javadoc tag.

Lines 6-12 checks if the tag has any values specified for name and package, reverting to default values if not.

Lines 14-23 instantiates a new StringTemplate object and supplies it with values for the packageName, builderName and resultClass attributes (the latter being the class that we want the builder to actually build). Lines 19-23 loops through the constructor arguments and creates value object holding the name & type of each.

Finally, a new file is created (lines 25-26) and written (lines 28-34).

Templating

There are an abundance of templating frameworks out there. I choose to go with StringTemplate for this project, since it makes it easy to use subtemplates in a functional way, and generally fits my way of thinking. I could just as easily have used something like FreeMarker or Velocity, though.

The templates are not that interesting, so I won’t go into it here. You can check them out on github if you are curious.

Conclusion

Automatic code generation in java projects is not only feasible, but also quite convenient with maven. We’ve been using BuilderBuilder internally in production for a few months now, and it greatly cuts down on code maintenance.

It would be nice, though, if there was a more general approach to code generation. Just for BuilderBuilder, I had to dive into Maven internals (which are surprisingly poorly documented), learn QDox (ditto) and write all the java code that solved this particular problem. Imagine instead if there was some kind of transformation language that operated on java sources, kind of like what XSLT does for XML? Now, that would be cool.

The Night Circus

I’ve been playing The Night Circus a lot lately. Well, ‘playing’ is a strong word; I’ve been mostly clicking on random cards and enjoying the athmosphere of the game. I’m not really making any informed choices that influence the story, and I have little control over the game flow. This isn’t really the type of game I usually bother with, but it does quite a few things right:

  • You don’t have to create an account to play; just sign in with you twitter or facebook account.
  • Every move is represented by a card that you play. You get to keep six cards in your and you’re issued a new card randomly(?) every four minutes. This, in conjunction with that there is very little state to keep track of, means that it’s very easy to return every now and then, play a few cards and then do something else. In short, a perfect game to play while you’re code’s compiling.
  • There’s a constant stream of small rewards. Almost every card played gains you a memento of some sort, and most of the mementoes can be used to unlock new paths in the game.
  • Also, the setting is very athmospheric in a kind of fantasy victorian, Jonathan Strange & Mr Norrell kind of way. The writing is well done, and you get the feeling that there’s a lot to be discovered. I don’t know if there’s a conclusion or if the game ever really ‘ends’. New story segments are added weekly however, so there’s hope for some kind of coherent story.

The game builds Erin Morgenstern’s upcoming novel with the same name, and as a promotion piece it works pretty well. At the very least, it got me engaged and actually considering getting the book at some point.