Library Versioning 14 Dec 2011

Libraries should be versioned and packaged such that they are easy to use over time, and in combination. The best way I have found to do this is to abide by three rules:

Use APR versioning
Re-namespace on major version changes
Change the artifact ID on major version changes

Use APR style versioning

APR versioning basically defines the meanings of changes for versions like {major}.{minor}.{bugfix}.

A bugfix release is both forwards and backwards compatible. It is a drop in, binary compatible, replacement for anything with the same {major}.{minor} numbers. Going from 2.28.0 to 2.28.1 would be a bugfix release.

A minor release is backwards compatible but not forwards compatible. That is, a 2.29.7 version can be dropped in to replace any other 2.29.X, or earlier minor version numbers such as 2.27.4 or 2.1.0. It would not be a drop in replacement for 2.30.0, though. Typically minor releases add new functionality through additions to the API.

A major release is not backwards compatible with anything lower – a 3.0.0 cannot be dropped in to replace a 2.30.7 – it has a different API. Nor can it replace a 4.2.89 release, which has a higher major version number.

Version numbers are used to encode compatibility for the API.

Re-namespace on major version changes

When making a major version change, that is a backwards incompatible change, always use a new namespace. In Java or C# use a new package name, in Ruby use a new module name, in C use a new function prefix, etc.

Re-namespacing allows you to use both the old and new versions in the same process. This is particularly important for transitive dependencies. To look at a concrete counter-example demonstrating the pain of not doing this let’s look at a personal mistake I made in jDBI. jDBI uses a StatementContext to make information available to to extensions, such as custom statement rewriters. Between 2.15 and 2.16 I changed StatementContext from an abstract class to an interface, but did not change it’s API. I believed this was a backwards compatible change because I thought the same bytecode was generated for method invocations against interfaces and abstract classes.

I was wrong, different bytecode is generated. Heavy users of jDBI tend to create small libraries which bundle up their extensions, and then they rely on their small libraries. At Ning we call ours ning-jdbi. If rely on ning-jdbi 1.3.2 which relies on jdbi 2.14, and I rely on jdbi 2.28 then I am in trouble, as ning-jdbi will get runtime errors when trying to run against the more recent version of jDBI. I have to go cut a new version of ning-jdbi, which is now backwards incompatible, and the chain continues. By introducing an accidental backwards incompatible binary change I forced backwards incompatible changes on the whole dependency chain.

Oops.

Change the artifact ID on major version changes

Using a seperate namespace for backwards incompatible changes is not enough on its own in most circumstances. Yes, both versions can coincide in the same process, but most build and packaging tools cannot handle loading two libraries with the same name and different versions. As I don’t want to write yet-another-dpkg or yet-another-build-tool merely to work around this issue, you save tons of grief by also changing the library identifier as fas as build and packaging goes.

Once again, transitive dependencies are the main driver here. If I depend on com.google.guava:guava:10.0.1 and a library I use depends on com.google.guava:guava:r9 (using maven coordinates) I am only going to get 10.0.1. Unfortunately, Guava is not backwards compatible between r9 and 10.0.1, and the library I depend upon may or may not work. (I pick on Guava here because it is such a damned useful library, so it tends to get used everywhere, at least by me).

I used a Java example here, but it applies everywhere. For example, in the Ruby world most gems are pretty lackadaisical about compatibility, not to mention packaging. Heck, just look at the history of rubygems itself.

Working around bad libraries

Even if you strictly follow these rules, when you rely on a library which does not follow them, you break. Your only options are to stop using it or to repackage it. In general you can repackage by just grabbing the source and changing the namespace, then building against and including your renamespaced version. This locks you into a specific version, but it protects you and your users from an unhygienic dependency. Depending on how you build you may renamespace the source at build time, such as by using the maven shade plugin, or during development by just pulling the source into your source tree.

Maybe in Java 17 Nov 2011

One of the more elegant concepts in a lot of functional languages is that of Maybe. Haskell drove home the magic of Maybe for me. The general idea is representing the possibility of a value. In Haskell, you’ll generally use it such that processing will continue if the value is needed and present, or short circuit if the value is needed and not present. The end result of a computation which includes a Maybe will also be a Maybe, but one representing the result of the computation.

In expression-oriented languages like Haskell, that works out very nicely, but I spend most of my time working in Java, which is decidedly statement-oriented. Concepts will sometimes move nicely between worlds, sometimes not. When I started working on Atlas recently I decided to see how well Maybe ported. I started with Nat Pryce’s maybe-java and took off from there.

Nat’s class encourages a model analogous to Haskell’s, executing within the context of a Maybe instance by passing functions into the Maybe instance, for example:

Maybe<String> name = Maybe.definitely("Brian");
Maybe<String> message = name.to(new Function<String, String>() {
    public String apply(String s)
    {
        return "hello, " + s;
    }
});
System.out.println(message.otherwise("No one here!"));

This treats the sequence of Maybe instances as an evaluation context or control flow, which works nicely in some languages, but sadly, as with most attempts to do higher-order functions in Java, it got awkward rather quickly. Part of it is purely syntactic, the syntax isn’t optimized for it, but part of it is semantic as well. Idiommatic Java uses exceptions for non-happy-path control flow, and most of the libraries which provide the main reason for using Java behave this way.

Given that, I switched from using Maybe to control evaluation to using Maybe purely to represent the possibility of a value, and things fell into place very nicely – even playing within Java’s exception idioms. Take for example this snippet:

SSHCredentials creds = space.lookup(credentialName, SSHCredentials.class)
    .otherwise(space.lookup(SSHCredentials.DEFAULT, SSHCredentials.class)
    .otherwise(new IllegalStateException("unable to locate any ssh credentials"));

In this case there may exist named credentials, if not there may exist some default credentials, and if there is neither the world explodes. In the typical case you would see either a test for existence and then use, or a fetch and check for null. Both of which are, to my mind, less clear and certainly more error prone (largely in needing to remember to check everywhere, particularly in the case of a this-or-that situation, etc).

Other bits of using Maybe extensively are not completely clear, but I am pretty confident that I will be using some evolution this flavor of Maybe in most of my Java-based code going forward.

Using s3 URLs with Ruby's open-uri 11 Aug 2011

Ruby’s open-uri is a wonderful hack, and I recently got to figure out how ot plug in additional URL schemes. Here is a quick and dirty to allow urls of the form s3://<bucket>/<object> :

require 'aws/s3'

module URI

  class S3 < Generic
    def initialize(*args)
      @bucket, @file = args[2], args[5][1,args[5].length]
      super(*args)
    end

    def open &block
      http_url = AWS::S3::S3Object.url_for @file, @bucket
      URI.parse(http_url).open &block
    end
  end
  @@schemes['S3'] = S3
end

It uses the AWS::S3 library, but could be adapted pretty easily to the AWS SDK for Ruby. It does require the normal initialization but then just works :-)

open("s3://skife/whiteboard.jpg") do |in|
  # do stuff with the contents...
end

Fundamental Components in a Distributed System 26 Jul 2011

In the last several weeks I have had a surprising number of conversations about the fundamental building blocks of a large web-based system. I thought I’d write up the main bits of a good way to do it. This is far from the only way, but most reasonably large systems will wind up with most of this stuff. We’ll start at the base and work our way up.

Operational Platform

At the very base of the system you need to have networking gear, servers, the means to put operating systems onto the servers, bring them up to baseline configuration, and monitor their operational status (disk, memory, cpu, etc). There are lots of good tools here. Getting the initial bits onto disk will usually be determined by the operating system you are using, but after that Chef or Puppet should become your friend. You’ll use these to know what is out there and bring servers up to a baseline. I personally believe that chef or puppet should be used to handle things like accounts, dns, and stable things common to a small number of classes of server (app server, database server, etc).

The operational platform provides the raw material on which the system will run, and the tools here are chosen to manage that raw material. This is different than application management.

Deployment

The first part of application management is a means of getting application components onto servers and controlling them. I generally prefer deploying complete, singular packages which bundle up all their volatile dependencies. Tools like Galaxy and Cast do this nicely. Think hard about how development, debugging, and general work with these things will go, as being pleasent to work with during dev, test, and downtime will trump idealism in production.

Configuration

Your configuration system is going to be intimately tied to your deployment system, so think about these things together. Aside from seperating the types of configuration you want there are a lot of tradeoffs. In generally, I like immutable configuration obtained at startup or deployment time. A new set of configs means a restart. In this case, you can either have the deployment system provide it to the application, or have the application itself fetch it. Some folks really like dynamic configuration, in that case Zookeeper is going to be your friend. Most things don’t reload config well without a restart though, and I like having a local copy of the config, so… YMMV.

Application Monitoring

Application level monitoring and operational level monitoring are very similar, and can frequently be combined in one tool, but are conceptually quite different. For one thing, operational monitoring is usually available out of the box from good old Nagios to newer tools like ‘noit. Generally you will want to track the same kinds of things, but how you get it, and what they mean will vary by the application. Monitoring is a huge topic, go google it :-)

Discovery

Assuming you have somewhere to deploy, and the ability to deploy, your services need to be able to find each other. I prefer dynamic, logical service discovery where services register their availability and connection information (host and port, base url, etc) and then everything finds each other via the discovery service. A lot of folks use Zookeeper for this nowadays, and most everyone I know who has used it loves it. One of the best architecty type guys I know would probably have its baby if he could, based on how effusive he is about. That said, you can do lots of different things.

I have heard discussion about using the reporting capabilities of a tool like Galaxy, or the CMDB capabilities of Chef to accomplish this, but I think these are ill suited. Firstly, they operate on either concrete deployment units, or on specific low level roles, rather than on logical services. Secondly, they are quite outside the lifecycle control of the service itself.

In an ideal world the location of your discovery system is the only well known address in the whole system. Some things don’t participate well in discovery out of the box – those being fully formed components such as databases, caches, and so on. How you integrate these will vary, but two good techniques I have seen are the use of companion processes which interact with the discovery service, and static entries in the discovery service. In the case of a companion process, the companion generally does a very basic health check (is the server running?) and provides a local view of whatever is needed from the service. In the case of static entries, the entry may be placed and removed by the startup script, or via some alternate channel (doing it by hand, etc).

Yet More of the Long Tail Treasure Trove 15 Jul 2011

Another addition of the long tail treasure trove in blog form.

Brian's Waste of Time About Archive Feed

Library Versioning 14 Dec 2011

Use APR style versioning

Re-namespace on major version changes

Change the artifact ID on major version changes

Working around bad libraries

Maybe in Java 17 Nov 2011

Using s3 URLs with Ruby's open-uri 11 Aug 2011

Fundamental Components in a Distributed System 26 Jul 2011

Operational Platform

Deployment

Configuration

Application Monitoring

Discovery

Yet More of the Long Tail Treasure Trove 15 Jul 2011

SQLite JDBC

Connector/MXJ

ConcurrentLinkedHashMap

Greplin’s Bloom Filter Library

SSHJ

JLine2

ReflectASM

snakeyaml

QueueFile

Mail

CMPH

Diff Match and Patch