The return of the associative database?

In May 2013, the US Navy put out a requirements document for the development of an open source, US-based, “scalable, secure associative database”. You probably missed the announcement. Three companies were short listed to fulfil the Navy’s needs: Applied Technical Systems Inc., Boston Fusion Corporation and Systems & Technology Research. The first of these was awarded the contract.

There are a couple of interesting things here. The first is the fact that the Navy wants an associative database in the first place. Why? The proposal document is littered with references to graph databases, triple stores and the like and it’s not as if there aren’t any open source graph database projects, though they might not be as secure as the Navy requires. In any case, why not go for graphs? Well, actually, associative databases have one crucial advantage over graph databases, which is that while the latter is about entities and relationships between those entities, an associative database not only encompasses associations (aka relationships) between entities but also associations between associations. For example, in a graph database expressing a nuclear family you might have father related to mother, father related to son and mother related to son: that’s three entities and three relationships. In an associative database, on the other hand, you have a ‘child of’ association that associates with the mother-father association. In other words you still have three entities but only two associations.

This may not seem like much of a difference but suppose you have 100 guests at your wedding then each guest has exactly one association with the wedding as opposed to one relationship with each of the parties to be married. Thus there are twice as many relationships as associations. And this is before we consider relationships between guests. In other words, associations are much more efficient as relationships proliferate. This can get important when you want to scale to petabytes (and, presumably, still get good performance), as the Navy does.

Associative databases have been around longer than graph databases, at least commercially, but they have been more widely deployed as underlying technology than as platforms. For example, Kalido (now Magnitude Software) uses associative techniques, while X88’s Pandora is based upon an (in-house developed) associative database. The only database offering per se, at least that I am aware of, is Sentences from Lazy Software (so called because it was designed to make software development easy: “lazy”). This enjoyed some success in the early years of this century only to go dormant for a significant period before being revived over the last couple of years.

So, I like the concept of associative databases. However, it is too early to say whether we will see a major revival of interest, not least because Applied Technical Systems, according to its web site, specialises in developing software for the US Navy. This may mean—I hope it does not—that the eventual product does not get marketed more widely. If that is the case then Lazy Software is going to be left to fly the flag largely on its own and, while I wish it every success, that is not going to be easy in the face of the rising tide of graph databases.