The ideas for relational databases were worked out in the 1970s and the first commercial implementations appeared around 1980. By the 1990s relational databases were the dominant way to store data. There were some non-relational databases in use, but these were not popular. Hierarchical databases seemed quaint, clinging to pre-relational approaches that had been deemed inferior by the march of progress. Object databases just seemed weird.

Now the tables have turned. Relational databases are still dominant, but all the cool kids are using NoSQL databases. A few years ago the implicit assumption was that nearly all data lived in a relational database, now you hear statements such as “Relational databases are still the best approach for some kinds of projects,” implying that such projects are a small minority. Some of the praise for NoSQL databases is hype, but some is deserved: there are numerous successful applications using NoSQL databases because they need to, not because they want to be cool.

So why the shift to NoSQL, and why now? I’m not an expert in this area, but I’ll repeat some of the explanations that I’ve heard that sound plausible.

1. Relational databases were designed in an era of small, expensive storage media. They were designed to conserve a resource that is now abundant. Non-relational databases may be less conservative with storage.
2. Relational databases were designed for usage scenarios that not all applications follow. In particular, they were designed to make writing easier than reading. But its not uncommon for a web application to do 10,000 reads for every write.
3. The scalability of relational database transactions is fundamentally limited by Brewer’s CAP theorem. It says that you can’t have consistency, availability and partition tolerance all in one distributed system. You have to pick two out of three.
4. Part of the justification for relational databases was that multiple applications can share the same data by going directly to the same tables. Now applications share data through APIs rather through tables. With n-tier architecture, applications don’t access their own data directly through tables, much less another application’s data.

The object oriented worldview of most application developers maps more easily to NoSQL databases than to relational databases. But in the past, this objection was brushed off.  A manager might say “I don’t care if it takes a lot of work for you to map your objects to tables. Data must be stored in tables.”

And why must data be stored in tables? One reason would be consistency, but #3 above says you’ll have to relax your ideas of consistency if you want availability and partition tolerance. Another reason would be in order to share data with other applications, but #4 explains why that isn’t necessary. Still another reason would be that the relational model has a theoretical basis. But so do NoSQL databases, or as Erik Meijer calls them, CoSQL databases.

I’m not an advocate of SQL or NoSQL databases. Each has its place. A few years ago developers assumed that nearly all data was best stored in a relational database. That was a mistake, just as it would be a mistake now to assume that all data should now move to a non-relational database.

More database posts

• ACID versus BASE for database transactions
• Million dollar cutoff
• Big data cube

Erik Meijer’s paper Your Mouse is a Database has an interesting illustration of “The Big Data Cube” using three axes to classify databases.

The volume axis is big vs. small, or perhaps better, open vs. closed. Relational databases can be large, and non-relational databases can be small. But the relational database model is closed in the sense that “it assumes a closed world that is under full control by the database.”

The velocity axis is (synchronous) pull vs. (asynchronous) push. The variety axis captures whether data is stored by foreign-key/primary-key relations or key-value pairs.

Here are the corners identified by the paper:

• Traditional RDBMS (small, pull, fk/pk)
• Hadoop HBase (big, pull, fk/pk)
• Object/relational mappers (small, pull, k/v)
• LINQ to Objects (big, pull, k/v)
• Reactive Extensions (big, push, k/v)

How would you fill in the three corners not listed above?

Related links:

• Big data is not enough
• Big data and humility
• coSQL (exploring the variety axis)

From Erik Meijer:

Functional Hacker := Think like a fundamentalist, code like a pragmatist.

Related posts:

• 85% functional language purity
• You wanted a banana but you got a gorilla holding the banana

Walter Bright made an interesting point in his talk at GOTO Aarhus this afternoon. He says that software developers like to think of their code as pipelines. Data comes in, goes through an assembly line-like series of processing steps, and comes out the end.

input  -> step1 -> step 2 -> step3 -> output

And yet code hardly ever looks like that. Most software looks more like a whirlpool than a pipeline. Data swirls around in loops before going down the drain.

These loops make it hard to identify parts that can be extracted into encapsulated components that can be snapped together or swapped out. You can draw a box around a piece of an assembly line and identify it as a component. But you can’t draw a box around a portion of a whirlpool and identify it as something than can be swapped out like a black box.

It is possible to write pipe-and-filter style apps, but the vast majority of programmers find loops more natural to write. And according to Walter Bright, it takes the combination of a fair number of programming language features to pull it off well, features which his D language has. I recommend watching his talk when the conference videos are posted. (Update: This article contains much of the material as the conference talk.)

I’ve been thinking about pipeline-style programming lately, and wondering why it’s so much harder than it looks. It’s easy to find examples of Unix shell scripts that solve some problem by snapping a handful of utilities together with pipes. And yet when I try to write my own software like that, especially outside the context of a shell, it’s not as easy as it looks. Brian Marick has an extended example in his book that takes a problem that appears to require loops and branching logic, but that can be turned into a pipeline. I haven’t grokked his example yet; I want to go back and look at it in light of today’s talk.

Related posts

• The D programming language
• Comparing the Unix and PowerShell pipelines

Last night I ran into Damian Conway at a speaker’s dinner for this week’s GOTO conference. He’s one of the people I had in mind when I said I enjoy hearing from the Perl community even though I don’t use Perl.

We got to talking about Norris’ number, the amount of code an untrained programmer can write before hitting a brick wall. Damian pointed out that there’s something akin to the Norris’ number for skilled programmers. He said that a few years ago he took this quote from Brian Kernighan to heart:

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.

He said he’d realized that he’d written some very complex code that was approaching his level of cleverness and that couldn’t move forward himself, much less get anyone else to take it ownership. “That’s when I decided to dial back the cleverness a bit.”

Related post: Two kinds of software challenges

One of the things that makes Russian novels hard to read, at least for Americans, is that characters have multiple names. For example, in The Brothers Karamazov, Alexei Fyodorovich Karamazov is also called Alyosha, Alyoshka, Alyoshenka, Alyoshechka, Alexeichik, Lyosha, and Lyoshenka.

Russian novel programming is the anti-pattern of one thing having many names. For a given program, you may have a location in version control, a location on your hard drive, a project name, a name for the program executable, etc. Each of these may contain slight differences in the same name. Or major differences. For historical reasons, the code for foo.exe is in a project named ‘bar’, under a path named …

I thought about this today when looking into a question about a program. A single number had different names in several different contexts. There’s the text label on the desktop user interface, the name of the C# variable that captures the user input, the name of the corresponding C++ variable when the user input is passed to the back-end numeric code, and the name of used in the XML file that serializes the variable when it goes between a database and a web server. Of course these should all be coordinated, but there were understandable historical reasons for how things got into this state.

Related posts

• Baklava code
• Software to slice bread
• Why Shakespeare is hard to read