Introducing JSDB

Yesterday, I released version 1.0 of JavaScript Database (JSDB), a new database for Node.js optimised for use with Small Web sites and apps.

It does things a little differently to other databases.

It’s an in-memory JavaScript database that persists to an append-only transaction log.

And it’s all JavaScript.

What I mean by that is that even the data is stored as JavaScript code. Not JSON. JavaScript.

(It’s also very fast and easy to use.)

Get started

You will need Node.js installed¹.

1. Set up your project and install JSDB.

# Create a folder to hold your project and switch to it.
mkdir lovecraft-country
cd lovecraft-country

# Initialise it with a package.json file.
npm init -f

# Install JSDB.
npm i @small-tech/jsdb

# Create a file to hold your code.
touch index.js

2. Open a database and write something to it.

const JSDB = require('@small-tech/jsdb')

// Open your database (creating it if it doesn’t exist).
// It will be stored in a directory called “db”.
const db = JSDB.open('db')

// Create a “table” on it. Tables can be arrays or objects.
// This is a list of the Lovecraft Country episodes and
// their ratings from IMDB. (It’s missing the last three.)
if (!db.episodes) {
  db.episodes = [
    {title: 'Sundown', rating: 8.4},
    {title: 'Whitey’s on the Moon', rating: 7.1},
    {title: 'Holy Ghost', rating: 7.7},
    {title: 'A History of Violence', rating: 7.6},
    {title: 'Strange Case', rating: 7.1},
    {title: 'Meet me in Daegu', rating: 8.4},
    {title: 'I am.', rating: 7.1}
  ]
}

3. Run it.

node .

In your terminal, you should see output similar to the following:

💾 ❨JSDB❩ No database found at …lovecraft-country/db; creating it.
💾 ❨JSDB❩ Creating and persisting table episodes…
💾 ❨JSDB❩  ╰─ Created and persisted table in 2.169 ms.
💾 ❨JSDB❩ Table episodes initialised.

The table is created and persisted into a file called db/episodes.js

Data as code

To dispel the magic and see how the sausage is made, take a look at what’s inside it this file by running the following command:

cat db/episodes.js

You should see something like this:

globalThis._ = [ { title: `Sundown`, rating: 8.4 }, { title: `Whitey’s on the Moon`, rating: 8.4 }, { title:
`Holy Ghost`, rating: 8.4 }, { title: `A History of Violence`, rating: 8.4 }, { title: `Strange Case`, rating: 8.4 }, { title: `Meet me in Daegu`, rating: 8.4 }, { title: `I am.`, rating: 8.4 } ];
(function () { if (typeof define === 'function' && define.amd) { define([], globalThis._); } else if (typeofmodule === 'object' && module.exports) { module.exports = globalThis._ } else { globalThis.lovecraftCountryEpisodes = globalThis._ } })();

Yes, that’s plain-old JavaScript. Not JSON. JavaScript.

And it’s a bit hard to read. If you were to prettify it, this is what you’d get:

globalThis._ = [
  { title: `Sundown`, rating: 8.4 },
  { title: `Whitey’s on the Moon`, rating: 8.4 },
  { title: `Holy Ghost`, rating: 8.4 },
  { title: `A History of Violence`, rating: 8.4 },
  { title: `Strange Case`, rating: 8.4 },
  { title: `Meet me in Daegu`, rating: 8.4 },
  { title: `I am.`, rating: 8.4 }
]
;(function () {
  if (typeof define === 'function' && define.amd) {
    define([], globalThis._)
  } else if (typeofmodule === 'object' && module.exports) {
    module.exports = globalThis._
  } else {
    globalThis.lovecraftCountryEpisodes = globalThis._
  }
})()

This is how JSDB stores data. It stores it as code in a format I call JavaScript Data Format (JSDF).

JavaScript Data Format (JSDF)

The first line is a single assignment of all the data that existed in the table when it was created or last loaded. In this case, it is basically the same array declaration you wrote to create the table.

The second line is a UMD-style declaration. What this means is that you could, if you wanted to, use require() in Node to load a JSDB table in directly or even load a JSDB table in using a script tag from the browser.

For example, create a file called index.html in the same folder with the following content:

<script src="db/episodes.js"></script>
<h1>Lovecraft Country</h1>
<p>Episodes and ratings from <a href='https://www.imdb.com/title/tt6905686/episodes?season=1'>IMDB</a>:</p>
<ul>
<script>
  episodes.forEach(episode => {
    document.write(`<li><strong>${episode.title}</strong> ⭐ ${episode.rating}</li>`)
  })
</script>
</ul>

Display this web page in your browser using something like Site.js and you should see the following.

Important security note

JSDF is not a data exchange format.

Since JSDF is made up of JavaScript code that is evaluated at run time, you must only load JSDF files from domains that you own and control and have a secure connection to.

To exchange data, use JSON, that’s what it’s for.

Updating data

So what happens when we update the existing data?

To see this, let’s add the missing episodes from the first season to our database and watch what happens to the contents of the table as we do.

Start by opening up an interactive Node.js session in one Terminal window (type node) and tailing (following) the database table from another one:

tail -f db/episodes.js

First, let’s open our database again in the interactive Node.js session:

JSDB = require('@small-tech/jsdb')
db = JSDB.open('db')

Then, let’s add the missing episodes for the first season, one at a time:

db.episodes.push ({title: 'Jig-a-Bobo', rating: 8.4})
db.episodes.push ({title: 'Rewind 1921', rating: 8.6})
db.episodes.push ({title: 'Full Crcle [sic]', rating: 7.0})
db.episodes.push ({title: 'Happy ending', rating: 10.0})

As you add them, you should see them appear at the end of the table. When you’ve added all four, your table should resemble the following:

// first line is the same as before
// second line is the same as before
_[7] = { title: `Jig-a-Bobo`, rating: 8.4 };
_[8] = { title: `Rewind 1921`, rating: 8.6 };
_[9] = { title: `Full Crcle [sic]`, rating: 7 };
_[10] = { title: `Happy ending`, rating: 10 };

And now you know how JSDB stores data that gets added or changed: it adds the code to affect that change to the end of the table. This is what makes it an append-only transaction log. And this is what keeps writes fast (in the single digit milliseconds for the most part on my development laptop) as they’re being streamed to the end of the file.

Ah, but we’ve made some errors, so let’s correct them.

First of all, there is no 11th episode called “Happy Ending”, so let’s remove that from the list:

db.episodes.pop()

Finally, we made an intentional spelling mistake in the title of episode 9. Let’s fix it in our interactive Node.js session:

db.episodes.where('title').includes('Full').getFirst().title = 'Full Circle'

Woah! What’s all that new stuff?

It’s a query in a language I call JavaScript Query Language (JSQL).

We’ll see a bit more of JSQL a little later on.² But for now, let’s take a penultimate look at the JSDF file for the episodes table. Here’s what it should look like now:

globalThis._ = [ { title: `Sundown`, rating: 8.4 }, { title: `Whitey’s on the Moon`, rating: 7.1 }, { title:
`Holy Ghost`, rating: 7.7 }, { title: `A History of Violence`, rating: 7.6 }, { title: `Strange Case`, rating: 7.1 }, { title: `Meet me in Daegu`, rating: 8.4 }, { title: `I am.`, rating: 7.1 } ];
(function () { if (typeof define === 'function' && define.amd) { define([], globalThis._); } else if (typeofmodule === 'object' && module.exports) { module.exports = globalThis._ } else { globalThis.episodes = globalThis._ } })();
_[7] = { title: `Jig-a-Bobo`, rating: 8.4 };
_[8] = { title: `Rewind 1921`, rating: 8.6 };
_[9] = { title: `Full Crcle [sic]`, rating: 7 };
_[10] = { title: `Happy ending}`, rating: 10 };
delete _[10];
_['length'] = 10;
_[9]['title'] = `Full Circle`;

Hmm, all our changes are there but do you see a problem?

Because I do.

We’ve deleted the last episode but the actual data for it is still in our table, on disk. In this case, that’s not much of a problem. But what if instead of a TV episode, it was a private piece of information that you wanted to delete? For the sake of privacy, it’s important that data we think is deleted is actually deleted. This is a problem that all append-only logs suffer from due to their inherent nature and it’s one that JSDB solves using compaction.

Compaction

By default, every time a table is loaded, any changes that were made to it in the last session are compacted into the single-line initialisation statement on the first line. This compaction process is important for various reasons:

• It preserves privacy by actually deleting and updating changed data everywhere.
• It makes subsequent loads faster as only a single assignment statement is run to create the data graph in memory instead of possibly hundreds or thousands of statements.

To see compaction in action, exit your current interactive Node.js session and start a new one. Then, open the database again:

JSDB = require('@small-tech/jsdb')
db = JSDB.open('db')

You should see output that resembles the following:³

💾 ❨JSDB❩ Loading table episodes…
💾 ❨JSDB❩  ╰─ Loading table synchronously.
💾 ❨JSDB❩  ╰─ Table loaded in 1.515 ms.
💾 ❨JSDB❩ Compacting and persisting table episodes…
💾 ❨JSDB❩  ╰─ Compacted and persisted table in 4.371 ms.
💾 ❨JSDB❩ Table episodes initialised

Now, if you take one final look at the episodes table, you should see that it’s back to being two lines and that the first line, shown below, includes all the changes we made so far:

globalThis._ = [ { title: `Sundown`, rating: 8.4 }, { title: `Whitey’s on the Moon`, rating: 7.1 }, { title:
`Holy Ghost`, rating: 7.7 }, { title: `A History of Violence`, rating: 7.6 }, { title: `Strange Case`, rating: 7.1 }, { title: `Meet me in Daegu`, rating: 8.4 }, { title: `I am.`, rating: 7.1 }, { title: `Jig-a-Bobo`,rating: 8.4 }, { title: `Rewind 1921`, rating: 8.6 }, { title: `Full Circle`, rating: 7 } ];

JavaScript Query Language (JSQL)

You got a tiny introduction to JavaScript Query Language (JSQL) in the previous section. Now that we have all the episodes correctly entered in our database, let’s use JSQL one last time to get the names of the episodes that have ratings higher than 8 stars.

In the same interactive Node.js session as before, enter the following query:

// Get the highly rated episodes.
highlyRatedEpisodes = db.episodes.where('rating').isGreaterThan(8).get()

// Print them out nicely.
highlyRatedEpisodes.forEach(episode => console.log(`${episode.title} (⭐ ${episode.rating})`))

You should see the following list:

• Sundown (⭐ 8.4)
• Meet me in Daegu (⭐ 8.4)
• Jig-a-Bobo (⭐ 8.4)
• Rewind 1921 (⭐ 8.6)

I hope this has whetted your appetite and that you’ll have a play with JSDB and consider how you could use it in your own apps.

Also, I’m in the process of integrating it into Site.js so you can use it when creating Small Web sites and apps.

You can find further details and examples in the JSDB documentation.

Enjoy and, no matter what you do, please do not use it to farm people for their data.