Hypercore protocol deep dive

24 Jan 2019

2019/01/24: This is a Work In Progress (WIP). I will be live-updating this post as I work on the spike. If you want to get streaming updates without having to refresh your browser, open the DAT version in Beaker Browser and toggle the live reloading feature. Please feel free to talk to me about this on the fediverse as I work on it, perhaps via Mastodon.

Last modified: Thu, 24 Jan 2019 07:29:23 UTC

At the heart of Dat is the hypercore protocol. Understanding it is fundamental to grokking how and why the higher-level libraries and tools in the ecosystem work the way they do and have the properties they have.

This is a documentation of my study of the hypercore-protocol code. It’s meant to be a personal reference. For introductory material as well as general references, please see the references section.

Overview

The hypercore protocol is defined using protocol buffers in schema.proto. From this, message.js is generated via npm run protobuf.
Initial message sent when establishing a connection is a Feed message that includes the discovery key and a nonce¹. If both parties have the read key (public signing key of the hypercore), then they can replicate from the second message onwards via an encrypted connection that uses XSalsa20 to encrypt the messages with the read key and the nonce.

Limitations

Lack of general ephemeral messaging channel

For certain use cases (e.g., ability to share public keys in multi-writer replication/presence), it would be useful to have a generic and ephemeral (non-persisted) messaging channel. There is currently a DEP by Paul Frazee for using the extension message feature in the protocol to implement this (DEP-0006: Session Data)

Also see: DEP: Ephemeral message extension pull request. This PR was closed but I’m not sure why exactly as – unless I’m missing something – the privacy concerns can be addressed by signing the messages with the Feed key as part of the extension protocol itself (comment).

The discussion on the DEP’s pull request is very valuable and concerns the use cases that we have for Hypha also.

Note: there are ephemeral messages within the protocol (e.g., keepalive) but no general means to send arbitrary ephemeral messages.

References

How Dat works (blog post): documentation for the Dat protocol
DAT protocol spec
Protocol buffers

Number only used once. ↩︎