Hypha Spike: DAT 1
Source code (See iteration plan for links to specific tags.)
Following on from Hypha Spike: Diceware, this spike aims to explore:
- Creating an in-browser DAT data store using the keys generated in the previous spike
- Replicating that datastore over a web socket connection with the always-on node and making it available over UTP
- ✔ Create hypercore in browser using the generated public and private signing keys.
- ✔ Expose hypercore state and events on the page itself.
- ✔ Replicate hypercore to server using websocket connection.
- ✔ Join hyperswarm and replicate hypercore from server from a native client.
Note: since the original spike, a further iteration has also added WebRTC support.
In-browser hypercore gotcha
When creating a hypercore in browser by manually specifying the read and write keys, you must convert the keys to Node.js’s Buffer type, you cannot use ArrayBuffer. Just submitted a pull request to the hypercore readme to make this explicit as this caught me out initially:
secretKeyare Node.js buffer instances, not browser-based ArrayBuffer instances. When creating hypercores in browser, if you pass an ArrayBuffer instance, you will get an error similar to
key must be at least 16, was given undefined. Instead, create a Node.js Buffer instance using Feross‘s buffer module (
npm install buffer). e.g.,
const storage = someRandomAccessStorage const myPublicKey = someUint8Array const Buffer = require('buffer').Buffer const hypercorePublicKeyBuffer = Buffer.from(myPublicKeyAsUint8Array.buffer) const hypercore = hypercore(storage, hypercorePublicKeyBuffer)
Callback in onwrite hook gotcha
If you implement the
onwrite hook in the options passed to the hypercore constructor, you must explicitly call the passed callback at the end of your handler (
cb()) or your appends will not work. (Pull request to update docs.)
WebSocket/replication gotcha with budo and livereload
Initially, I was getting the following errors while trying to replicate over the web socket connection:
The connection to wss://localhost/livereload was interrupted while the page was loading. The connection to wss://localhost/hypha/f86a223b93b19929eee4e402480ac4d69ad4d8342b2f39b03f3dfd7d0fafbe93 was interrupted while the page was loading.
[Error] WebSocket connection to 'wss://localhost/livereload' failed: Compressed bit must be 0 if no negotiated deflate-frame extension [Error] WebSocket connection to 'wss://localhost/hypha/78575ce623d7e7ef8e55c7d888e36f64c4fcea9404b1073ca517f94cc32b08b4' failed: Compressed bit must be 0 if no negotiated deflate-frame extension
The issue is that budo’s web socket server and mine are clashing. It looks like Jim’s had it turned off also in his Shopping List Example (it defaults to off).
- A strong passphrase is generated via EFF’s Diceware Word List.
- From the passphrase, Ed25519 (signing) and Curve25519 (encryption) key material is derived via session25519.
- The signing keys are used to create a hypercore in the browser.
- The hypercore is replicated via web socket to the unprivileged always-on node (server)
- The always-on node joins the hyperswarm and announces the hypercore via its discovery key.
- A native client is run with the read key of the hypercore. It calculates the discovery key from the read key and uses hyperswarm to find and replicate the hypercore from the always-on node that originated in the browser.
This spike proves only a subset the absolute basics of the Hyphanet design. See areas for future study.
Areas for future study
- ✔ Browser-to-browser discovery and replication via WebRTC. See Hypha Spike WebRTC 1
Jim Pick’s DAT Shopping List demo (Tokyo version)