Back in 2016-2017, I was bored at Qualys, and I spent that boredom on Bitcoin. That period left a trail of proofs of concept and experiments, most of them long forgotten, and a handful of posts on this weblog: addresses, RPCs, a genesis block…

One of them had a longer life: chainstate, a C++ tool reading the chainstate directory of Bitcoin Core, which I wrote about in Bitcoin: Parsing the chainstate. The chainstate is where Bitcoin Core keeps the UTXO set: every transaction output that has not been spent yet. That is all the coins that can still move, and so, once grouped by address, what every wallet actually holds. I even ran a small website, balances.crypto-nerdz.org, selling the whole dump of balances for a few satoshis.

I recently rebuilt it in Rust, or rather had Claude regenerate it from the old tool and from Bitcoin Core’s sources. Here’s rs-chainstate.

What’s in the chainstate Link to heading

The chainstate is a LevelDB database. Each unspent output is one record:

  • the key is C, the transaction id and the output index;
  • the value is the height of the block holding the transaction (with a coinbase bit), then the amount, then the output script.

Everything is compressed the Bitcoin Core way: integers are its own VARINT (not the CompactSize used in transactions), amounts go through CompressAmount, and scripts through ScriptCompression, which squeezes the common script types into a one-byte tag and a hash or a key. On top of that, every value is XORed with an obfuscation key stored in the database itself, so that antivirus software stops flagging LevelDB files because a transaction happened to contain a virus signature.

Decompressing the script is the same switch as in the C++ version, minus the hand-written base58, bech32 and public key decompression, which now come from the bitcoin crate:

let script = match size {
    0 => { /* P2PKH: 20-byte key hash */ }
    1 => { /* P2SH: 20-byte script hash */ }
    2..=5 => { /* P2PK: a 32-byte key, 4 and 5 meaning it is stored compressed */ }
    n => ScriptBuf::from_bytes(take(buf, usize::try_from(n - 6)?)?.to_vec()),
};

Everything else, SegWit and Taproot outputs included, is stored as a raw script, and turned into an address by the crate.

Don’t open bitcoind’s database Link to heading

The 2018 post already warned that the chainstate had to be copied before being parsed. Opening a LevelDB database is never read-only: it replays the log, rewrites the MANIFEST and deletes obsolete files. Do that on bitcoind’s own directory and you are in for a reindex.

This time the tool enforces it. snapshot copies the directory without opening it as a database, and drops a marker file into the copy; the other commands refuse to open a directory without that marker.

They also refuse a directory whose LOCK is held, and that one was a nice trap: rusty-leveldb locks with flock(2), while bitcoind’s LevelDB uses fcntl(2) locks. On Linux, the two don’t see each other, so rusty-leveldb would happily open a database bitcoind is writing to. rs-chainstate probes the fcntl lock itself, with F_GETLK.

Running it Link to heading

The whole thing, from bitcoind’s directory to a CSV of balances:

$ cat export
#!/bin/sh

cd ~/dev/rs-chainstate
rm -fr chainstate
cargo run --release -- snapshot /data/bitcoin/data/chainstate chainstate
cargo run --release -- dump-balances --p2pk-as-p2pkh --skip-zero --header --minimum 1000 -o balances_1000.csv ./chainstate

$ time sh ./export
     Running `target/release/rs-chainstate snapshot /data/bitcoin/data/chainstate chainstate`
snapshot written to chainstate
     Running `target/release/rs-chainstate dump-balances --p2pk-as-p2pkh --skip-zero --header --minimum 1000 -o balances_1000.csv ./chainstate`

________________________________________________________
Executed in  356.39 secs    fish           external
   usr time  287.56 secs   10.00 micros  287.56 secs
   sys time   15.45 secs  976.00 micros   15.45 secs

Six minutes, for a 15 GB chainstate read from a spinning disk. The snapshot itself is instant: on btrfs, the copy shares the original file’s blocks instead of duplicating them. In 2018 the chainstate weighed 2.8 GB and held about 59.5 million outputs; today it’s over 165 million.

A few words on the options:

  • dump-balances decodes every output and sums the amounts by address, in one pass. dump alone prints the outputs, one per line, and balances can aggregate such a dump later.
  • --p2pk-as-p2pkh: the oldest coins, most of the 2009-2010 coinbases, are pay-to-pubkey outputs, which have no address. Block explorers show them as the P2PKH address of their key, since the same private key spends both; this option does the same.
  • --skip-zero and --minimum 1000 drop empty outputs and dust.
  • Balances are aggregated in memory, which peaked around 11 GB here. If that doesn’t fit, --temp-dir spills them to disk, partitioned by address hash, for the same output.

Outputs without an address show up as p2pk:<pubkey> (without --p2pk-as-p2pkh), multisig:<script> or nonstandard:<script>.

The result Link to heading 

$ head -6 balances_1000.csv
address;amount;utxo_count
34xp4vRoCGJym3xR7yCVPFHoCNxv4Twseo;24859759275302;5108
3M219KR5vEneNb47ewrPfWyb5jQ2DjxRP6;17734150534516;128
bc1ql49ydapnjafl5t2cp9zqpjwe6pdgmxy98859v2;14084998554169;56
bc1qgdjqv0av3q56jvd82tkdjpy7gdp9ut8tlqmgrpmv24sq90ecnvqqjwvw97;13001007903066;51
bc1qjasf9z3h7w3jspkhtgatgpyvvzgpa2wwd2lr0eh5tx44reyn2k7sfc27a4;9693240685262;8

A 2.7 GB file: 51,906,457 addresses holding at least 1,000 satoshis, across 156,685,439 outputs, for a total of 20,099,132 BTC. 970,366 of those addresses hold at least one bitcoin. The richest one sits on 248,597 BTC.

Some lines further down are fun to look at:

$ grep -E '^(1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa|nonstandard:76a90088ac);' balances_1000.csv
nonstandard:76a90088ac;260936304319;23
1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa;5767544735;80439
  • 1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa is the genesis block’s address. Its 50 BTC coinbase was never added to the UTXO set, so it isn’t in the chainstate at all: these 57.67 BTC, in 80,439 outputs, are tributes people have been sending to it ever since.
  • 76a90088ac is OP_DUP OP_HASH160 0 OP_EQUALVERIFY OP_CHECKSIG, a P2PKH script with an empty hash, which no key can ever satisfy. Those are the 2,609 BTC that Mt.Gox burned with a buggy transaction in 2011.

Next Link to heading

Nothing much, honestly. The C++ version was an excuse to learn how Bitcoin stores its state, and this one is the same itch scratched again, with a fraction of the code and none of the hand-rolled crypto. The sources are on GitHub.