Hubble first launch: 1500x cheaper getRepo for Bridgy Fed

Hubble is a mirror for public AT Protocol data. It consumes atproto sync, and produces atproto CARs. It's working, and will be officially released veeerrrry soon. but first, this!

Hubble's purpose is to mirror the whole Atmosphere, but another very interesting (smaller!) use-case came up.

Ryan

@snarfed.org

2mo

sadly no, we're still trying to optimize our getRepo for cost, it's still way too expensive for us to serve, so it's largely disabled right now. 😢 we need to figure out a better answer here. cc @quillmatiq.com

optimize `getRepo` · Issue #88 · snarfed/arroba

Right now, getRepo is too slow. It can take 30-60s for big repos with 10ks of records. Should ideally be at least 10x faster. cc @jcalabro @whyrusleeping

github.com

1

Bridgy Fed bridges the Atmosphere and the Fediverse. You can follow friends on Mastodon from your Bluesky account, and seamlessly jump into in conversations that span protocols. It's very cool! So what's a getRepo and what does it have to do with Hubble?

Atmosphere sync

getRepo (legal name com.atproto.sync.getRepo) is one of two core methods to sync from PDSes to the network. The other is the firehose (or com.atproto.sync.subscribeRepos), and they compliment each other:

• Firehose: a stream of verifiable data changes.

• getRepo: a snapshot of verifiable data state.

Atmosphere sync is a careful coordination of these: fetching initial account data (getRepo, "backfill"), tracking changes (firehose), and recovering from errors by resetting (getRepo, "resync").

Without getRepo, there was a Bridgy-shaped hole in the Atmosphere. Bridgy Fed emitted changes on the firehose, but you couldn't backfill initial state or reset after errors for bridged Fediverse accounts.

It's one endpoint, Micheal. What could getRepo cost?

A lot, actually!

getRepo produces a complete archive of all public data for one account, structured as a Merkle Search Tree (MST), serialized into a Content-Addressable aRchive (CAR). Collecting all this content-addressed content, from a tree with up to millions of pieces of content, adds up to a lot of random reads.

Bridgy Fed stores atproto content in Google Cloud Datastore, which is highly scalable and pretty efficient for producing verifiable data changes for the firehose. But with pay-per-entity-read pricing and an entity-per-piece-of-content-stored design, the cost to serve getRepo can approach $1 in datastore reads alone for a single export of a very large account.

With many apps syncing the Atmosphere these days, serving getRepo like this would be unsustainable for Bridgy Fed.

Mirror for a PDS

Hubble is an alternate place you can make getRepo requests. It's kind of like how apps subscribe to one relay for the global firehose instead of subscribing to each and every PDS themselves, but the subscription endpoint is the exact same shape for both,

a whole-Atmosphere Hubble instance is one place you can call getRepo, with the exact same shape as if you were talking to a PDS. Atmosphere interop! (more about this if you're curious).

This same-shape thing means the reverse works too, of course! You can listen directly to any PDS "firehose" with any firehose tool. Behold, the Eurosky PDS firehose: pdsls.dev/firehose?instance=wss://eurosky.social

ALT

screenshot of pdsls's firehose page, subscribed to wss://eurosky.social. only the top summary is shown, with uptime (2m57s), total events (965), events/sec (5.0), and top collections (bsky like, post, repost, follow, postgate).

So… what if you ran Hubble, but subscribed it directly to Bridgy Fed, instead of the whole network? We tried it, and

it pretty much just worked?!

It took about 5 mins over a call with the wonderful Bridgy Fed folks for us to figure out how to set it up:

• Bridgy Fed added an auth check to getRepo.

• The Hubble mirror makes auth'd getRepo requests, so it can backfill from Bridgy Fed and stay in sync.

• Bridgy Fed serves everyone else a redirect, sending their getRepos over to the mirror!

We've been live with this in production for a bit over a week now. The Hubble mirror has served terabytes of archives, billions of records, for millions of getRepo requests already.

For the Atmosphere, Bridgy Fed can now be synchronized. 65,000 accounts, now truly part of the network!

For Bridgy Fed, it only has one downstream getRepo consumer, at negligible cost after initial backfill.

For microcosm it's one little USD$10/mo VPS¹, the chance to load-test Hubble under real backfill demand, and a way to help keep the network healthier for everyone 😌

If you have a full-network Tap instance, a lightrail server, or have been playing with the JetstreamV2 pre-release, you are probably already finally synchronized with Bridgy Fed!

Bridgy Fed stays in control

So microcosm is serving requests for data that Bridgy Fed holds. What happens if this microcosm ╳ Bridgy Fed collab falls apart?²

The microcosm threat is actually pretty limited:

1. 1.

   The archives are signed (that's the A for Authenticated in AT Protocol). Whatever Hubble serves is verifiable. Hubble can't tamper undetected.

2. 2.

   Bridgy Fed serves a 302 Found redirect to the mirror, a non-permanent redirect. Bridgy Fed is always in the loop for every getRepo request, and can turn it off instantly.

And of course, they could switch to self-hosting the mirror. Like all microcosm projects, Hubble is open-source:

Hubble's getRepo and what's next

Hubble is built to serve a lot of getRepo³. It's doing well on the Bridgy Fed-sized mirror, and looking good so far at whole-network scale too.

fig (aka:[phil])

@bad-example.com

1d

The Paul Test hubble exports @pfrazee.com 2.4x faster than bluesky's reference pds deployment cold cache, no compression, bad laptop wifi

fig (aka:[phil])

@bad-example.com

forgot i made metrics by record counts for this too: 1.33M records/sec exported to STAR-lite almost 700K/sec exported as CARs (again this is limited by 1gbit/sec network egress, though STAR-lite is also *much* cheaper to produce)

5

I'll write more about how Hubble works soon. But it's basically cheaper because it doesn't have to be a full PDS implementation. It gets away with no content-addressing internally, no stored Merkle Search Tree at all, and no pay-per-read cloud storage.

Next up for microcosm is to officially, publicly, properly, for real announce the whole-network Hubble mirror launch. I'm busy wrapping up details, it's really almost finally there!

(it has been a journey, but we got a sync1.1 library out of it, which will hopefully be the next post-announce post!)

This is one of a series of posts about building Hubble! In case you missed the original announcement:

Microcosm runs on community support! If you liked this article, if you like community infra, or if you like when the whole Atmosphere can be synchronized:

more footnotes:

• clickbait⁴

• the bluesky thing 👀⁵