It's microcosm's first half-birthday! I'm marking the moment with a slight change in trajectory: from building out APIs for ATProto, to inviting in everyone to build apps on them. Things like:

  • Games & events: helping launch at://advent of atproto (coming soon!)

  • More & better docs: guided demos, how-tos, and other on-ramps for devs new to microcosm!

There is still a lot to keep building, but that work is ready for more collaborative input too.

half-birthday portrait: four raspberry pi's with googly eyes on a wooden surface with messy networking and power cables around them. each has a poorly-drawn birthday hat edited over the photo. the edge of an old thinkpad can be seen on the left side of the image, and a power bar in the upper right.

the front-most pi is a raspberry pi 4B with eyes on its USB-A ports and a tiny heatsink resting on its CPU.

behind it is an original model B with eyes on its protruding sd card.

in the centre is a raspi 5 with NVME card hat and eyes over its fan shroud

the final pi is another 5 with eyes again on its fan shroud (its NVME is below the pi)

a microcosm

ATProto is a transformative new way to build social apps that centres users' control over their own data.

Microcosm is a collection of building blocks to ship and experiment with: self-hostable APIs, independent community infrastructure, and small utility apps for the ATProto developer ecosystem. A quick rundown of what's already online:

Constellation indexes every social interaction globally.

Spacedust transforms the global event firehose into a global interaction firehose: the live-updating light-weight complement to Constellation.

Slingshot pre-caches content and identities from the firehose to an edge near you, for fast access and high availability, closing the UX gap for client-first ATProto apps.

UFOs API keeps statistics and retains most-recent-records feeds for every ATProto collection.

Full-network firehoses to run without relying on Bluesky infrastructure (and outside of US jurisdiction):

And some utilities: UFOs lexicon explorer, a universal atproto notifications, a relay comparer, a (slightly outdated) link explorer.

~~~

So, that's where things are at six months in! If you're curious about things have gone, I'll get into it below. What's next is at the end.


here, a microcosm grafana dash to break up the flow:

a large, busy grafana dashboard with 17 widgets of various types. network stats, storage available, jetstream info, link counts, CPU stats, hardware temperature, request rates, and counts by NSID prefix

The rest of this post is a little less structured, because writing things well takes me forever.

Things that went differently than expected

  • Nobody wants very few people actually want to run infrastructure, to self-host things. So community infrastructure became a core part of microcosm.

  • Nobody wants not a lot of people want to build backends. Or maybe: lots of people want to play around building client-first apps. All microcosm APIs became CORS-friendly, with networking architecture to support this approach.

  • Nobody used the UFOs recent-records APIs to build feed-based app prototypes, at least that I'm aware of. Hoping to change this with some tutorials soon!

  • Everyone loves #indiesky but not many want to actually build on it. The microcosm firehoses are under-utilized. (They're also probably not that easy to find out about)

  • Raspberry pi computers are more capable than I would have guessed. Constellation still runs on a single raspberry pi 5, and not because I can't move it to a Real Machine. It just has so much read performance left that it hasn't been necessary. The smallest microcosm service runs on an original model B 699MHz ARM6/220MB from 2012.

Failures

  • A week spent on a survey of linked data in atproto, that took too much time and which I did not manage to pull together into a post. (did get a custom tool out of it)

  • Months spent building the data storage, APIs, and an app for UFOs delayed progress on other more-app-oriented projects. Sunk time debugging storage engine deadlocks and side-questing a rewrite of Cloudflare's cardinality-estimator crate after finding memory safety violations.

  • Couple weeks building who-am-i, an ATProto DID-verifying service for demos. I had hoped that an extremely limited scope could unlock a simple service with good UX. Landed at a complicated, unreliable, not-great-UX tool that doesn't quite meet the scope I had set out for. Auth is hard.

  • The notifications demo app. A lot has worked out with it! But it took too long, and thanks to limitations with Web Push, a lot of state has to be pushed up to the server. Less like the thin push adapter for Spacedust I wanted; more of a full backend app.

  • Writing updates about microcosm! They're valuable, but take me a long time; I spent most of it building instead. I wrote two whole newsletters for my Github sponsors, but at this point I don't know if they reached anyone. Seriously, where do they go?

  • Getting grants. I blew an application by spending all my budgeted time on what turned out to be only half the application when I went to submit it.

Everything else happened

  • 30 people have sponsored me on github/supported me on ko-fi!!! Still feels surreal. I have only gratitude and gratefulness to everyone keeping this going.

  • I published a slightly-viral blog post shortly after launching microcosm, about scaling atproto down. I think it helped move the conversation (and the goalposts) a little.

  • Talked at atproTO in Toronto

  • Launched multiple free public independent full-network relays including under EU jurisdiction, for a lot less than two million dollars.

  • deer.social shipped custom verifiers with Constellation.

  • Hosted a public instance of the amazing AppViewLite for a couple months. (retired for now, it got overrun by crawlers)

  • Helped (or in some cases just watched) a steady stream of self-hosters get their own Constellation instances set up!

  • Moved microcosm source code to tangled.sh (code hosting on ATProto!)

  • Contributed features, bug fixes, and bug reports to for Bluesky-owned projects, community SDKs, and other adjacent open-source projects.

  • Built and launched all the stuff listed earlier,

  • ...with help from dozens of contributors in the community!

  • Kept all the infra online, most of it raspberry pi-self-hosted.

  • About a dozen apps shipped using microcosm services.

And probably a lot more that I'm forgetting!


What's next

More writing. I've got backlogged posts to finish for protocol proposals, a linked-data notation spec, an indie relay operator experience report, posts about the microcosm networking setup, ... and of course, more documentation!

More collaboration. With a vision laid out and initial versions of the core microcosm services shipped, it's time to engage with aligned projects like at://advent of ATProto, building up the dev community. We go further together.

API evolution. Every microcosm service ships with APIs that work for the known use-cases in scope. Since new use-cases often challenge the existing system constraints, they are expected to evolve, if imperfectly. My intention is to invest more time designing nicer APIs once changes in requirements slow down for a service.

Infra improvements. Redundant instances and network paths for fewer single points of failure, performance-oriented deploys in more locations globally for the best possible experience for everyone.

New services. Tools for backfilling, firehose cutover, trusted timestamps, an OAuth client assertion backend, a sidecar index, an xrpc notifications delivery service, and ultimately something like graphql for atproto.

Terms of use, code of conduct, privacy policy. As people are starting to ship more on the services, these have to exist. There won't be any surprises.

~~~

If you've read this far: thank you!

Thanks especially to every single person who has sponsored and supported, and double-especially to everyone opening issues and pull requests.

Triple-especial thanks if you built something with microcosm, I love hearing about it 💜

—@bad-example.com