This post is meant to help you think through message propagation on the Farcaster network. I wrote a few casts about this earlier (see below) but figured I would write a longer form now.
The goal is of this article is to explain the technical details of what I'm seeing while building on the protocol, in case it helps you build. 🙂
Varun and Dan have written in length about the protocol, read architecture doc here. In quick summary, every app / client on Farcaster reads and writes messages from an underlying network of hubs (nodes). Each hub stores a state of the network and communicates (aka gossips) any incoming messages to other hubs its peered with. Through this gossiping between the hubs, all hubs eventually receive the same messages and can serve it back to the clients.
However, this does mean that there will be times where individual hubs have a different state of the network because they have not yet finished gossiping all the messages to other hubs. This can happen either at the hub level if some hub is running slow or at the client level where the client can have bottlenecks and not push data to their hub fast enough. In the latter case, the client has data that users of that client can see without problems, but that data is not available to the rest of the network. This creates disjointed experiences between clients that are built on the same underlying protocol.
The thing to note here is that if you're coming to Farcaster after having built on blockchains, this can be a bit surprising. Blockchains have a deterministic current state which is the head of the chain and apps know whether they are reading the latest block in the chain or not. Farcaster is not a blockchain, and thus, the concept of time within Farcaster is vague. Hubs don't know whether they have the latest message. The idea being that for social data having the latest message is less important compared to financial data. In the case of financial data, if you don't have the latest record, you don't know how much money was moved and who owns what. Dan and Varun made a bet that social data does not need that level of finality. That bet allows Farcaster to move data at much cheaper prices than a blockchain would be able to. Writing all Farcaster data to blockchains would be extremely cost prohibitive, something that stopped prior decentralized social networks from growing.
I personally think this is a good bet, at least for the time being. If blockchains ever get cheap enough, maybe Farcaster can introduce protocol upgrades that respect finality more. However, till then, current version of the protocol allows sufficient decentralization with eventually consistent state. What does this mean for you as a developer for Farcaster? It means accounting for states where hubs and clients have different data, especially during times of peak network load that Farcaster has been experiencing lately.
See these casts to get a sense:
This level of sudden growth is insane and Merkle has been handling it extremely well. Neynar gets a fraction of that load and we have been fully under water for the last ten days, scrambling to get our server traffic under control.
I expect such issues to reduce as the network scales and grows more resilient, though we are far from that yet. I am optimistic that this is just the beginning of the growth curve. There will be growing pains as with everything and the best products will figure out how to provide a robust user experience in spite of such gaps. There will be innovations not only at the consumer product level but also at the developer product level e.g. better debugging tools - take a cast hash or fid as input and see the state of the cast or the user across multiple known hubs and public replicators. I have been pushing developers to try a version of this by checking the network state on clients that run on different hubs e.g. Supercast vs Warpcast.
I am not sure what all the solutions look like here but I am curious to see what people come up with.
At Neynar, our main focus has been on scaling the Hubs and replicator infrastructure to better support the growing traffic. Manan and I will write more about our solution stack in detail in a different post. If you have been building using Neynar, you know we've been seeing issues. I appreciate you bearing with us.
If you read this far and have ideas (or corrections I should make to this article), reach out @rish on Farcaster. If you chose to build on Farcaster, we're glad to have you! 🪐