Aral Balkan

Mastodon icon RSS feed icon

Is the fediverse about to get Fryed? (Or, “Why every toot is also a potential denial of service attack”)

Screenshot of Stephen Fry’s Mastodon account banner on mastodonapp.uk

Stephen is a big fish to fry. (I’m here all week.)

Warning: the fediverse is about to get Fryed.

Stephen Fryed, that is.

Following the recent takeover of Twitter by a proto-fascist billionaire man-baby, people have been fleeing1 to the fediverse2. Among them are folks who, on Twitter, at least, had millions of followers like Greta Thunberg and, more recently, Stephen Fry.3

“Well, surely that’s a good thing? It’ll get everyone talking about the fediverse, decentralisation, and maybe even that Small Web thing you keep harping on about all the time, Aral, no?”

Well, yes and no… you see, there is such a thing as too much of a good thing. And, on the fediverse today, that appears to be “engagement when you’re popular.” In fact, it could be deadly (to Mastodon instances, that is).

Read on and I’ll try to explain what I mean by using my own account as an example.

Needless to say, I’m not a celebrity.

And yet, on the fediverse, I find myself in a somewhat unique situation where:

  1. I have my own personal Mastodon instance, just for me.4

  2. I’m followed by quite a number of people. Over 22,000, to be exact.5

  3. I follow a lot of people and I genuinely enjoy having conversations with them. (I believe this is what the cool kids call “engagement”.)

Unfortunately, the combination of these three factors creates a perfect storm6 which means that now, every time I post something that gets lots of engagement, I essentially end up carrying out a denial of service attack on myself.

Mastodon: denial-of-service as a service?

Yesterday was my birthday.

So, of course, I posted about it on my Mastodon instance.

It got quite a few replies. And, because it’s only polite, I started replying to everyone with thank-you messages.

Oh, no, you poor, naïve man, you. What were you thinking?!…

I’ll let my friend Hugo Gameiro, who runs masto.host and hosts my instance, explain what happened next:7

You just get a lot of engagement and that requires a ton of Sidekiq power to process.

For example, let’s look at your birthday post …  besides requiring thousands of Sidekiq jobs to spread your post through all their servers (you have 23K followers, let’s assume 3K different servers8), as soon as you create the post 3K Sidekiq jobs are created. At your current plan you have 12 Sidekiq threads, so to process 3K jobs it will take a while because it can only deal with 12 at a time.

Then, for each reply you receive to that post, 3K jobs are created, so your followers can see that reply without leaving their server or looking at your profile. Then you reply to the reply you got, another 3K jobs are created and so on. 

If you replied to the 100 replies you got on that post in 10 minutes (and assuming my 3K servers math is right). You created 300K jobs in Sidekiq. That’s why you get those queues.

So what does that mean if you’re not into the technical mumbo-jumbo?

It means I was too chatty while being somewhat popular.

Screenshot of my Sidekiq stats, showing 175,082 enqueued tasks as I was replying to folks on my birthday post

What a traffic jam looks like in Mastodon.

So, what’s the solution?

Well, there’s only one thing you can do when you find yourself in such a pickle: scale up your Mastodon instance.9 The problem with that? It starts getting expensive.

Prior to the latest Twitter migration10, I was paying around €280/year (or a little over €20/month) for my Mastodon instance on a custom plan I had with Hugo from the early days. This week, I upped that to a roughly €50/month plan. And that’s still not enough as my birthday post just showed so Hugo, kindly, has suggested he might have to come up with a custom plan for me.

And yet, the problem is not one that will go away. We can only kick the ball down the road, as it were.

(Unless I piss everyone off with this post, that is.)

Thankfully, by running my own instance, the only person I’m burdening with this additional expense is me. But what if I’d been on a public instance run by someone else instead?

Musk you?

If Elon Musk wanted to destroy mastodon.social, the flagship Mastodon instance, all he’d have to do is join it.11

Thank goodness Elon isn’t that smart.

I jest, of course… Eugen would likely ban his account the moment he saw it. But it does illustrate a problem: Elon’s easy to ban. Stephen, not so much. He’s a national treasure for goodness’ sake. One does not simply ban Stephen Fry.

And yet Stephen can similarly (yet unwittingly) cause untold expense to the folks running Mastodon instances just by joining one.12

The solution, for Stephen at least, is simple: he should run his own personal instance.

(Or get someone else to run it for him, like I do.)13

Running his own instance would also give Stephen one additional benefit: he’d automatically get verified.

After all, if you’re talking to, say, @stephen@social.stephenfry.com, you can be sure it’s really him because you know he owns the domain.

Personal instances to the rescue

My speech at the European Parliament on the problem with Big Tech and the different approaches provided by Mastodon, the fediverse, and Small Web.

Wait, I’m confused… didn’t you say that personal instances were part of the problem?

Yes and no: they are and they shouldn’t be.

If ActivityPub (the protocol) and Mastodon (a server that adheres to that protocol) were designed to incentivise decentralisation, having more instances in the network would not be a problem. In fact, it would be the sign of a healthy, decentralised network.

However, ActivityPub and Mastodon are designed the same way Big Tech/Big Web is: to encourage services that host as many “users”14 as they can.

This design is both complex (which makes it difficult and expensive to self-host) and works beautifully for Big Tech (where things are centralised and scale vertically and where the goal is to get/own/control/exploit as many users as possible).

In Big Tech, the initial cost of obtaining such scale is subsidised by vast amounts of venture capital (rich people investing in exploitative and extractive new businesses – which Silicon Valley calls Startups™ – in an effort to get even richer) and it leads to the amassing of the centres15 we know today as the Googles, Facebooks, and Twitters of the world.

However, unlike Big Tech, the stated goal of the fediverse is to decentralise things, not centralise them. Yet how likely is it we can achieve the opposite of Big Tech’s goals while adopting its same fundamental design?

When you adopt the design of a thing, you also inherit the success criteria that led to the evolution of that design. If that success criteria does not align with your own goals, you have a problem on your hands.

What I’m trying to say is:

Do not adopt the success criteria of Big Tech lest you should become Big Tech.

Bigger is not better

Today, we equate the size of mastodon.social (the instance run by Eugen) with how successful Mastodon (the software created by Eugen) is. This is very dangerous. The larger mastodon.social gets, the more it will become like Twitter.

I can almost hear you shout, “But Aral, it’s federated! At least there’s no lock-in to mastodon.social!”

This is true.

You know what else is federated? Email.

Have you ever heard of a little old email instance called Gmail? (Or perhaps the term “embrace, extend, extinguish?”)

Do you know what happens to your email if Google says (rightly or wrongly) that you’re spam? No one sees your email.

You know what happens if mastodon.social blocks your instance? Hundreds of thousands of people (soon, millions?) do not get a choice in whether they see your posts or not.

What happens when your instance of one blocks mastodon.social? Nothing, really.

That’s quite a power imbalance.

Decentralisation begins at decentring yourself

Mastodon is a not-for-profit, and I have no reason to believe that Eugen has anything but the best of intentions.

However, decentralisation begins at decentring yourself.

It’s in the interests of the fediverse that mastodon.social sets a good example by limiting its size voluntarily.

In fact, this should be built right into the software. Mastodon instances should be limited from growing beyond a certain size. Instances that are already too large should have ways of encouraging people to migrate to smaller ones.

As a community we should approach large instances as tumours: how do we break them up so they are no longer a threat to the organism?

If you take this approach to its logical conclusion, you will arrive at the concept of the Small Web; a web where we each own and control our own place (or places).

Small Is Beautiful (Oct, 2022): What is the Small Web and why do we need it?

Tweet, tweet?

I’m not saying that the current fediverse protocols and apps can, will, or even necessarily should evolve into the Small Web.16 In the here and now, the fediverse is an invaluable stopgap that provides a safer haven than the centralised cesspits of Silicon Valley.

How long the stopgap lasts will depend on how successful we are at resisting centralisation. Protocol and server designs that incentivise vertical scale will not necessarily make this easy. However, there are social pressures we can use to counter their effects.

The last thing you want is a handful of mini Zuckerbergs running the fediverse. Or worse, to find yourself having become one of those mini Zuckerbergs.

I love that the fediverse exists. And I have the utmost respect for the gargantuan effort that’s going into it.

And yet, I am also very concerned17 that the design decisions that have been made incentivise centralisation, not decentralisation. I implore us to acknowledge this, to mitigate the risks as best we can, to strive to learn from our mistakes, and to do even better going forward.

So to the ActivityPub and Mastodon folks, I say:

Consider me your canary in the coal mine…

«Chirp! Chirp! Chirp!»

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

  1. After 16 years on Twitter, even I finally deactivated my account and asked for all by data to be deleted last week. It was easy to do as I’ve been on the fediverse for over five years and I’d basically stopped using the hell site almost entirely for over a year. I was just keeping my account active so as not to break over a decade-and-a-half of web links. (Let this be a lesson to you: if you care about not breaking links/content on the web, make sure that you own them instead of Startup Of The Week, Inc.) ↩︎

  2. If you’re wondering what the fediverse is, it’s likely what you call Mastodon. I could write a whole other blog post about why Mastodon is not the fediverse but, thankfully, others have done a great job of explaining the basic concepts already. ↩︎

  3. Greta joined the mastodon.nu instance and already has over 44,000 followers (and is following 50) and Stephen has joined mastodonapp.uk and amassed 27,000 followers in a day or so (and isn‘t following anyone in return at the moment). ↩︎

  4. It’s what I’d call a “personal instance”, an “instance of one”, or “a single-tenant instance.” It’s basically a server where only you have an account. Because Mastodon is federated using the ActivityPub protocol, you can communicate with anyone on any other instance but being on an instance of one is very different to being on an instance of hundreds of thousands, like mastodon.social, for example. ↩︎

  5. On Twitter, my follower count was at roughly 42,000 people before I deactivated my account. Keep in mind that that was after being on the site for 16 years. Similarly, I’ve been on the fediverse for over five years, basically since the beginning, and Stephen Fry has amassed more followers in a single day. This is important to remember as we use my experience to forecast what the instance he joined – mastodonapp.uk – will begin to (or has already begun to) experience. ↩︎

  6. Given the design decisions underpinning the ActivityPub protocol and Mastodon server. ↩︎

  7. When I asked Hugo if I could quote him for this post, he said yes but with the following caveat, which I’m including here so you don’t go off complaining to him (if you have any issues with this post, you can complain to me instead): “You can quote my explanation but as an illustration because there are several aspects that are not 100% accurate and others that I probably don’t even know. For example, if a post/reply includes an image, are two Sidekiq jobs created or only one? If it includes multiple images? If there is a custom emoji that has not federated yet in those posts/replies, etc. But the explanation is probably not very far from the reality in terms of the general functionality of the protocol.” ↩︎

  8. After posting this article from my instance, I watched the Sidekiq queue during the original posting and subsequent replies and it would appear that, currently, the actual number of unique instances my followers are on is ~1,377. ↩︎

  9. Or start blocking followers, or unfollowing people, or staying quiet. None of which are viable options. (Especially the last one… I mean, do you even know me?) ↩︎

  10. Somewhat ironically, I’m apparently responsible for one of the first Twitter migrations (although on a much smaller scale than today’s), back when Mastodon was just starting out. I guess you could say all this is just karma. ↩︎

  11. He’d then get his followers on Twitter to join as many different Mastodon instances as they could and follow him. Finally, with his bootlicker army of incels in place, he’d just have to get chatty with them and watch Eugen’s instance burn to the ground under the weight of it all. ↩︎

  12. On an instance run by donations, for example, this would mean those donations would go to subsidise his account far more than the other accounts on the instance. That is, if they even managed to cover the cost of it to begin with. In Big Tech, like Twitter, the burden placed on the system by celebrities and other exceedingly popular accounts (journalists, etc.) is just a cost of doing business. The cost is subsidised by the corporation because these accounts are the bait that attracts the true assets of the business: you. In Big Tech, you are the livestock being farmed. They just have to keep you distracted enough with enough sparkly things that you don’t realise you’re being farmed.Also, there are economies of scale present in centralised Big Tech systems that simply do not (and should not) exist in decentralised systems. ↩︎

  13. Unlike me, I don’t think Stephen would have to sweat the cost of the server although it will be considerably more than the heavily-subsidised $8/month that Elon would have been charging him. ↩︎

  14. The term “user” is an othering. In Small Tech, we call people “people.” ↩︎

  15. Some would say “tumours”. Hello, I am Some. ↩︎

  16. If anything, the design decisions behind servers like Mastodon show that we need radically different approaches and first-principles design optimised for single-tenant servers and peer-to-peer connections if we want to nurture a Small Web. ↩︎

  17. And I have been from the start. ↩︎