I think you probably underestimate how far one can get with “vertical” scaling. Here’s the dockerfile: https://raw.githubusercontent.com/LemmyNet/lemmy/release/v0.17/docker/prod/docker-compose.yml

• It includes 4 different containers… so there’s a way to scale out to 4 machines right away. Maybe not every container is doing an equal amount of work… but there’s some amount of immediately available machine-splitting.
• I’m no expert, but I believe that at least the lemmy and lemmy-ui containers are stateless. If so, they’re horizontally scalable already.
• Postgres then would likely be the main bottleneck. But postgres offers read-replicas, so again the write-load and the read-load can be hosted on separate machines. And if there’s enough read-load, you can have many replicas.

Other comments from the admins have shown that lemmy.ml today is running on a single eight-core box and it’s currently hosting 30k registered users and over 1k active. So how much more compute capacity can we throw at “vertical” scaling on the current software architecture?

• Just by going to a bigger single box, we can get 128 cores with no problem, a 16x bump in capacity. Does that get us to at least to 300k registered + 10k active?
• Splitting the containers onto 4 separate machines. Does that get us 2x more?
• Adding PG read-replicas and additional lemmy/lemm-ui containers would allow us to expand our instance footprint to maybe 6 physical machines should get us another 2x or more in performance.

Conservatively, that’s 100x the computing capacity of the current hardware and could potentially support 1m registered users and 50k active. Now, I don’t REALLY expect this to be possible today, there will be many software bottlenecks found along the way to scaling a single instance this large. But my point is that there’s already a medium amount of horizontal scalability built into lemmy, and if the software doesn’t fall over for algorithmic reasons (which is will at first), the current infrastructure architecture allows quite a lot of growth. There’s plenty of time between now and a federation of million user instances to adopt a truly distributed storage backend if needed.

Yeah, it would be really nice to be able to look at the “popular” communities by sorting by most-subscribed, most posts, and most comments.

I’m familiar with that, does it have the sorting options that OP and I are discussing? It looks to me like the sort order is both hard-coded and confusing, although less-popular communities do seem to be at the bottom… I can’t identify any specific order that entries are listed in.

Doesn’t solve the availability issues, though. I know of no seriously hosted system that doesn’t have at least two replicas in different availability zones.

I’m not sure why you think the setup I’ve described can’t have coverage in multiple availability zones. If the lemmy and lemmy-ui containers are stateless as I suspect, you can autoscale them. Pictrs is new to me, not sure there… but it appears to support object-storage which would likely make it stateless and the object-storage can replicate to multiple-az’s. Postgres read-replicas can be placed in multiple az’s as well. The only component that presents an issue is the Postgres write-leader, and failovers there can be done in minutes. Many many popular sites run with an infrastructure like this and achieve excellent uptimes.

I do get the power of horizontal scalability, I specialize in distributed databases. But they come at a cost in flexibility relative to something like Postgres… and we’re very far from “needing” horizontally scaling database writes here. Everything else looks like it can be scaled horizontally if someone wants to take on the headache of doing so.

I think the communities are sorted by the number of subscribers.

It’s definitely not a strict sort by subscriber count. The top-5 communities in order from 1 to 5 have: 2491, 4461, 1414, 1138, 7109. It is sort of roughly ordered by some measure of popularity, but not one I can follow, the second entry is more popular than the first in every metric shown.

Don’t get me wrong, it’s a super useful service and one I use and recommend. But I was wishing for more sortability when I was setting up my subscriptions the other day, and if the number of communities grows that will be even more useful.

If cockroach is truly PG compatible, lemmy admins can swap it in without developer support. I suspect Cockroach constrains some SQL features and has poor performance on others, but that or AWS Aurora are things you can experiment with without dev support if you’re passionate about the proving out the value of scale-out.

The statement that spawned my response though was this:

I think lemmy will be bitten in the ass by not having considered clustering/horizontal scaling from the start. Federation alone as a scaling mechanism is only feasible for “nerds”. But if the network wants to grow, we will need a few scale-able large hosted instances.

I still don’t think it’s true that we need horizontal scaling to support sufficiently large instances. The amount of vertical and horizontal scaling ability built into Lemmy today is both useful, and likely to outstrip the current ability of its code to scale a single instance. Any algorithms that scale super-linearly with respect to comment-count, post-count, user-count, or community-count, will fail just as hard with distributed backends as they do with an RDBMS. And as you note, PG-compatible distributed systems provide a potential lower-engineering-cost on-ramp to distributed systems once the codebase is efficient-enough to warrant such a transition to scale further. I suspect I’ve contributed everything of use I have to this thread though, and don’t expect to respond further.

There are other reports of this behavior as well: https://lemmy.ml/post/1159652

This may be obvious and not what you want to do, but Lemmy’s own UI is optimized for threaded conversations within a post. Browsing Lemmy communities from an account on a Lemmy instance looks pretty sensible.

That doesn’t help you manage comment toots from Mastodon, but if you don’t find another solution… it will look better from Lemmy itself.

You can’t use your lemmy.world login to login to pathfinder.social, but you don’t have to. You can “subscribe” to communities hosted on pathfinder social, and federation will copy the the posts/comments to lemmy.world for you to read and vote on. Your own comments will get copied in the other direction as well for others to see.

You and I randomly happen to have accounts on the same lemmy instance, and I’ve already subbed to a bunch of the p2fe communities, check out this set of community search results and just click subscribe on whatever interests you.

To find other fun communities to subscribe to, check out the list of communities other lemmy.world users have already subbed. There’s ones for rpg, fate, shadowrun as well. Then if you’re still looking for new communities to sub, check out https://lemmy.blahaj.zone/post/61827 which explains the unfortunately confusing process of discovering new remote communities and teaching lemmy.world about them. It’s a bit weird and involves searching in multiple places and then doing a bit of a dance to set it all up, but it’s not THAT bad once you get the hang of it. Just confusing if no one teaches you the steps.

Good luck, have fun.

You should post this as an end-user workaround in https://github.com/LemmyNet/lemmy/issues/506. LOTS of people are interested in this, and they commonly get redirected to that ticket to watch progress. If there was a workaround that didn’t depend on the devs available, lots of semi-technical folks could help themselves before the feature is done. It could maybe even help someone understand how to get started on the ticket to see what API calls are involved.