This site is currently struggling to handle the amount of new users. I have already upgraded the server, but it will go down regardless if half of Reddit tries to join.
However Lemmy is federated software, meaning you can interact seamlessly with communities on other instances like beehaw.org or lemmy.one. The documentation explains in more detail how this works. Use the instance list to find one where you can register. Then use the Community Browser to find interesting communities. Paste the community url into the search field to follow it.
You can help other Reddit refugees by inviting them to the same Lemmy instance where you joined. This way we can spread the load across many different servers. And users with similar interests will end up together on the same instances. Others on the same instance can also automatically see posts from all the communities that you follow.
Edit: If you moderate a large subreddit, do not link your users directly to lemmy.ml in your announcements. That way the server will only go down sooner.
I think you probably underestimate how far one can get with “vertical” scaling. Here’s the dockerfile: https://raw.githubusercontent.com/LemmyNet/lemmy/release/v0.17/docker/prod/docker-compose.yml
- It includes 4 different containers… so there’s a way to scale out to 4 machines right away. Maybe not every container is doing an equal amount of work… but there’s some amount of immediately available machine-splitting.
- I’m no expert, but I believe that at least the lemmy and lemmy-ui containers are stateless. If so, they’re horizontally scalable already.
- Postgres then would likely be the main bottleneck. But postgres offers read-replicas, so again the write-load and the read-load can be hosted on separate machines. And if there’s enough read-load, you can have many replicas.
Other comments from the admins have shown that lemmy.ml today is running on a single eight-core box and it’s currently hosting 30k registered users and over 1k active. So how much more compute capacity can we throw at “vertical” scaling on the current software architecture?
- Just by going to a bigger single box, we can get 128 cores with no problem, a 16x bump in capacity. Does that get us to at least to 300k registered + 10k active?
- Splitting the containers onto 4 separate machines. Does that get us 2x more?
- Adding PG read-replicas and additional lemmy/lemm-ui containers would allow us to expand our instance footprint to maybe 6 physical machines should get us another 2x or more in performance.
Conservatively, that’s 100x the computing capacity of the current hardware and could potentially support 1m registered users and 50k active. Now, I don’t REALLY expect this to be possible today, there will be many software bottlenecks found along the way to scaling a single instance this large. But my point is that there’s already a medium amount of horizontal scalability built into lemmy, and if the software doesn’t fall over for algorithmic reasons (which is will at first), the current infrastructure architecture allows quite a lot of growth. There’s plenty of time between now and a federation of million user instances to adopt a truly distributed storage backend if needed.
Doesn’t solve the availability issues, though. I know of no seriously hosted system that doesn’t have at least two replicas in different availability zones. I don’t expect any hobby instance to offer any kind of availability guarantee. But if we want to have one or two central instances that the typical reddit user can flock to, this would IMO be essential to have.
Also, in my experience it is FAR cheaper to have a few low to mid range systems for vertical scaling, than to throw a high end machine at it for vertical scaling. If you look the the pricing, the monthly costs for vertical scaling goes up exponentially once you want much more RAM and CPU cores (and storage, and so on).
Being able to scale horizontally solves both issues: hardware is cheaper and reliability is higher.
That lemmy is so damn efficient would then simply mean, that we can achieve excessively good results with low resources, where Reddit would already struggly and needs to put much more machines in place. That would be a nice “business” advantage.
Doesn’t solve the availability issues, though. I know of no seriously hosted system that doesn’t have at least two replicas in different availability zones.
I’m not sure why you think the setup I’ve described can’t have coverage in multiple availability zones. If the lemmy and lemmy-ui containers are stateless as I suspect, you can autoscale them. Pictrs is new to me, not sure there… but it appears to support object-storage which would likely make it stateless and the object-storage can replicate to multiple-az’s. Postgres read-replicas can be placed in multiple az’s as well. The only component that presents an issue is the Postgres write-leader, and failovers there can be done in minutes. Many many popular sites run with an infrastructure like this and achieve excellent uptimes.
I do get the power of horizontal scalability, I specialize in distributed databases. But they come at a cost in flexibility relative to something like Postgres… and we’re very far from “needing” horizontally scaling database writes here. Everything else looks like it can be scaled horizontally if someone wants to take on the headache of doing so.
Well, one could try to swap postgres for cockroachdb. But a ticket in github that asked for clustering support was closed with being out of scope. So might be lemmy is not stateless. Haven’t checked the code yet, though.