As I have been playing this game for the past week, I have been wondering about the internal mechanics of the server engine. Obviously without the code I can't know what is going on , but I can guess.
1. There are numerous microservices in a distributed environment.
2. While load is light, these work just fine, very efficiently.
3. When load is heavy, the queues get very deep and cause deadlocks between different microservices
4. Users start cancelling requests with other actions
5. The old actions are clogging up the queues, more incomplete actions pile up between the microservices as users get frustrated and repeat actions quickly.
6. Everything grinds to a halt until a bunch of people are forced to lose connections or the servers are reset.
1. Have a max queue depth of no longer than 10 millisecond's worth of data. Deeper queues are NOT better.
2. consider using parallel queues, if one gets stuck for too long, just kill the whole thing and proceed with the unstuck one.
3. Use backpressure to notify fillers of the queue to wait. It is easier to cancel actions if portions of it haven't been submitted to other microservices yet.
4. Consider using tags to quickly remove cancelled requests from heavily used queues.
Anyways, I hope NQ can solve this as I love the game and want to see it thrive.