First let me explain what I observed
While mining, the dreadful errors of came up once more and actions were rolled back.
So far so good, while annoying this is to be expected for certain server problems.
Where it gets interesting what I observed several times afterwards.
Like seeing some actions succeed, give ore and then further actions rollback the terrain modifications.
Sometimes several at once and this still is somewhat explainable by regular issues, be it extra buggy.
But now it gets interesting, I left the spot or ore alone a little bit and mined on the other side of a cliff (for as far as that succeeded).
What happened when I switched again is that terrain that was long gone came back on my first action and then disappear again as I interacted with the terrain.
And not just one action I did, but several and not all at once!
Some educated guess at what is wrong
This all looks like there are inconsistencies in the server state, different servers or server sub-systems see a different reality and are not properly synchronized. The same I have observed with mining with another player. Terrain modifications made by one were not visible to the other. While at another time and with another player we were seeing each-others actions just fine.
This leads me to the conclusion that the method used to accept a huge volume of player actions, likely a form of partitioning/load-balancing to divide the workload in combination with caching sub-systems are the root cause of most of the experienced issues. Work is assigned to incorrectly up-to-date systems that subsequently refuse the action to take place. Because if a player does an action based on what the client shows and the server executing the action sees another reality, this will trigger code that invalidates said action. It has to as to protect the integrity of the server and counter manipulation attempts. Yet in this case, it is the server that is validating against an incorrect internal state. This result in retries by players causing even more load and more "random" phenomena. Some of these actions might end up served by a correct sever and succeed while other attempts fail, all worsening the inconsistencies over time.
So the primary cause does not seem to be a network issue, database issue or the general amount of players hitting the server.
Obviously each of those can have/be an issue as well and those won't help (except to muddy the waters).
But the behavior I described here is not explainable by just network or database issues or a mixture of the two, it clearly has to do with how work is divided/assigned and/or caching.
