Jump to content

It's all about the db


Recommended Posts

Anyone else read this as

 

"once the game was said to be no-wipe the grinders emerged. We didn't know there would be so many people mining and digging tunnels 3 to 7 hours a day, our database is is so big its collapsing into a black hole"

 

"So we now want to move to surface mining, bot based mining which doesnt dig tunnels and caves, and spawned asteroids which can be despawned and deleted, to remove the mining burden on the database"

 

Everything is about the database ballooning because they didnt have a clue how much gamers can grind

 

  • Make it easier for newcomers to gather resources from the surface of planets without the need to dig;
  • Then, transition players to deploying mining units once they’ve claimed a territory. Mining units will supply a steady stream of ore, depending on the specifics of the tiles the player has claimed. These mining units come in tiers and should add a sense of progression even to early mining. There will also be a production optimization gameplay if you want to use several mining units.
  • For players who want to specialize in mining, we will introduce asteroid mining. Think of asteroids as epic mining with high reward potential. Asteroids will be spawned in the universe. Players will be able to scan clues in space to discover where they’re located. Some asteroids will contain not just regular ores, but also rare and valuable ones. Once discovered, there will be a delay before their location is broadcast to all players. This is an opportunity for explorers who find these asteroids to reap their resources first or monetize their location. The control of valuable asteroids will also create opportunities for combat, information trading, and collaboration between players. Please also note there will be asteroids in safe zones with lower-value ores.

 

Link to post
Share on other sites

What @Maxim Kammerer said..

 

It seems that NQ really has no grasp on the way their game works or should work and continue to entirely miss the point of why what they expect/envision does not happen or creates problems. The problem her eis not that players do things that are unexpected, the problems is that NQ has not actually thought their game design ideas through in advance AND does not see why what they design does not work, despite the many pieces of feedback they receive on this. 

 

That is my main issue I have with the recent dev blogs, they show a continued structural and chronic issue with NQ which is they have no idea why something happens to then go on to fix the symptom instead of the root cause (because they do not know or investigate what that is and do not pick up on the many comments from players about this). Nothing new here though, this has been the case since pre alpha.

 

From an initial look at the mission system it seems that the return on time spent there is not even close to selling ore to bots so unless NQ removes the latter or seriously nerfs the income for that, I 'm not sure how the missions wil even begin to provide a viable alternative.

 

Link to post
Share on other sites

Unlike some other technologies, database managment is not something we have enough info to even start commenting on.

 

As so, i think this thread is 100% speculation. 

 

But we do have some info about the ammount of data we store. Just look at our cache files. 

That ammount of data is abnormally large. 

 

If we take that number and the network IO we experience when traveling and we "speculate" the ammount of data the database has to serve and store, it is easy to understand that the database already has all those technologies or else it would be crashing under 1000 simultaneous players. 

 

But now for real, dont you guys think cache is too big? Meaning the ammount of data transfered is too big? 

I do. And i think its great that NQ is trying to find ways to reduce it. 

Link to post
Share on other sites

Also there are some easy ways to dismantle some of the unnecessary data, like holes, terrain changes, consequences of mining.

E.g.

- new mining units could do some tile clean up and fill the holes, delete the tunnels

- they can introduce "weather" changes. In RL the hole in the ground will disappear after some time, due to the weather, earthquakes, ... So they can slowly start to simulate the same effect and slowly delete this mess.

Link to post
Share on other sites

Ore doesn't have to be buried so deep, either -- they decided to make it a mole sim, now they don't like the fact that players dig to get ore...? Also they helped create this grind with 0.23 -- perhaps if schematics weren't so expensive people wouldn't need so much mole-time. 

 

How quick it goes from "millions of players" to "well even thousands of players is hard for us"...

 

They use mongoDB and DynamoDB as their DB engines. Mongo shards horizontally. Dynamo is cloud-based and should work at any scale...with enough money. 

 

That's the real issue -- not that the DB will "collapse" but that's its just too damn expensive...because of their own design choices. 

 

As for restoring terrain to save cost, there's no way to differentiate between terrain dug out for base construction and terrain dug out for mining. If they started to fill it, there'd definitely be some buried bases. 

Link to post
Share on other sites
1 hour ago, ShippyLongstalking said:

They use mongoDB and DynamoDB as their DB engines. Mongo shards horizontally. Dynamo is cloud-based and should work at any scale...with enough money. 

 

I dont even know if they are using a relational or a non relational database. 

 

But you cant change from relational to to non relational database without a total project reformat. 

 

It seams wrong to mention dynamo db. 

Makes it look like they have a choice. 

Either they already use it or they will never do. 

Link to post
Share on other sites
7 minutes ago, joaocordeiro said:

 

I dont even know if they are using a relational or a non relational database. 

 

But you cant change from relational to to non relational database without a total project reformat. 

 

Eh? I just explained what DB engines they use.

 

Mongo is document-oriented, not a relational DB. Switching off mongo wouldn't change anything, they'd still have scale issues. Relational isn't better than non-relational, they have different advantages and disadvantages. 

 

12 minutes ago, joaocordeiro said:

It seams wrong to mention dynamo db. 

Makes it look like they have a choice. 

Either they already use it or they will never do. 

 

I'm not sure what you mean by being "wrong" to mention DynamoDB -- yes this is something they use today. They use both Mongo and Dynamo.

 

I know what DB engines they use because in the past I've checked their jobs site and they make the details of their stack pretty clear. 

Link to post
Share on other sites
5 minutes ago, ShippyLongstalking said:

Eh? I just explained what DB engines they use.

Sorry m8, long day here.

Then they did a great design choice there.

They can always scale with cost.

But at some point, the amount of data transfer between the client and the servers will be unacceptable.
PPl can't be expected to have 1gbit fiber with 5ms latency to play the game.

At some point, they will have to figure a way to reduce the data required to play the game.

I for once think the hole we make should slowly refill with time, reverting to "unaltered terrain" status. Excluding ofc the terrain change near bases.

Link to post
Share on other sites

Frankly, no I didn't read it that way. I read it as the current start-up grind is pretty bad (and it is) if you want to do any meaningful manufacturing and build a base and some ships without relying on others to provide things for you - which would still be a grind as you then have to buy everything and go on endless trips back and forward after mining to pay for it all.

 

Basically - digging for ore is amusing and a little challenge to do efficiently for a while - then there should be an alternative once you get to a certain point.

 

Of course one could just have them all on the surface just not so obnoxiously obvious and have skills that governed how much you get. Depends on whether or not we need this to be Minecraft with space ships.

Link to post
Share on other sites

So now the last thing to do in the game, mining, is getting removed?

If you are surprised at how quickly players can grind millions of tunnels into your planets, you will be utterly shocked at how fast auto miners will deplete the entire solar system of resources. Then in most cases delete the resources selling to bots.

And clues to find asteroids, really? This is not Tomb Raider or Puzzle Quest....This is not the kind of gameplay the majority of DU's demographic is interested in. Just spawn in a couple of giant asteroid fields with hundreds of thousands of asteroids to mine and be done with it. Will promote exploration, politics and combat like no other content ever could.

Link to post
Share on other sites

Completely mineable planets is going to be a big DB draw, no question about that. 

 

Of course I assume they don’t give me the entire layout of a hex I am just flying or walking over down to the core. I just need it down to where I can expect to make it soon, the range of my sensors as soon as I turn one on I suppose. But even the sensors don’t need every hole, just the materials they can pick up.

 

More surface gathering would mean less holes, but who does that once they have left Sanctuary. Only time I ever do it now is when I need enough scrap to repair a container on a crashed ship.

 

I find it more enjoyable than deep mining, but there is just no coin in it. Maybe if they made talents and higher tier ores they could get more of us wanting to keep our territory nice and original so we could wizz around it collecting ores.

Link to post
Share on other sites
6 hours ago, ShippyLongstalking said:

As for restoring terrain to save cost, there's no way to differentiate between terrain dug out for base construction and terrain dug out for mining. If they started to fill it, there'd definitely be some buried bases. 

It is actually quite easy to detect terrain dug outs for base construction. Just don't touch caves with constructs inside (maybe with a reasonable safety margin around the building grid) and underground facilities would ba safe.

Link to post
Share on other sites
9 hours ago, joaocordeiro said:

Unlike some other technologies, database managment is not something we have enough info to even start commenting on.

As so, i think this thread is 100% speculation. 

 

Much of what is discussed here is pure peculation, nothing wrong with that IMO. 

 

As I read the devblogs, I took it the overall Database design is not so much the issue, it's more the cost of running the database due to the amount of I/O needed. AWS cost is modelled on that traffic more than anything else so NQ pays through the nose because of that. While I'm guessing here I do believe that to be fairly close to what is happening. I would also not be surprised if the database servers are actually not in the same location

 

Many here have suspected NQ has been throttling I/O traffic in order to cut cost and have made drastic changes to the way the game interacts with the database to keep things alive. We know the latter is true, I assume the former is the reason for this. It is what the big "database update" that took many hours was about last year

 

The choice for rented servers was a short term choice, if you look over many years, owning hardware yourself will pay itself back and then some. Hiring rack space may be a much more efficient use of funding over time. But knowing pretty much exactly what CCP uses to run EVE, the hardware in the DU cluster will be as expensive if not more so.

Link to post
Share on other sites
38 minutes ago, blazemonger said:

The choice for rented servers was a short term choice, if you look over many years, owning hardware yourself will pay itself back and then some


That choice is a lot more complicated than that.

Owning your own hardware has lots of downsides:

 

Hardware costs.

They would have to pay, in one time or a rent, for all the hardware. Not just the servers, but also routers, backups, databases. And also pay for datacenter rack space.

But they would have to pay for a prediction of what they think they would grow. Not for the real usage.

Enterprise class servers take 2 weeks to be ordered. And rack space requires prediction. So it would take weeks to months for NQ to get more infrastructure ruining if they needed it.
Then if they lose players/subscriptions they could not easily return all that extra hardware no longer needed. They would have to pay for the rack space and any renting even if they were not using the servers.

 

Human costs.

They would need a team of 2 or 3, very highly skilled and highly paid infrastructure architect/manager.


Support.

They would need to pay allot to have support for the technologies they use. If they had a problem with the database, they would need to pay their support. If they have a problem with their router, pay the support. If they have a problem with their backup solution, pay the support..... 

Licensing support.
Currently, professional software will have much lower licensing costs of their software is acquired via cloud provider and deployed into a VM. mostly because they don't need a sales team, or a mechanism to check the licensing and even the support team is quite reduced, with most tickets being solved by the cloud provider and only the hard ones being forward to them.
Also, again, with the unknown growth, NQ would have to buy extra licensing to predict the future needs, And also be stuck with what to with the rest of that year's long license for the server they no longer need....

Security.
If NQ was to have their own infrastructure they would be 100% responsible for all their infrastructure security.
Cloud providers will automatically, and without extra cost deal with most cyberattacks and most DDOS.
Again NQ would have to hire 1 or 2 full time, very good, very well paid cybersecurity experts. And even with those, their chances of success would be low.


What I would say is, if your project will mainly need EC2 instances, yes, AWS is too damn expensive for raw computing.

AWS shines if ppl use other services like docket containers, CloudFront, lambda, autoscaling groups.
You would probably do batter with some virtual infrastructure from OVH or similar VPS provider.
But owning your own hardware, putting it into a colocation regime in a datacenter, is 100% out of the question for a startup like NQ.

Link to post
Share on other sites

 

Actually, many of the things you describe would be included in the rack space rental agreement they would use for this. And that owning hardware has higher upfront cost is obviously true but those cost would be one time and over the long term (which was the point I made) are more cost efficient.

 

It's very much the same as buying or renting a house. the choice between the two is not just a matter of whether you can afford buying. Also the cost for server infrastructure, being a physical asset, has a real world value that can be paid for by ways of lease/loans.

 

I get your point but it's not that different from mine. You seem to put more weight on the related cost which in most cases are included in the different way of getting servers in a DC and running.

Link to post
Share on other sites
7 hours ago, joaocordeiro said:

Owning your own hardware has lots of downsides:

[...]

External servers have the same downsides and the customer pays for it. That makes sense in short-term (because it saves the initial investment) but not in the long-term (because the owners of the harware want to make profit). In the long-term the game should be based on in-house servers and external servers should be rented for exceptional high traffic only.

Link to post
Share on other sites

while developing maybe the right approach, but they should consider migrating once they are up and running and know what they are in need of. Use cloud services while you are setting up and developing and don't really know what you require, which from their post on DB usage seems true. But once you have a better picture look at moving more inhouse

Link to post
Share on other sites
15 hours ago, joaocordeiro said:

I for once think the hole we make should slowly refill with time, reverting to "unaltered terrain" status. Excluding ofc the terrain change near bases.

And again..... This is a problem that was pointed out years ago in the pre-alpha tests. The fact that having a consistent world in a MMO, where digging holes is the main (and arguably only) source of income does not scale well, should not be a surprise to anyone.

Storing world scale geometry in a voxel DB may seem very doable at first glance, since unaltered space takes zero storage and server-client I/O. But when you then add a MMO game on top, that incentivizes all players to make world changes everywhere (digging) it is bound to collapse.

Link to post
Share on other sites
9 hours ago, CptLoRes said:

And again..... This is a problem that was pointed out years ago in the pre-alpha tests. The fact that having a consistent world in a MMO, where digging holes is the main (and arguably only) source of income does not scale well, should not be a surprise to anyone.

Storing world scale geometry in a voxel DB may seem very doable at first glance, since unaltered space takes zero storage and server-client I/O. But when you then add a MMO game on top, that incentivizes all players to make world changes everywhere (digging) it is bound to collapse.


Agree, how they didn't see this as a very easy fix i won't know, instead of getting players to dig pointless, and horrific looking holes all over the place to mine why didn't they do away with that and make mining work in a very different way, perhaps have some kind of tool that you would put down on a hex, scan, get a concentration reading then play some kind of fun little mini game, instead of what we have which at times makes me a bit queasy not going to lie.

 

Link to post
Share on other sites

I actually found the mining experience quite fun. I liked the feeling of being deep underground hunting for riches. Especially when finding a mega node, the cavern experience was epic in the sense of scale. I never cared about building wealth just for the sake of it, so for me mining was a job to buy what I needed and not too tedious. Once set up there are other forms of income possible to a creative player after the early game grind.

What did strike me as odd though was the many, many small nodes scattered around the hex. My own idea if I had been making this game would be vast deposits, much larger than the megas we had. Mines that would be active for months, infrastructure built into them, hangars with haulers ready to go, defences in place to secure the asset, mining settlements and so forth. A mine should be a semi-permanent feature on the map that is utilised over a very long term and ownership should be able to be contested, mining rights sold, etc.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...