Jump to content

I really want to believe in this but ... really, this doesn't add up!


war

Recommended Posts

 So how I see it ... 

This is either being massively over sold or there are some "gotchas" that are still being overcome (and I seriously want them to be) but these guys must be on to something that through years of digging around and researching this particular technology (Voxels) I have not found anyone that can do this or anyone who even remotely has an idea how to do this on this scale. 

 

That on it's own raises questions does it not? 

 

I really don't want another no mans sky ... but I do want to know that this stuff really is possible so a little insight in to the tech would be really nice to have if the devs over at NQ are in a sharing mood (just a little technical peek ... please!).

Honestly, I don't think your questions or skepticism can be answered here. Any reply will just lead to more questions and "but what if"s. The media released by NQ gives a good indication of what they have now. How it works beyond that is unknown.

 

Are there concessions? Yes. Constructs will not be able to break in half: the two halves will just float together. There are more covered in various locations.

Link to comment
Share on other sites

They bake their voxel constructs as a single mesh when built to reduce the load...

 

Your numbers are not taking into account occlusion... that's hiding what you can't see, like, the inside of a planet, or a ship, or a moon...

 

Also, you are seeming to not be familiar with LOD... that's the detail of an object when it's far away being lower, which is what they use on constructs far away... which are baked already.

 

Also, I don't know what Voxelfarm engine you speak of, Landmark uses ForgeLight . 

 

 

Planet side and H1Z1 use forge light but landmark uses voxel farm. They are trying to merge the two at daybreak but its hard. I also have a copy of voxel farm but it's not the same version. Generally twerks right and TehWardy is out of date.

DU and Voxel farm are close but not the same and both are 2 generations beyond the limits set by minecraft almost a decade ago. The limits that TehWardy is talking about are also broken easily by some minecraft modding tools and the newer optimisations. . IE minecraft bakes the top surface of a chunk in some mods. 

World gen is three things. The seed, the biome library, and the chunk system. 

All major voxel worlds use these, and all bake a set of mesh's at the surface. One serves as collision, one occlusion, there are lighting and in some games, placement meshes, pathing meshes, hazard maps,etc. What you see is created by a lot of things that your computer sees but never draws for you.   

 

The next stages of voxel evolution will involve:

  1. Localised seeds. I.E. a different seed for part of a planet to the other. Send 3 to 5 continental 'plate' seeds per planet. Giving more diversity than a single seed planet can do. 
  2. Reversing the seed. Taking the player builds in and area and reversing the seed algorithms or searching seed algorithms and seeds for a close match too them saving the seed and any voxels that differ from that reversed seed. Both humans and computers are only pseudo-random. 
  3. Hashing a build. Getting a hash does not save the detail of the build, it's a lossy compression, but it may allow some long term saving off line or client side of a build detail with only the hash stored centrally on a server or in a blockchain to prevent tampering and ascertain value in the case of compensating a build error. Comparing two hashes verifies that the data is the same or has changed since two hashes of identical data are always identical. 

A 100 kilometer world is not a big file.

Vladimir Romanyuk's Space Engine stores the entire known universe with to scale planets, Galaxies, asteroids, comets, nebula using a few public databases and some nice world generation code. It is not a game yet that's planned. However you can fly around the universe in ships already.  

Link to comment
Share on other sites

Wow ... some great feedback guys.

Did someone say Unigine! ... I love that engine, their Heaven demo makes me want to build an MMO in that world!

 

I'm not looking to rock the boat like maybe ... BlueDrake42 or whatever his name is ... but I do want to understand what it is that NQ is saying they have here.

 

For example ...

 

Procedural generation 

I know that Voxelfarm worlds are procedural and look amazing but my experience in a shard based world was poor when using a real world MMO solution built on the engine.

The alternative is something like no mans sky that did not meet expectations at all and both didn't save the players chances (no mans sky didn't even allow any changes).

 

Voxel Technology

Oddly enough I have been talking about doing exactly this for a couple of years now but couldn't find enough support from a seasoned developer community to get help with this idea of using Voxels on this scale because I had developer after developer tell me "it's impossible on todays tech without breaking the laws of physics"

...

 

That said ...

 

I am believer in the technology and do feel it can scale, using procedural in combination with static data and compression / bit packing techniques for the transmission brings the scale in terms of raw bytes in to the possible but it does come with a few "negotiations" you have to make. 

 

As I said when I started this thread, the raw numbers of Voxels that represent a large object are just insane and with something like a space station or a large ship you simply can't start from a procedural base so every Voxel has to be considered a change.

 

Given that "fact of how Voxels work" you can then take a data source containing any number of voxels and generate a mesh from that data easily enough and using some pretty common programming techniques known by many today construct meshes at varying "qualities" for use in game.

 

So in short ...

 

My question really wasn't "Using this number of Voxels is impossible right?" it was more something along the lines of ...

 

Given that I as a keen user of the game spend say 2 hours per day for say 10 days in a row editing Voxels I could as a single player build a pretty hefty space station looking at the tooling available here it's reasonable that I could build one of those space stations.

Looking at the raw numbers I presented before, lets assume my space station creation was something like 10 million Voxels, that's 10 million individual Voxels that 1 player has created that the server has to process and store and hand out whenever someone looks the right direction from the right spot on the game.

 

I am however, only 1 player ... so lets scale that up ... add another 999 players and we have 1000 x 10 million Voxels that are all entirely non procedural and could be reasonably constructed on the game server.

even if those ridiculous number of Voxels are not right next to each other that's still a lot of data to be managing and processing.

 

For an MMO that's a tiny number of players and for a single shard that's an insane amount of mesh data being handed around, the previously mentioned "developer community" tell me this is not possible.

 

So how I see it ... 

This is either being massively over sold or there are some "gotchas" that are still being overcome (and I seriously want them to be) but these guys must be on to something that through years of digging around and researching this particular technology (Voxels) I have not found anyone that can do this or anyone who even remotely has an idea how to do this on this scale. 

 

That on it's own raises questions does it not? 

 

I really don't want another no mans sky ... but I do want to know that this stuff really is possible so a little insight in to the tech would be really nice to have if the devs over at NQ are in a sharing mood (just a little technical peek ... please!).

That's not how you save a voxel construct. The simplest way is to save the voxel type and number. log the position of the constructs and the important voxels.  Mesh over the majority non air voxels to get the surface mesh, Whats visible. That's the first thing you see from a distance. Lod reduces that further. You do not need to record the location of all air voxels unless the air voxels are a small minority. Likewise if the construct is mostly rock you don't need to log the location of every voxel of rock just everything else that is not air and rock. The size of the save will depend on the diversity of the voxel type and complexity available. 

 

Another compression trick is to take a cross section though the ship. Compare that to several other adjacent cross sections; If they are all the same you can log that as a repeating plane and only record the number of repetitions. Then log any exceptions to the rule. Also if the ship is built with a symmetry setting on you only need to record and save one side or quarter and the number of planes and the exceptional asymmetrical objects. 

 

Thirdly there is a key compression possible for plumbing and electrical's. Pipes only connect to pipes, power to power so the first level of save only needs to log it as a power/ pipe system without saying which. The engine can work out the type from the connections. 

Link to comment
Share on other sites

Yes, a naive implementation would be impossible for a planet of 100km, assuming a generous 1 byte per meter cubed you would need to enclose an area of 200,000 meters on a side giving you about  200,000 ^ 3 bytes or about 8 petabytes not including all the overheads involving chunking and hermite data. You could never store that.

 

What they are doing as far as I understand, is a combination of procedural generation and oct-tree compression.

 

Chunks of planets/asteroid are are generated proceduraly and then only modifications are stored, so that when a player removes a bit of the terrain only the fact that a sphere at x,y,z was removed, the voxel is only "materialized" and stored on the server as "voxel data proper" when the size of the log of changes is greater than storing the voxel data if that makes sense.

 

As for large constructs, this is where oct-tree based compression techniques come into play, you don't need to store a 3 dimensional array for every when lots of areas will have the same material or dead space, you store the data in a tree structure where each node has 8 descendant children. You only need leaf nodes when you have individual changes at the lowest resolution. This is great as it gives you LOD by only sending the top levels of the tree when the construct is far away.

A good way of visualizing this is by looking at the equivalent quad-tree in 2D space, each box is a leaf in the tree, areas where it is all the same material type.

 

https://en.wikipedia.org/wiki/Quadtree

 

image002.gif

 

quadtree.gif

 





 

Link to comment
Share on other sites

and what's the disaster with Star Citizen? Its still in development, taking longer then some would like but steady progress is being made. I don't really like their modular method, would prefer them to make the full game then add to it not peace meal like they are now but as far as I know there no disaster or anything around the project.

It's now entering its sixth year of development and what little has been delivered to backers is a buggy mess. The technology to make the game as of the current vision is either impossible or simply won't exist for a very long time. They're using a modified version of Cryengine 3 (why you would use a high fidelity graphics FPS engine for an MMO is beyond my knowledge) and the current modules can't even handle limited multiplayer as of yet, let alone proper MMO numbers. Squadron 42 sounds like it's a "failed Hollywood developer makes a video game" kind of thing ala The Order 1886. They've collected literally hundreds of times the originally asked for budget, yet are still raising money (via the selling of images of ships that don't yet exist in the game, no less) for a game that's now two whole years late. They've made dodgy changes to the ToS. Feature creep has been impacting the game since 2012. They are rumoured to be spending 3 million per month (aka six times more than they originally asked for, which would have lasted two years) through their four different studios in LA, Texas, Manchester and Frankfurt. Why does an indie game need so many studios?

 

Basically there are a ton of problems that would have gotten the game canned, or they would have been reined in heavily, had they a proper team of producers and a publishing company above them.

Link to comment
Share on other sites

It's now entering its sixth year of development and what little has been delivered to backers is a buggy mess. The technology to make the game as of the current vision is either impossible or simply won't exist for a very long time. They're using a modified version of Cryengine 3 (why you would use a high fidelity graphics FPS engine for an MMO is beyond my knowledge) and the current modules can't even handle limited multiplayer as of yet, let alone proper MMO numbers. Squadron 42 sounds like it's a "failed Hollywood developer makes a video game" kind of thing ala The Order 1886. They've collected literally hundreds of times the originally asked for budget, yet are still raising money (via the selling of images of ships that don't yet exist in the game, no less) for a game that's now two whole years late. They've made dodgy changes to the ToS. Feature creep has been impacting the game since 2012. They are rumoured to be spending 3 million per month (aka six times more than they originally asked for, which would have lasted two years) through their four different studios in LA, Texas, Manchester and Frankfurt. Why does an indie game need so many studios?

 

Basically there are a ton of problems that would have gotten the game canned, or they would have been reined in heavily, had they a proper team of producers and a publishing company above them.

Yeah wasn't really defending them just haven't follow very closely at all. I bought a ship like when the KS first happened but thats it. Based on what you said glad i didn't spend any more than that.

 

Think thats why I like Dual above most any other game being made right now. They starting simple, making a foundation to build upon as the game grows.

Link to comment
Share on other sites

For those not signed up to the newsletter, you can read the latest release here. Check out the animated gif teaser.

 

It looks like they will be sending out one of these every month. My own take is that this is an acceptable frequency (just about) but only because I recognize that the project is in its infancy and there probably isn't that much to report on right now. At some point, though, I'd still think this should be increased to a weekly or bi-monthly release.

Link to comment
Share on other sites

  • 10 months later...

You guys seem to think I know nothing about Voxels / rendering.
The fact is even with things like occlusion you still have to compute what's been occluded and what hasn't, and with Voxels you normally render chunks, so if any piece of a chunk is visible then you need to render the whole chunk at a given LOD you can't "quickly generate a single mesh at various LOD's" you generate the whole mesh at a given LOD.

My understanding is that to make the data editable something has to store "the edits", from my own tinkering in building voxel based worlds I've figured out that on day 1 for an entirely generated volume (like the planets would be) you'd effectively have a storage requirement of 0 bytes as you only need to store the changes made that deviate from the algorithm / generator.

With something like a space station though, there's never any part of that that's procedural, you effectively always have to store every Voxel.
You could store only the mesh data but then you need a means to "figure out" the volume data when the mesh gets edited, which is a considerably more complex task than generating a mesh from Voxels in the first place.
The other way is to store "actions" and use things like run length encoding to reduce the storage load ... but that's still a lot of data :) 

...

So here's where i'm at ...
As you guys should be able to see from my Avatar ... I'm IN ... i'm taking a gamble ... it's (i think) a risky one but if these guys deliver even a fraction of what i've seen so far I still like I'd get value for money.
So ... @novaquark ... I donated up to Ruby, the money was going to in to my own game project but i've decided to start a new game dev project ... developing LUA driven DU stuff :) 
I can't wait!

Link to comment
Share on other sites

In about 3 weeks' time, we'll see how it all hangs together when around 2000 pre-alpha players all try to login simultaneously ! :D

 

I'd guess well have at least 1000 concurrent, with most trying to build various things in or near the arkship zone. It will be an excellent stress test.

Link to comment
Share on other sites

The scary part... er, one of the scary parts, is that they said it wasn't ready for the planned alpha release date. Unfortunately, this puts the game into the same crowded pool of other alpha/pre-alpha games that missed deadlines and/or released too early. It's a non confidence inspiring trend that has become the norm these days.

 

The upside is that they were honest about their progress and are going to put it out there anyway in it's current state. It's a bold move.

 

I'm excited to see their progress first hand, but I'm trying my best to temper my expectations. I fully expect a broken, unfinished, PRE-Alpha proof of concept that if nothing else, should give us a baseline to measure progress from here on out.

 

Point being, not everything has to work, just enough to make it believable.

Link to comment
Share on other sites

10 minutes ago, Captain Jack said:

The scary part... er, one of the scary parts, is that they said it wasn't ready for the planned alpha release date. Unfortunately, this puts the game into the same crowded pool of other alpha/pre-alpha games that missed deadlines and/or released too early. It's a non confidence inspiring trend that has become the norm these days.

 

The upside is that they were honest about their progress and are going to put it out there anyway in it's current state. It's a bold move.

 

I'm excited to see their progress first hand, but I'm trying my best to temper my expectations. I fully expect a broken, unfinished, PRE-Alpha proof of concept that if nothing else, should give us a baseline to measure progress from here on out.

 

Point being, not everything has to work, just enough to make it believable.

I may be totally wrong, but I think the decision to proceed with the very limited pre-alpha will give NQ the chance to nail down many of the technical challenges that they are not confident about at this point. They are clearly not happy with the current state of development progress, hence the further delay of "alpha proper".

 

Having 2K players on the server doing random things will give a far better approximation of what the system can handle than running a few thousand scripted bots around in a field. Real players bring unpredictability to the system that cannot be fully simulated by bots.

 

Once the limits of the architecture are established, it defines the scope of any features that will be added later, because it gives a much clearer picture of what's possible and/or what's simply not practical to attempt.

Link to comment
Share on other sites

17 minutes ago, Captain Jack said:

The scary part... er, one of the scary parts, is that they said it wasn't ready for the planned alpha release date. Unfortunately, this puts the game into the same crowded pool of other alpha/pre-alpha games that missed deadlines and/or released too early. It's a non confidence inspiring trend that has become the norm these days.

 

The upside is that they were honest about their progress and are going to put it out there anyway in it's current state. It's a bold move.

 

I'm excited to see their progress first hand, but I'm trying my best to temper my expectations. I fully expect a broken, unfinished, PRE-Alpha proof of concept that if nothing else, should give us a baseline to measure progress from here on out.

 

Point being, not everything has to work, just enough to make it believable.

 

At the risk of sounding like a denying blind sheep, I will say what I said back when they pushed the alpha off from the "first half of 2017". I think that their vision of what they want the alpha to be has changed; yes they may be behind their original schedule, but as they have made the game I think they realized that what they wanted to put out (and what they should be putting our for public perception) has changed over time.

 

It looks like they're making big development advancements every month. Each week they push the alpha off means another week they can make the game just a little bit better. Theres no good stopping point (or perhaps there is which is the pre-alpha that they are giving us now), so they just have to pick a point and go with it.

 

I could be totally wrong of course. :P

Link to comment
Share on other sites

On 26.10.2016 at 8:15 PM, TehWardy said:

Looking at the raw numbers I presented before, lets assume my space station creation was something like 10 million Voxels, that's 10 million individual Voxels that 1 player has created that the server has to process and store and hand out whenever someone looks the right direction from the right spot on the game.

 

It seems you are expecting players to see these 10 million voxels at once. But that would require maximum LOD over a distance of several thousand voxels. That doesn't sound like a realistic assumption.

Link to comment
Share on other sites

Correct me if I'm wrong but some of you speculating something along those lines.

 

Google map? Impossible to see entire map of Earth in full details. According to our calculation its a 360 billion tiles 256x256 pixels. It's fake news!

Yet we using it on daily basis including overlayed information about traffic real time. ;)

 

 

Link to comment
Share on other sites

17 hours ago, MaximKammerer said:

 

It seems you are expecting players to see these 10 million voxels at once. But that would require maximum LOD over a distance of several thousand voxels. That doesn't sound like a realistic assumption.

Yeh I see the confusion now.
I'm obviously (with the lack of input from NQ at least) making assumptions that may or may not be true.
I also understand why NQ would not want to explain the inner workings of their tech ... they've basically got the holy grail of all things gaming going on here.

But I will try to explain myself ...
By this comment I meant something like (based on assumptions I was making about how "typical Voxel code i've seen works ") ...

day 1:
No data in the database, all planets, moons, asteriods, ect can be "procedurally generated" by client based code.
Server load is trivial, clients can simply "compute their view on the fly" and all NQ serves up is "function trees" for use in compute operations (which could be downloaded as part of the client install).

day 7: 
Lots of edits on the surface of a planet, each "Voxel instruction" resulting in a "net change" to a subset of an area of the starting planet (Alioth).
Server load is higher as each user within range of those changes needs to be fed that data in order to compute the meshes on the client.

......

Then my perception changed.
I went from thinking ...
"NQ will serve up raw Voxel data to clients"
   ... to thinking ...
"NQ will compute mesh data and serve that up to clients, edits all happen on the server and clients get the resulting mesh and thus never see any raw Voxel data".

....

day 7: 
The changes have been applied gradually, each change results in a "server side mesh recompute for the affected chunk at various LOD's".
Server load is reasonable as over time, those that come across the changed area are simply fed the updated "pre-computed mesh info" .

day N:
Massive changes to the planet, massive mesh changes, huge structures built (like space stations) all of this stuff is "pre-computed as various LOD meshes" and stored on the server.
Server load per user remains effectively constant with the calculation of that being something like  "meshes in range * number of LOD's"  where range is "configurable" by NQ.

Mesh data further away can be served up much slower than mesh data near by and there's only so much you can get in a scene but also unless a mesh is edited once a player has it they have it.

My thoughts now:
Over time, there's effectively (due to the nature of Voxel tech) an unlimited number of possible meshes that could be generated and storing that data could be pricey (in terms of raw drive space.
But if NQ are smart ... a bunch of TB aint that bad ... the real issue I see here is "sharing that mesh information on demand in real time on top of normal network load".


So consider this scenario ...
It's now about 2 years after release, Stargate's have been a thing for a few months and I am early adopter of the Stargate system so i've migrated way from the bulk of Alioth to pursue some venture elsewhere.

I'm in a Large org and my org needs me for a large battle ...
So I jump in to my ship, pop through the nearest Stargate to Alioth and make my way to area where the battle is taking place.

There are several things I need in order to render this battle ...
 1. it's near the planet so I need mesh information of the view of the planet from space 
 2. I've been out of touch with the local market so I likely need a bunch of ship models
 3. There's probably a new  station near by I might or might not have seen before so I need that mesh data.

All the while the same thing is going on for the 10,000 other users in the same fight but what they need may be the same set or a different set of data but we are all asking for what essentially amounts the sum total of either all or a portion of the scene in which we just entered in to.

This work combined with the actual battle means ... Network load goes through the roof.


So my question was more ... 
How is NQ going to handle this as there's only so much an "average internet pipe" from the server and too my machine that can be handled?

 

Link to comment
Share on other sites

50 minutes ago, Captain Jack said:

seriously @TehWardy, your cred as a developer is diminishing with every post.

I think he meant, "How is NQ going to handle this, as there's only so much an "average internet pipe" from the server to my machine can handle?"

There is no need to be a jerk about grammar mistakes, especially since English is at least a second language for a lot of people in DU. If you are complaining about him using non-technical terms, I don't see how that diminishes his reputation. I would say that it increases it since he isn't only talking in super technical terms that many people wouldn't be able to follow.

Link to comment
Share on other sites

5 minutes ago, Shockeray said:

I think he meant, "How is NQ going to handle this, as there's only so much an "average internet pipe" from the server to my machine can handle?"

There is no need to be a jerk about grammar mistakes, especially since English is at least a second language for a lot of people in DU. If you are complaining about him using non-technical terms, I don't see how that diminishes his reputation. I would say that it increases it since he isn't only talking in super technical terms that many people wouldn't be able to follow.

Heavens no, I was not insulting his use of grammar, nor his person. I withdraw my comment and will crawl back into the hole from which I came.

Link to comment
Share on other sites

4 minutes ago, Captain Jack said:

Heavens no, I was not insulting his use of grammar, nor his person. I withdraw my comment and will crawl back into the hole from which I came.

I'm sorry, I didn't mean that your opinions weren't valid, just that your approach was not constructive. It would be helpful for you to point out where you think his logic is flawed rather than just claiming he isn't knowledgeable on the topic with a blanket statement. We can't all be experts in everything.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...