Jump to content

I really want to believe in this but ... really, this doesn't add up!


Recommended Posts

Ok i'm a developer myself, oddly enough this is my dream and I have been crunching this problem for years trying to wait out hardware that can handle the kind of workload involved here so please don't take this as an insult but I'd like to see some rationality here as to how this really works (although I accept that there's some reasonable limits due to IP protection on your part).

 

So gut feeling here ...

Take Minecraft and smarten up the voxel tech a lot adding LOD and lets face it much better volume management (that should help it scale a little bit) and then allow the creation of multiple "volumes" within the game world. 

 

Lets put some real numbers in to this to exlain the problem I am facing ...

Take the moon (that's 3474km in diameter), now lets assume that the voxel data is the same as Minecraft's where "each voxel is 1 cubic meter" so you need a volume of 3,474,000^3 which is a total of 41,926,580,000,000,000,000 voxels to represent the moon (ever seen a planet that small? ... hmmm, no i heaven't either).

 

Lets be reasonable about this ...

Even at 1 bit per voxel that's a lot of data to crunch but the reality here is that you would need more than just 1 bit per voxel, with the bulk of voxel engines I've seen or written using at least 1 byte usually 2 per voxel.

 

Taking a "non uniform object" example ...

In one of the videos you guys said the space station was 4km in size (so that's a cubic volume of 64,000,000,000 voxels asusming I did my math right), even that seems to raise a few red flags for me but that volume is mostly empty space so i'd estimate you're using maybe an 8th of that or less but that's still a station with actual values for maybe 8,000,000,000 voxels.

 

I just bought a brand new nVidia 1080 GTX graphics card and even if I have that crunch the numbers on each of its 2.5k+ gpu cores it's still going to busy for some time handling that amount of raw data for a whole scene so it's unlikely any client ever sees that raw data instead there must be some magic going on on the server to only send "mesh data" / some form of "view data" from a given view of a volume at a given LOD.

Doing that though, puts a hell of a lot of computation on the server farm when you have say 1,000+ players all editing stuff in a small area all that computation is going to get pricey, and farming that load only adds network traffic and the need for more copies of that data to reduce data access overheads!

 

...

 

Speculation:

A view of a "planet" is generated by using a fractal algorithm that describes the shape at varying levels of detail depending on how close you are and then any changes are applied to that using some clever "this 1 Voxel change isn't visible here but that 1,000 Voxel change does X" type algorithm.

 

The raw Voxel data is only stored when edited / player created ... but that could get expensive over time as Voxels are easily editable and very fast raising the question is this a sustainable platform or is there some plan to "retire old data"?

 

Large "ships" / "stations" are beyond me ... you can't "algorithmic-ally compute" those so how does this all work without going beyond the laws of physics for computing the data or network load?

 

...

 

The raw numbers are massive but computers are good with raw numbers and large data sets so the problem appears to be "interesting" to say the least.

 

The closest we have seen to this is something like Everquest Landmark and the failed Everquest Next project, those games were built on the Voxelfarm engine. 


 

I once had a discussion with the Creator of Voxelfarm by email when doing some research in to Voxel engines (I was debating writing my own or using one someone else wrote to build a pet project) and 2 key points came out ...

  1. Small or big, apparently doesn't matter if your search algorithm is smart enough you have a fixed cost (but that lost me)

  2. Physics is where it gets hard not simply drawing!

 

I tried landmark out but it suffered from some serious limitations ...

  1. The sharding issue (to handle the network load) which you have somehow solved (or claim to) yet not even the eve team with it's years of experience has cracked that particular problem (simulations don't count).

  2. The game ran slow, even on a high end PC typical frame rates were poor <30 fps was normal (although I admit I was playing with it in beta)

  3. Persistence just didn't happen ... I would dig a tunnel, walk away from it, come back ... and the hole is gone! ... and why ... disk space is expensive when you need that much of it.

  4. The raw numbers here just don't add up unless you have massive voxels (like greater than 10 cubic meters) or you have a chunking algorithm that can spread the load over many servers (something landmark didn't do), the problem there of course is network load leading back to point 1.

 

...

 

So i'd like to ask for more details on how this actually works as looking at raw numbers for Voxel data clearly is not helping me and you guys must have some answer to this raw scale problem!

FYI: I wanted to pledge during the Kickstarter and this is the reason I haven't yet ... really hoping you guys can put my mind at ease as I'm seriously considering pledging enough to get me alpha access (if that's still an option).

 

Link to post
Share on other sites
  • Replies 51
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

The wider gaming community has long displayed some negative characteristics. One that springs to mind is the desire to see a project fail. So these type of questions are going to arise again and again

Ok i'm a developer myself, oddly enough this is my dream and I have been crunching this problem for years trying to wait out hardware that can handle the kind of workload involved here so please don't

The planets in game are not life size, I believe the starting planet is 100km in diameter, if that helps. That is the same size as the planets in Space Engineers and there aren't any server issues in

The planets in game are not life size, I believe the starting planet is 100km in diameter, if that helps. That is the same size as the planets in Space Engineers and there aren't any server issues in regards to modified planets and asteroids even without the zoning tech that DU has (SE is open-source btw), in SE the issue is mainly with active floating player-made constructs that does not scale; but in DU the player constructs use the same voxel tech as the planets. I don't know any of the technical stuff so that's just my outside observation. Hope you can get some more detailed answers.

Link to post
Share on other sites

They bake their voxel constructs as a single mesh when built to reduce the load...

Your numbers are not taking into account occlusion... that's hiding what you can't see, like, the inside of a planet, or a ship, or a moon...

Also, you are seeming to not be familiar with LOD... that's the detail of an object when it's far away being lower, which is what they use on constructs far away... which are baked already.

Also, I don't know what Voxelfarm engine you speak of, Landmark uses ForgeLight . 


 

Link to post
Share on other sites

They bake their voxel constructs as a single mesh when built to reduce the load...

 

Your numbers are not taking into account occlusion... that's hiding what you can't see, like, the inside of a planet, or a ship, or a moon...

 

Also, you are seeming to not be familiar with LOD... that's the detail of an object when it's far away being lower, which is what they use on constructs far away... which are baked already.

 

Also, I don't know what Voxelfarm engine you speak of, Landmark uses ForgeLight . 

 

 

 

Baking a construct takes processing time, and I fully understand LOD (as I previously stated I am a developer myself).

 

I think you're missing the point I was making that this is not so much unachieveable on a small scale and with a few tricks but more a problem of large scale.

 

If 100 people are working on a large space station visible from a nearby moon say, the changes would be insane and thus trigger lots of "re-baking", the cost would quickly result in a ton of network traffic to communicate the baked changes to each of those 1,000 people looking at the object from nearby.

 

You can "cheat" by basically making a few assumptions in code but the problem is still a pretty immense scaling issue.

that said, if Microsoft can convert the whole of Wikipedia from English to Russian in 1/10th of a  second i'd really love to know what DU can do in the same time :)

Link to post
Share on other sites

The planets in game are not life size, I believe the starting planet is 100km in diameter, if that helps. That is the same size as the planets in Space Engineers and there aren't any server issues in regards to modified planets and asteroids even without the zoning tech that DU has (SE is open-source btw), in SE the issue is mainly with active floating player-made constructs that does not scale; but in DU the player constructs use the same voxel tech as the planets. I don't know any of the technical stuff so that's just my outside observation. Hope you can get some more detailed answers.

 

Hmm ... might have to do some digging through the SE code ... see if that lends any interesting light on the subject. 

Thanks for that, didn't know it was Open Source :)

Link to post
Share on other sites

So, your post is quite large, and I didn't have enough to to go through it all, but I think I can make some assumptions and possible tell you more about this.

 

To start off, planets are not physically stored as I bet you already knew. The changes to them is stored. This is due to it being generated off of a seed and then the changes to the voxel. You should hopefully know this.

 

As far as rendering, it would make sense to be speculative as this is quite a task for any computer, but let's take a look at some examples. In SE, rendering is done in quite a unique way. Although, I bet there is some optimization to be done so that's why there is quite a bit of lag. Nevertheless though, when planets are rendered in SE, they are done differently from all objects in the game. The maximum view distance in SE is normally around 40Km if I remember correctly. When planets are rendered they are done using a slightly different rendering technique. I've forgotten most of the details, but it means planets are viewable from long distances.

 

Now, if we are to assume that DUs planets are rendered similarly, then we can also make a number of assumptions. I'm going to do a quick rant that's probably only going to scratch the process due to my lesser knowledge of this subject, but I continue.

 

Let us guess that we load into the game and that takes some time while the user get's all the information it needs. This includes what is able to be viewed. When this happens, we got one hundred percent of the information we need to render something, and then we render it.

 

So we know the system of rendering is done in updates. The update frequency is dependant on the users distance from the object. So to send an update for a planet is quite simple. We can estimate very roughly that a planet far, far away is only going to be sending updates every once in three minutes we'll say. Again an assumption. So this planet only sends updates so often. The question is now, will it send the updates for all of its objects or only some?

 

This is completely theoretical but it's something we can take into consideration of how a process like this might work. I would love for the devs to clarify some of this, but we'll continue. Let's assume the updates for a single planet and its objects are somewhere near 100 thousand(rough number. Probably more though) updates. So let's assume we take an algorithm that downsizes most of those updates into a reasonable amount without affecting the quality. An example would be how you downsize picture resolution, although this is much more complicated. This would mean that we get less updates because we don't need them all. From there, let's guess we have one thousand updates.

 

You could then send these to the client somewhat reasonably. The client takes the data, and updates existing data on the planet.

 

Again, this isn't any details based on evidence, and Novaquark probably had a solution better than mine, but this theory makes sense. All you need to do is update some things that are currently being rendered, and you don't need to update it all.

 

The calculations of rendering don't seem like it'd be that bad either. Most computers already render insane amounts of information. Space Engineers renders so much, and it's not the most optimized game either. If NQ optimizes their game, and applies a technique like this or even better, it doesn't appear that it will be a problem.

 

I am limited in knowledge on some of this subject though, so take it with a grain of salt.

Link to post
Share on other sites

You can reduce storage space with procedural generation.  Then you just need to store the seed that made it... and any player modifications.  The planets themselves will not be Earth size... they'll be smaller... smaller possibly even than the Moon.

 

Mercury is only slightly bigger than the Moon and Pluto is smaller than the Moon...  Our Moon is really big as moons go... it's so large that the Earth Moon system could almost be called a Binary Planet.

 

But back to voxel storage.  It could take decades for there to be enough player changes to a planet for storage to become an issue... and by then both bandwidth and storage capacity will have expanded.

Link to post
Share on other sites

For constructs.

 

When you make a construct like a ship or station... You start with the Core unit and add voxels... these get registered with the unit and when you're all done a mesh for it is generated.  So you're flying around in a single mesh object.

 

 

There currently isn't collision damage... or combat deformation of constructs... 

 

Collision damage might never be a thing as they don't want people making their own missiles. They want all combat damage to be done through their weapon system... Which could be made to calculate and make changes to the mesh for combat damage.

 

That eliminates the huge amount of data that would otherwise need to be transmitted.

Link to post
Share on other sites

You can reduce storage space with procedural generation. Then you just need to store the seed that made it... and any player modifications. The planets themselves will not be Earth size... they'll be smaller... smaller possibly even than the Moon.

 

Mercury is only slightly bigger than the Moon and Pluto is smaller than the Moon... Our Moon is really big as moons go... it's so large that the Earth Moon system could almost be called a Binary Planet.

 

But back to voxel storage. It could take decades for there to be enough player changes to a planet for storage to become an issue... and by then both bandwidth and storage capacity will have expanded.

Just to be sure: Pluto is not a planet, it's a planetoid.

 

With a radius of only 100km, storing only changes, generate the planet with only math, updating far away/not seen players/constructs only once in a while I think it makes sense that it could work. But we will see in alpha how good this will be

Link to post
Share on other sites

Just to be sure: Pluto is not a planet, it's a planetoid.

 

With a radius of only 100km, storing only changes, generate the planet with only math, updating far away/not seen players/constructs only once in a while I think it makes sense that it could work. But we will see in alpha how good this will be

Precisely, you can say it works fantastic... a perfect, flawless system.  But the fact of the matter is there will probably be hiccups when they integrate everything to the live server.  Wouldn't worry about it though.

Link to post
Share on other sites

Baking a construct takes processing time, and I fully understand LOD (as I previously stated I am a developer myself).

 

I think you're missing the point I was making that this is not so much unachieveable on a small scale and with a few tricks but more a problem of large scale.

 

If 100 people are working on a large space station visible from a nearby moon say, the changes would be insane and thus trigger lots of "re-baking", the cost would quickly result in a ton of network traffic to communicate the baked changes to each of those 1,000 people looking at the object from nearby.

 

You can "cheat" by basically making a few assumptions in code but the problem is still a pretty immense scaling issue.

that said, if Microsoft can convert the whole of Wikipedia from English to Russian in 1/10th of a  second i'd really love to know what DU can do in the same time :)

Yeah, you have no idea whatever I said means. And smooth moves there buddy, calling ForgeLight a "voxelfarm" engine. Yeah, because we all know Planetside 2 is all about them voxel-buildings.

 

You don't understand what baked means and you are no Dev. You clearly can't understand what occlusion and LOD are, as well as not understanding what procedural generation is.

 

They don't need to store stuff, there's an algorithm that does that for them. IF you were a game dev, you would know what mathematical concepts go behind algorithms, which is why the Devs have said their planets and their ships follow the same algorithm in the code. PRocedural generation means that the changes are in your client always and never.

 

It''s a mathematical thingie, go to CSD school, you'll learn a lot of it.

Link to post
Share on other sites

Just to be sure: Pluto is not a planet, it's a planetoid.

 

With a radius of only 100km, storing only changes, generate the planet with only math, updating far away/not seen players/constructs only once in a while I think it makes sense that it could work. But we will see in alpha how good this will be

 

It's a dwarf Planet... which is a class of ... Planet...  but I digress.

 

 

Also I don't think anyone is saying or expects all this to work flawlessly on the first day of alpha...  Everything is going to need tweaking and modification.

Link to post
Share on other sites

Yeah, you have no idea whatever I said means. And smooth moves there buddy, calling ForgeLight a "voxelfarm" engine. Yeah, because we all know Planetside 2 is all about them voxel-buildings.

 

 

You're the only one who keeps mentioning Forgelight twerk.

 

Landmark uses BOTH Forgelight and Voxelfarm.

 

Forgelight is an MMO game engine created by SOE/Daybreak and used in many of their games including Landmark.  https://en.wikipedia.org/wiki/ForgeLight

 

Dual Universe is using Unigine.  

 

Voxelfarm is the Voxel Engine SOE used to create the fully destructible voxel world of Landmark.  http://www.voxelfarm.com/index.html

 

SOE/Daybreak did NOT create Voxelfarm.

 

Novaquark actually considered using Voxelfarm at one point.  https://board.dualthegame.com/index.php?/topic/14-voxel-tools-pre-alpha-game-design/page-4#entry2080

 

 

 

Hi Daddio!

 

While at some point we pondered the possibility to use Voxelfarm, we didn't in the end.

We are now using our own voxel engine. 

Link to post
Share on other sites

It's a dwarf Planet... which is a class of ... Planet...  but I digress.

 

 

Also I don't think anyone is saying or expects all this to work flawlessly on the first day of alpha...  Everything is going to need tweaking and modification.

A dwarf planet is a type of celestial body that has some characteristics that resemble a planet, but not enough to be classed as a planet.

 

Programmers have been working with voxels for several years now.

The devs think they have some revolutionary code.

We'll just have to wait and see.

Link to post
Share on other sites

A really good post from the OP. One that I think is deserving of an official response.

 

NQ have gone to ground lately - which I suppose is understandable at the moment - and when they reemerge I'd love for them to run something like 'Questions of the week / month' newsletter or screencast.  As I see it NQ will be presented with tough (and hopefully respectful) questions about the feasibility of their product and unless they want their very own equivalent of Derek Smark hounding them * they better start answering these questions and answering them well.

 

* I can't tell if he's a crank obsessed with mud slinging or somebody obsessed unearthing the truth.

Link to post
Share on other sites

They have already answered really, not much more they can do right now besides releasing the source code, it's just a question of believing it. And if you don't all you can do is wait until you can see it for yourself.

 

The way I see it, there are no issues with massive universes and procedural voxel planets in other games. The issue is scaling the number of clients in a single shard. NQ have focused on those issues and come up with their dynamic space splitting for the server side, and what I call "Network LOD" for the client side. In theory it makes sense.

Link to post
Share on other sites

They have already answered really, not much more they can do right now besides releasing the source code, it's just a question of believing it. And if you don't all you can do is wait until you can see it for yourself.

 

The way I see it, there are no issues with massive universes and procedural voxel planets in other games. The issue is scaling the number of clients in a single shard. NQ have focused on those issues and come up with their dynamic space splitting for the server side, and what I call "Network LOD" for the client side. In theory it makes sense.

Yeah this. I think they've already answered and addressed most of the stuff they can at this point. The rest will have to wait until they actually get to working and bring these systems online to test. They've made lots of claims with the biggest being their new tech they created for many continuous players in the same area, the use of severs to load on demand areas of the game world. If they get that working than the rest should seem like a walk in the park. It won't be obviously as there's always unexpected delays and issues that arise but bottom line if even half the stuff they are promising works as advertised then should be in for treat of a game haven't seen for a long time.

 

So yeah instead of trying to tell others how this all won't work might be more productive to try and get all the info and learn how it can and will work. New game, new ideas, new Devs with those new ideas and such. Give them a chance at least I say.

Link to post
Share on other sites

Well after the disasters with SC and NMS, people are a bit nervous with new ideas

Well the problem wasn't with new Ideas about No Mans Sky. It was purely a over hype under deliver situation. and what's the disaster with Star Citizen? Its still in development, taking longer then some would like but steady progress is being made. I don't really like their modular method, would prefer them to make the full game then add to it not peace meal like they are now but as far as I know there no disaster or anything around the project.

 

Its perfectly fine to be cautious and skeptical and we all should be at this point but lets at least see the game and hear more before just saying outright "it won't work".

Link to post
Share on other sites

They have already answered really, not much more they can do right now besides releasing the source code, it's just a question of believing it. And if you don't all you can do is wait until you can see it for yourself.

 

The way I see it, there are no issues with massive universes and procedural voxel planets in other games. The issue is scaling the number of clients in a single shard. NQ have focused on those issues and come up with their dynamic space splitting for the server side, and what I call "Network LOD" for the client side. In theory it makes sense.

 

The wider gaming community has long displayed some negative characteristics. One that springs to mind is the desire to see a project fail. So these type of questions are going to arise again and again and again and if NQ are not careful they will be feeding into fears. And that's not helpful when you are trying to crowdfund.

 

Let me put it this way. When I see a well-considered post such as the OP's go largely unanswered by the community (and that's not an attack on anyone here) or ignored by the devs a horrible question arises in my mind: "have I may have bought into another Starforge, NMS or any other turkey I care to mention?" And I'm sure I'm not the only one thinking this. Of course, there might be many legitimate reasons why the OP can't be answered and I've not lost faith in DU as a concept. However, I do think it would be very wise if NQ broke their radio silence very soon and started addressing such questions on a regular basis.

Link to post
Share on other sites

Well the problem wasn't with new Ideas about No Mans Sky. It was purely a over hype under deliver situation. and what's the disaster with Star Citizen? Its still in development, taking longer then some would like but steady progress is being made. I don't really like their modular method, would prefer them to make the full game then add to it not peace meal like they are now but as far as I know there no disaster or anything around the project.

 

Its perfectly fine to be cautious and skeptical and we all should be at this point but lets at least see the game and hear more before just saying outright "it won't work".

 

I'd point you towards the Derek Smart link I provided above. And I don't think that anyone has said at DU won't work. Rather, the OP expresses some reasons for his scepticism and came her looking for some answers. I hope that he(?) got what he was looking for.

Link to post
Share on other sites

Wow ... some great feedback guys.

Did someone say Unigine! ... I love that engine, their Heaven demo makes me want to build an MMO in that world!

 

I'm not looking to rock the boat like maybe ... BlueDrake42 or whatever his name is ... but I do want to understand what it is that NQ is saying they have here.

 

For example ...

 

Procedural generation 

I know that Voxelfarm worlds are procedural and look amazing but my experience in a shard based world was poor when using a real world MMO solution built on the engine.

The alternative is something like no mans sky that did not meet expectations at all and both didn't save the players chances (no mans sky didn't even allow any changes).

 

Voxel Technology

Oddly enough I have been talking about doing exactly this for a couple of years now but couldn't find enough support from a seasoned developer community to get help with this idea of using Voxels on this scale because I had developer after developer tell me "it's impossible on todays tech without breaking the laws of physics"

...

 

That said ...

 

I am believer in the technology and do feel it can scale, using procedural in combination with static data and compression / bit packing techniques for the transmission brings the scale in terms of raw bytes in to the possible but it does come with a few "negotiations" you have to make. 

 

As I said when I started this thread, the raw numbers of Voxels that represent a large object are just insane and with something like a space station or a large ship you simply can't start from a procedural base so every Voxel has to be considered a change.

 

Given that "fact of how Voxels work" you can then take a data source containing any number of voxels and generate a mesh from that data easily enough and using some pretty common programming techniques known by many today construct meshes at varying "qualities" for use in game.

 

So in short ...

 

My question really wasn't "Using this number of Voxels is impossible right?" it was more something along the lines of ...

 

Given that I as a keen user of the game spend say 2 hours per day for say 10 days in a row editing Voxels I could as a single player build a pretty hefty space station looking at the tooling available here it's reasonable that I could build one of those space stations.

Looking at the raw numbers I presented before, lets assume my space station creation was something like 10 million Voxels, that's 10 million individual Voxels that 1 player has created that the server has to process and store and hand out whenever someone looks the right direction from the right spot on the game.

 

I am however, only 1 player ... so lets scale that up ... add another 999 players and we have 1000 x 10 million Voxels that are all entirely non procedural and could be reasonably constructed on the game server.

even if those ridiculous number of Voxels are not right next to each other that's still a lot of data to be managing and processing.

 

For an MMO that's a tiny number of players and for a single shard that's an insane amount of mesh data being handed around, the previously mentioned "developer community" tell me this is not possible.

 

So how I see it ... 

This is either being massively over sold or there are some "gotchas" that are still being overcome (and I seriously want them to be) but these guys must be on to something that through years of digging around and researching this particular technology (Voxels) I have not found anyone that can do this or anyone who even remotely has an idea how to do this on this scale. 

 

That on it's own raises questions does it not? 

 

I really don't want another no mans sky ... but I do want to know that this stuff really is possible so a little insight in to the tech would be really nice to have if the devs over at NQ are in a sharing mood (just a little technical peek ... please!).

Link to post
Share on other sites

Well the problem wasn't with new Ideas about No Mans Sky. It was purely a over hype under deliver situation. and what's the disaster with Star Citizen? Its still in development, taking longer then some would like but steady progress is being made. I don't really like their modular method, would prefer them to make the full game then add to it not peace meal like they are now but as far as I know there no disaster or anything around the project.

 

Its perfectly fine to be cautious and skeptical and we all should be at this point but lets at least see the game and hear more before just saying outright "it won't work".

Read the ToS of SC.... No refund on anything - even when there is no game. Selling of ships, making more money, even when everything is delayed (no sign of S42), that sandworm demo was on a map especially built for that con and so on...
Link to post
Share on other sites

Read the ToS of SC.... No refund on anything - even when there is no game. Selling of ships, making more money, even when everything is delayed (no sign of S42), that sandworm demo was on a map especially built for that con and so on...

Well they also demo'd other parts of the game at that con they had. They showed two players doing a mission in a fully developed world from space to planet etc. Point is while slow, progress is being made. I don't even follow the game that much, I bought one ship a few years ago but am just waiting till the actual full game launches, not interested in any of the little instanced test beds they have going. But yeah nothing odd or out of the ordinary going on there as I can see.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...