Post Mortem; The Asteroids issue this weekend from a player’s perspective [SUGGESTIONS]

[blazemonger]

TL;DR

Staff on call/duty did well, process management seems to be lacking

 

Short overview of events:

This weekend we saw an issue where Asteroids did not spawn for the better part of 24 hours after they should have. About 2 hours after the expected respawn most old asteroids de-spawned and less than a handful new ones appeared.

 

While tickets were logged, by I presume several players, no communication or observable action from NQ happened for almost 24 hours.

 

On Sunday I received a response from support which was very informative and helpful, from it, it seemed that a (known) issue was causing new asteroids to not respawn as the previous ones could not de-spawn. When asked, support clarified no communication on this had happened as that was for the CM to handle who were not available outside of office hours to do so.

 

Then more information arrived, indicating at least one server dev was looking into the issue and a bit later a response on the forums in a thread on the matter from a dev indicated the issue was manually resolved, a fix would be pushed in a coming patch and asteroids were back in play.

 

This was confirmed and players could go about their business in game.

 

 

Takeaways from this incident:

Support staff and the relevant dev (team) did what they could, and the issue was resolved.

 

What was missed and would be an opportunity for improvement for NQ here would be:

 

1 - Communication:

If support staff if not allowed or able to make a public statement on an issue like this while no CM staff is available, this should be reviewed. While it is fair to say that such communication would need to be generic as there is much unknown and possibly not within support staff abilities to either know or confirm at the time, If a general notice of the issue being known and worked on this would have prevented the community and active players from not having any understanding of what was happening while they were (severely) impacted by the incident.

Especially due to the limited timeframe of this loop, many set aside time to participate and make sure they are ready for it. By not communicating, NQ is principally ignoring their responsibilities towards their player base in this regard.
 

2 - Timely action taken:

For a crucial game loop like this, a live ops game should not have 24 hours without any perceived action, whether (financial) resources limit NQ’s abilities or not. This could be considered a major incident and should see a response time of far less than 24 hours, more like 4 hours at most, especially since we could see that, once a dev got involved, this was resolved fairly quickly and so it is a valid question to ask why this could not have been done the day prior.

 

In closing, I would thank the support staff and dev(s) involved. There is really not much, if anything that can be put on them here it seems, this is a process management issue that is for team leads/management to adjust and resolve.

I’d be more than willing to support/assist if needed, as this is part of my expertise and I invite NQ to contact me in DM if so desired to discuss.

FYI; @NQ-Abu @NQ-Sesch @NQ-Nyzaltar

[Maxim Kammerer]

3 - Don't implement basic gameplay loops as scheduled events:

 

If asteroids would randomly spawn and de-spawn every day most players would't even notice that there is such a technical problem.

[Frank3]

Being a former support staff, I know that first priority is information to the user/customer. It even has higher priority than solving the issue at hand.

 

Information must be given quick and regular, even if the only information that can be provided is "we are still working on this".

 

Users/Customs are more forgiving if they get information. 

 

I don't see why the support system can't include a message system, to inform the users/customs. This is not something that can wait until certain staff is available. 

 

I worked in the support for the largest shipping company in the world. And we, in the global helpdesk, was the ones sending out information to the customer on issues 24/7, not some custom management that only worked business hours. We would even get "fined" if we did not.

 

We users can't "fine" NQ, but we can stop paying for the game. Many players already left, where one of the reasons has been lack of communication from NQ. 

Edited January 19, 2022 by DevilsOwn

[Frank3]

6 minutes ago, Maxim Kammerer said:

3 - Don't implement basic gameplay loops as scheduled events:

 

If asteroids would randomly spawn and de-spawn every day most players would't even notice that there is such a technical problem.

Correct, that is another issue with asteroids. The spawning (and despawning) should not be limited to the weekend, it's bad game play + it's plain stupid to have a massive event like this schedule for when no devs are working to fix issues. 

[blazemonger]

41 minutes ago, Maxim Kammerer said:

3 - Don't implement basic gameplay loops as scheduled events:

 

If asteroids would randomly spawn and de-spawn every day most players would't even notice that there is such a technical problem.

 

While I agree with the point made by itself, it is not relevant in the context of the incident this weekend and how it was handled. What happened here (and I know the cause but will not disclose by request of NQ) would be an issue regardless of whether this is more integrated into the game, as a recurring gameloop instead of a weekly event with a set (re)start point, or not.

[Warlander]

Personally I dislike how the asteroids have been handled in general as now that mining has been removed, MUs added, and the focus and sole means of actually getting decent amounts of ore is only reserved for the weekends and since NQ goes home for the weekend any issues such as not respawning proper amounts or issues.

 

Personally I think that not really being able to mine on planets other than Calibrations is a huge issue since there is no other way to really actively mine ore other than asteroids unless you mine rocks. Just having one spawn per week is a terrible idea imo even if they want it to be a draw for DU should have been daily or perpetual or with RNG based spawns so they arent wiped out in a day or two and you could mine as much as you wanted as constant content.

 

Really they should have has asteroids aver being discovered and being broadcast with their locations start a countdown timer in which would automatically remove dead asteroids and then que the number of dead asteroids to the spawn and ore generation script to automatically repop asteroids perpetually with a random number range of asteroid spawns each 24 hour period.

 

But beyond that there really needs to be some kind of active on world mining system that is actually worthwhile between weekends since MUs should have a Calibration type instant spawn but also it should also randomly spawn those rocks tied to what the MUs are producing to make active ground mining for the other 5 days of the week when asteroids aare mined out so that if there are issues like minimal asteroids being spawned there is still other options of actually gaining ore even if you cant dig into the ground.

[BlackLionoftheStars]

23 hours ago, Warlander said:

But beyond that there really needs to be some kind of active on world mining system that is actually worthwhile between weekends since MUs should have a Calibration type instant spawn but also it should also randomly spawn those rocks tied to what the MUs are producing to make active ground mining for the other 5 days of the week when asteroids aare mined out so that if there are issues like minimal asteroids being spawned there is still other options of actually gaining ore even if you cant dig into the ground.

 

I concur with your other thoughts as well.

 

As to the quote, a possible solution would be temporary or recurring surfaces of planets. Once a mining unit has been placed, a local surface of a planet or moon could contain mineable ores measurable to the amount available in any given cycle at maximum for mining units in general. But once it is gone, the mining unit turns off and done is done. Reset wormholes every 24 hours based on individual automatic mining units. No change in surface composition, of any place since Demeter ore distributions necessary. Small server hit, but only at mining unit locations. Removes massive data everywhere and adds to it locally. Give new players the option to Sanctuary to start or buy a territory somewhere.

 

The whole process would be contingent on current or modified calibrations remaining.

 

Would need a sort of conglomeration of old and new surfaces, but randomly generating from rare to very rare events may be manageable.

 

Skip or fix the asteroids.

 

 

Thoughts?

[blazemonger]

I have several times suggested that NQ expand on the skittle Surface Harvesting. 

 

Remove the "everywhere" and create "random" pockets which sill respawn in different areas

Add items like gems in the mix, to be found in thse pockets as well and use these to buff/specialize elemnts like engines or use then to create research/invention mechanics, allowing players to buy only tier1 schematics and then use tools and talents to improve them or trun them in higher tier ones.. Yes, much like EVE but it works perfectly well there so why not copy something els ethat works from EVE

 

[Warlander]

25 minutes ago, BlackLionoftheStars said:

 

I concur with your other thoughts as well.

 

As to the quote, a possible solution would be temporary or recurring surfaces of planets. Once a mining unit has been placed, a local surface of a planet or moon could contain mineable ores measurable to the amount available in any given cycle at maximum for mining units in general. But once it is gone, the mining unit turns off and done is done. Reset wormholes every 24 hours based on individual automatic mining units. No change in surface composition, of any place since Demeter ore distributions necessary. Small server hit, but only at mining unit locations. Removes massive data everywhere and adds to it locally. Give new players the option to Sanctuary to start or buy a territory somewhere.

 

The whole process would be contingent on current or modified calibrations remaining.

 

Would need a sort of conglomeration of old and new surfaces, but randomly generating from rare to very rare events may be manageable.

 

Skip or fix the asteroids.

 

 

Thoughts?

 

Mix & Match or take your pick.

 

1: Instanced based mining via MUs

 

Honestly they could have a version of onworld mining that uses instances like the mining tutorials for mining. Honestly they could create a tile size instance that is 1km deep that is like traditional mining that relies on the calibration scans for where ore is located and allow players at the end of the calibration an option or chance to actively mine what shows up in the scan.

 

This could be achieved by introducing also some kind of element that is like a mine shaft elevator that acts like the entrance to the instance that eats or uses a tile scan result or after the calibration results linked to a mine shaft elevator element to generate an underground instance in which loads as you go down and when the ore has all been mined you go back to the elevator and it takes you back to the real world so that it has all the elements of controlled mining as well as how much ore is generated to have voxel based mining onworld since it was removed for asteroids to create a more active version of mining that closes after you jump back onto the elevator.

 

2: Retrofitting the old mining system to MUs without voxel based ore.

 

NQ should really consider retro fitting the old system of voxel based mining (minus the voxels) to match more with how MUs work now and have finite ore pools that are in line with random tiles you would find before demeter and tier the respawn of tiles to the weekly tax cycle.

 

Where if you look at MUs as they are now where there are ore types to choose from instead of the L/H limit they have a pool of ore in line with pristine tiles you would find pre-demeter with scan results from territory scanners. To where each tax cycle it generates a new pool of ore you can mine as fast or slow as you want with how many calibrations you and your org have at your disposal to make best use of what ore is on the tile that changes each week and you have a week to clean out the pool with as much or little amount of miners as you need to do so.

 

3: Generating L/H from MUs directly onto the surface on top of the production into containers

 

First I want to say that all the ground rocks already have a generation script every 24 hours and in its current state is useless to have billions of rocks spawning that bogs down performance that nobody will ever realistically mine that needs to be rendered and loaded in multiple territories around you wherever you go even if you pay attention to them or not. They have never really been needed or wanted or made use of since they are all 20h ea.

 

With that said I think that all those rocks should be removed from the game to streamline performance and instead of the rocks the calibration charges spawn the L/H output of MUs should spawn those rocks onto the ground instead so that you can gain ore from what the MUs generate already as well as the calibration charge spawns, but also that the MUs on all your territories should automatically spawn the equivelent of the L/H additionally onto the surface so that there is somewhat of an onworld mining component in place that you can collect at your leisure that is actually worthwhile to do rather than rocks that do nothing.

 

4: Sifting as a feature

 

Since you dont want to create an incentive to basically turn sifting into voxel mining 2.0 sifting should not actually dig into the ground as much as clone the mining tool that can be used in owned or un-owned tiles that generates some alternate resource like sediment, strata, clay, stone, etc that distinguishes tera voxels vs a different resource that can be generated without digging as a sifting tool instead of a mining tool.

 

SO basically you could walk around running the sifter to collect different sediments and as an alt function of the sifting tool actually sift it so that it creates a more active mining system that is more user friendly than collecting rocks on the ground but also at the same time sift sediment to produce ore, gems, or different things as well as the potential for industry sifting units that have better gains.

 

5: Prospecting to upgrade tiles via prospecting

 

Taxe as a quanta sink do not target those NQ sought to drain quanta from and instead targeted New Players, casuals, builders, and crashed the economy since it created a treadmiill of needing to make as many millions as you and your org have tiles for abysmal ore gains unless you find a legit workable tile and unless you do you are stuck with barely sustainable tiles. 500k per week is much more doable in taxes.

 

NQ should have a quanta version of the talent system baked into each TU where players can spend millions willingly that is not a stick as much as a carrot that player choose to spend their quanta to upgrade the ore L/H @ +1mil per ore tier for +10 L/H per hour of that ore or being able to discover another tier ore that wasnt origionally found there for +10 mil h per tier to open up +10 L/H of that ore.

 

This would do many things like instead of needing 30 tiles you could upgrade 1 tile to support all your miners or making more use of existing tiles that players can invest in their territories that also bakes those stats into the tiles themselves making them worth more by creating a sort of real estate market as well.

 

6: Changing MUs in general

 

Calibrations need to either be applied to all elements or be removed. No other talent tree has this much tedium and people would rage quit if you had to constantly calibrate ships for 3 days/300 SU of use or industry units every 3 days. Even though I think that there should be some kind of system to slow down how much can be produced via industry I think NQ has set expectations and the industry should remain as is.

 

The other side of MUs is it does not seem to be for miners as much as the quickest talent progression to allow industrialists to mine without mining as much as remove voxel mining from the game since you only need to grab rocks when calibrating and it really does not have any real form of active mining component similar to asteroid mining or mining in general.

 

I think that if you are only wanting to afk mine the calibration system should be changed to just constanntly run off nitron, kergon, or rocket fuel for whatever miner to opperate the miners and rocket fuel as a sort of nitrus boost type fuel if you choose to use it. And linking fuel tanks in whatever size you think you need directly to miners and just letting them run for those who want to just afk mine and collect ore in the output containers instead of the efficiency drops it at least would use something tangible to keep miners constantly running at full efficiency with a resource sink rather than a time sink if taxes aremeant to take quanta out of the system fuel consumption does essentially the same thing since it take mats out of the game to keep the ore being sold worth more and gives those who just want to afk collect ore the ability to do so without having to collect the rocks from calibrations.

 

AND since if you are just running the MU on fuel it is pretty much in use. BUT if you have MUs in your territories that are not actively producing ore then you should be allowed to do active mining in which you can keep playing the mini game to spawn ore to actively mine with a 15 min cooldown like territory scanners. If not doing the same thing if you link 1 territory scanner to 1 MU that spawns the ore so that it has an investment and time cost to spawning ore and an active mining component. 3-5kl every 15 min and then having to run and find it is still less than the old voxel mining system since you could get easily a container or two of ore per day it should be similar for MUs as asteroids are a weekend thing only unless you like wasting time chasing down mined out asteroids.

 

7: Mining Talents should also apply to ground rocks and calibration rocks

 

Its baffling that for whatever reason NQ does not consider mining ground rocks to be mining since you are "harvesting" ore and those with actual mining talents should get more ore since they have the talents for it as well as reduced time amounts since that is also part of the mining tree.

 

The mining tree in general is mostly filled with non mining talents as is and is the least developed tree in the game vs piloting, weapons, ammo, or industry and again it feels like MUs are more for industrialists rather than miners who get no additional benefits even if they have specced into mining or dredging talent tree branches or that everyone and anyone can take a month or two for the MUs and pretty much compete with asteroid miners at the cost of quanta.

 

 

Asteroid Solutions

 

1: Changing the Nature of Asteroids in general

 

The makeup of asteroids seems counter productive to the actual makeup of asteroids in which they are not made mostly of dirt, snow, or sand and should be made almost exclusively ore only as there is no real reason whatsoever to have dirt persisting in space for a week at a time that is the reson voxel mining was removed in the first place even though asteroids were dropped before the demeter update.

 

Asteroids should be more like a jaw breaker in the sense that the makeup of the asteroids of basic, ucommon, etc should have the outside layer being whatever the lowest tier ore is with inner layers being higher tiers if there are any. Personally I dont think that the DSAT system should even give a distinction of rarity in ore tiers should be distinguished as much as there is an asteroid there and you find out what is there when you get there so people dont just run to the better asteroids and it creates a bit more randomness to what you can find once you use the ore scanning tool.

 

2: Core Based Asteroids

 

If asteroids are just ore and are a version of teravoxels like dirt asteroids could and should be core based in the sense that the system can spawn asteroids via an unlimted golden blueprint script where it spawns an asteroid the size of various cores from XS-L or XL when those cores become available and since you cannot pick up a core until all that is contained inthe core is removed NQ can make a script for asteroids that checks every so often like 1 hour, 2 hours, 6 hours, etc so that if there is nothing left on the asteroid it will count as being despawned and added back into the que to be spawned again.

 

The cores can be invisible or collectible if NQ wants to reward players for mining out the full asteroid for a prize at the bottom of the box as it were. But it also allows for the asteroids to be both static and dynamic so that you can add some kind of element of danger if you come across dynamic core based asteroids if they have a rotation added or if they get caught in a planets gravity or are give some kind of speed component and or also a waypoint system toa ctually create a moving rotating asteroid belt as well as static asteroids. Making higher tier ores more dangerous to collect if they are either moving or rotating and you have to land on them.

 

NQ could also use this system for meteors, and asteroid impacts if they spawn dynamic core based asteroids they could litterally send them anywhere they want with a auto pilot script to bombard planets and if the core is given a PvP weapon based script to punch a hole in the ground like a giant teraforming tool or to burn up in the atmosphere depending on the speed or angle they come down on a planet would be an interesting option to seed voxel ore on planets even if it does no actual damage even though it would be interesting if it actually damaged elements or destroyed voxels.

 

3: Adding "harvestable" versions of asteroids or asteroid fragments

 

NQ should consider making asteroids like the ground rocks just massively larger in the sense that they should add in ship based elements for either both space or onworld that lock onto these asteroid type harvestable asteroids like PvP but instead of blowing it up it just has a harvest tool like wuality that if you are in range points a giant lazer at the asteroids to either break it down into smaller chunks until they hit the size of your ships mining equipment to harvest and you just collect giant chunks of ore without needing to spawn voxel asteroids to do the same thing or on top of voxel based asteroids.