Jump to content

Integrate LuaSocket Into The LUA Sandbox


yamamushi

Recommended Posts

JC posted this tweet this evening, 

 

 

Based on the responses in that thread, that puts a roadblock in the way of being able to use LuaSocket to push HTTP requests up to servers.

 

I should explain what that means...

 

If we had the ability to post data over HTTP to our own application servers, we'd able to do things like create phone apps that track the status of your ships, dashboards for having a high level overview of a fleet of ships, sites that track statistics about ships and other constructs, tools that notify you automatically if something happens through your phone, market tracking applications, etc.

 

(Note that this would not allow you to play the game remotely, this is entirely about receiving data and rendering it appropriately)

 

 

My "idea" is to integrate LuaSocket into the LUA sandbox for outbound requests only so that we can post data to our own applications for external use. That way we don't have to get explicit access to load our own modules. It would give us the opportunity to build the kind of tools mentioned above and would allow for NQ to specifically limit what can and can't be posted upstream. 

 

I think it would be a nice middle ground and a convenient way of giving us an "API" to use without NQ having to explicitly create one themselves (and arguably less work than creating a maintaining a whole API for this purpose). 

 

 

And since I don't want to keep bothering JC on twitter at 7 in the morning..... to address the issue about DDoS's, I think it should enforce a FQDN policy that only allows for remote hosts to be specified by domain name instead of IP addresses (thus not having stale IP's sticking around that could be flooded later on), and at that point it would be up to the application authors to be able to handle the traffic (which is a whole other conversation that I'd be happy to explain how to address). Or by the time there are millions of players, they might an API at that point we'd just migrate to using instead :P 

 

But I do understand the problem of stale scripts sitting out there that are abandoned by their authors/maintainers and then years down the line causing headaches for people. 

 

There's also the issue of someone maliciously targeting a site, getting the script on tens of thousands of ships, and then sitting back while that site gets flooded with traffic.

 

 

Perhaps someone has an idea about it that doesn't involve just turning off the feature later on down the road when they give us a real API?

 

 

The only "solution" that is coming to mind involves having some type of standard handshake for receiving the data, and if it fails, then not enabling the feature. That way if a site can't actually process the data, it doesn't receive any more requests from said client until the client is restarted. If it's getting too much traffic, the handshake would time out and fail quietly, otherwise, it would have the acknowledgment that the application can actually handle the data and it would proceed to send. Periodically it could check that the application is still there by initiating another handshake. It could even be something set in user options under "Enable Data Publishing Through Lua", and default to "no" or "off", thus minimizing the impact even further. 

 

The problem with that idea is that it now starts to put even work on NQ's side for what may amount to otherwise a trivial feature that only a handful of us end up using. I mean, I know there's plenty of people here that could hack together a handshake method like that in a few hours, but putting that into the game is a whole other thing altogether, on top of the already loaded roadmap. :(

 

 

Okay this post turned into me thinking out loud, sorry >_<

 

 

Link to comment
Share on other sites

I don't quite know what you were talking about there lol but I guess we all agree that some kind of api (I only know this from eve, so forgive me if I don't use the correct terms) would be nice to have for those 3rd party software as you said

Link to comment
Share on other sites

English (Translator used)

 

Hello yamamushi,
I hope you are already clear that the LUA script operator in the game and close to the script item must be? So what for the phone app?

 

 

German (orginal)

 

Hallo yamamushi,

ich hoffe dir ist schon klar das der LUA-Script Betreiber im Spiel und nahe des Script-Items sein muß ? Also wofür die Phone-APP ?

 

mfG Die Waldfee

 

Link to comment
Share on other sites

45 minutes ago, huschhusch said:

English (Translator used)

 

Hello yamamushi,
I hope you are already clear that the LUA script operator in the game and close to the script item must be? So what for the phone app?

 

 

These are actually useful for tracking things like a facility designed to produce things and timings as well as price lookups on external sites. They could also be used to distribute loads to other external servers so that NQ doesn't actually have to run the script but another server can do so and the script can just grab the result. There are also other things but a script should also activate if a player is nearby so you could use them as warning systems as well

Link to comment
Share on other sites

In terms of security concerns, I think they'll have to design their own custom implementation regardless to close out the possibility of any exploits, so regardless there would be quite a bit of development time involved as it's a section of the program at high risk for exploitation.

 

In terms of DDoS concerns, even with a neutered set of operations I agree that you'd need some kind of a protocol set up between the web server end implementation and the clients with lua. I.e. you couldn't form connections with just any server out there, you'd have to install the API on the server you want to connect to as well, with some kind of a protocol (at least in spirit) similar to what was suggested in OP.

 

I do think it would be useful to have this, but I can also see it being a headache to implement and iron the bugs/exploits out of. It's the sort of thing I wouldn't deploy in the live version of the game, but rather either early in the testing phase or on a dedicated testing branch to reduce impact on at least the in-game economy.

Link to comment
Share on other sites

33 minutes ago, LurkNautili said:

In terms of security concerns, I think they'll have to design their own custom implementation regardless to close out the possibility of any exploits, so regardless there would be quite a bit of development time involved as it's a section of the program at high risk for exploitation.

 

In terms of DDoS concerns, even with a neutered set of operations I agree that you'd need some kind of a protocol set up between the web server end implementation and the clients with lua. I.e. you couldn't form connections with just any server out there, you'd have to install the API on the server you want to connect to as well, with some kind of a protocol (at least in spirit) similar to what was suggested in OP.

 

I do think it would be useful to have this, but I can also see it being a headache to implement and iron the bugs/exploits out of. It's the sort of thing I wouldn't deploy in the live version of the game, but rather either early in the testing phase or on a dedicated testing branch to reduce impact on at least the in-game economy.

 

how about just registering an external app or something with them as anything else would take far to much development time and still be vulnerable to all the same things. The limits already should limit the DDoS type things because its much easier to just ping the server with a false IP then work around them most of the time

Link to comment
Share on other sites

1 minute ago, ShadowLordAlpha said:

 

how about just registering an external app or something with them as anything else would take far to much development time and still be vulnerable to all the same things. The limits already should limit the DDoS type things because its much easier to just ping the server with a false IP then work around them most of the time

Well, we're talking more about incidental DDoS rather than intentional. You think writing a custom back-end and front-end for a fully fledged application would be easier than just writing a communications protocol and building a web frontend for it? I'm not sure I follow entirely

Link to comment
Share on other sites

Just now, LurkNautili said:

Well, we're talking more about incidental DDoS rather than intentional. You think writing a custom back-end and front-end for a fully fledged application would be easier than just writing a communications protocol and building a web frontend for it? I'm not sure I follow entirely

I was talking about a list of valid, registered and approved IPs that LUA applications could connect to so how in the world is that a whole front and back end?

Link to comment
Share on other sites

Just now, ShadowLordAlpha said:

I was talking about a list of valid, registered and approved IPs that LUA applications could connect to so how in the world is that a whole front and back end?

I see... The whole "registering an external app" part confused me.

 

I'm not sure if hard coding in a bunch of IPs would result in the kind of functionality OP is asking for.

 

For one, only people with access to servers on these IPs could realistically use the luasockets type system, right? Secondly, domains and IPs change, so you'd need to have it be more dynamic than that, imo.

Link to comment
Share on other sites

Just now, LurkNautili said:

I see... The whole "registering an external app" part confused me.

 

I'm not sure if hard coding in a bunch of IPs would result in the kind of functionality OP is asking for.

 

For one, only people with access to servers on these IPs could realistically use the luasockets type system, right? Secondly, domains and IPs change, so you'd need to have it be more dynamic than that, imo.

 

a registration system would not need to be hardcoded and you could update it as things change.

Link to comment
Share on other sites

Just now, ShadowLordAlpha said:

 

a registration system would not need to be hardcoded and you could update it as things change.

Right, a changing list, fair enough.

 

Still, would that not restrict the system excessively? And if it were possible to get yourself  onto that list, what does it change exactly?

 

It still doesn't address the type of incidental DDoS to those IPs, the type that JC mentioned in his tweet. However, OP's suggestion in principle would.

Link to comment
Share on other sites

7 minutes ago, LurkNautili said:

For one, only people with access to servers on these IPs could realistically use the luasockets type system, right? Secondly, domains and IPs change, so you'd need to have it be more dynamic than that, imo.

 

English (Translator used)

 

And there should be more than one recipient.

 

German (orginal)

 

und auserdem sollte es mehr als einen Empfänger geben.

 

mfG Die Waldfee

 

Link to comment
Share on other sites

1 minute ago, LurkNautili said:

Right, a changing list, fair enough.

 

Still, would that not restrict the system excessively? And if it were possible to get yourself  onto that list, what does it change exactly?

 

It still doesn't address the type of incidental DDoS to those IPs, the type that JC mentioned in his tweet. However, OP's suggestion in principle would.

 

the type of DDoS you are talking about is normal and in general is fine because its nature of more traffic than expected. Also he only mentions DDoS not the type. To get on the list would depend whatever they decide but would allow them to track who added what to the list or things like that. Also the system would be less restrictive than simply not having it in the first place

Link to comment
Share on other sites

Just now, ShadowLordAlpha said:

 

the type of DDoS you are talking about is normal and in general is fine because its nature of more traffic than expected. Also he only mentions DDoS not the type. To get on the list would depend whatever they decide but would allow them to track who added what to the list or things like that. Also the system would be less restrictive than simply not having it in the first place

Well, though that might reduce the volume of traffic that they'd have to monitor, they'd be able to monitor it even without having a whitelist of IPs or domains, since the traffic originates from within their program (barring hacking and such but that's a whole other can of worms that I won't get into now).

 

Based on the context of what JC said: "I'm concerned with potential DDoS attacks with dormant scripts inside popular ships..." -- I'd be inclined to believe he is talking about the incidental kind. This, for the simple fact that there are much easier ways to implement intentional DDoS attacks. And yes, I agree, this is type of traffic spike or other "more traffic than the webserver bargained for" situation is pretty common on the web regardless, but this is what I believe both my post and OP were addressing.

Link to comment
Share on other sites

The OP mentions intentional DDoS not incidental as well as abandoned projects but that is normal with scripts in general. They could monitor it ingame but you run into other problems like copy pasted scripts and other things like that

Link to comment
Share on other sites

1 minute ago, ShadowLordAlpha said:

The OP mentions intentional DDoS not incidental as well as abandoned projects but that is normal with scripts in general. They could monitor it ingame but you run into other problems like copy pasted scripts and other things like that

I'm not sure we're talking about the same things anymore, so let me take the liberty to better define some stuff here.

 

So you have the client Lua scripts (call them internet capable DU scripts or something like that maybe)whose developers want to access the internet with them. Then you have the servers online (called... internet DU script targets, maybe) that would be what they want to connect to with their scripts. The concern in terms of DDoS, if originating from within the scripts running in people's game clients, would be that the connections would be either entirely unsolicited (probably what you meant, I now realize) as part of a malicious Lua script developer's DDoSing campaign (kind of an unlikely scenario since there are easier ways to DDoS, imo -- also easily stopped), or as a result of poorly designed scripts that idle around pinging web servers (what I was initially thinking about) without any real need to do so. Or just the sheer popularity of a script overwhelming unconsenting server endpoints (maybe this is what JC was refering to with his million DU users specifier).

 

Defining a small list of IPs would serve to protect e.g. web servers that haven't been included in the list from being flooded with any type of traffic from within the client, but that would already be solved by a consent-based two-ended protocol type of deal that OP posited. Restricting access by IP would also have the downside of being less flexible without much added benefit imo.

 

Idk if I left anything important out, but I tried to be a bit more verbose to prevent further miscommunication (whereas usually I try to keep things terse for the sake of short attention spans, as nobody likes walls of text).

Link to comment
Share on other sites

3 minutes ago, LurkNautili said:

I'm not sure we're talking about the same things anymore, so let me take the liberty to better define some stuff here.

 

So you have the client Lua scripts (call them internet capable DU scripts or something like that maybe)whose developers want to access the internet with them. Then you have the servers online (called... internet DU script targets, maybe) that would be what they want to connect to with their scripts. The concern in terms of DDoS, if originating from within the scripts running in people's game clients, would be that the connections would be either entirely unsolicited (probably what you meant, I now realize) as part of a malicious Lua script developer's DDoSing campaign (kind of an unlikely scenario since there are easier ways to DDoS, imo -- also easily stopped), or as a result of poorly designed scripts that idle around pinging web servers (what I was initially thinking about) without any real need to do so. Or just the sheer popularity of a script overwhelming unconsenting server endpoints (maybe this is what JC was refering to with his million DU users specifier).

 

Defining a small list of IPs would serve to protect e.g. web servers that haven't been included in the list from being flooded with any type of traffic from within the client, but that would already be solved by a consent-based two-ended protocol type of deal that OP posited. Restricting access by IP would also have the downside of being less flexible without much added benefit imo.

 

Idk if I left anything important out, but I tried to be a bit more verbose to prevent further miscommunication (whereas usually I try to keep things terse for the sake of short attention spans, as nobody likes walls of text).

 

No developing a whole new protocol would be far more restrictive normally than simply allowing sockets and the list would not be to protect others but allow a few to be access. I don't know how you think developing something new would be less restrictive but whatever. Scripts could also send payloads or have the IPs they connect to hidden in which case any poorly designed script would be the fault of its creator and not a user.

Link to comment
Share on other sites

1 minute ago, ShadowLordAlpha said:

 

No developing a whole new protocol would be far more restrictive normally than simply allowing sockets and the list would not be to protect others but allow a few to be access. I don't know how you think developing something new would be less restrictive but whatever. Scripts could also send payloads or have the IPs they connect to hidden in which case any poorly designed script would be the fault of its creator and not a user.

It's less of an issue of attributing blame and more of an issue of avoiding the thing in the first place.

 

Having to be listening for connections explicitly and dropping connections that you're not expecting isn't a difficult thing to write. It's generous to even call it a protocol.

Link to comment
Share on other sites

Just now, LurkNautili said:

It's less of an issue of attributing blame and more of an issue of avoiding the thing in the first place.

 

Having to be listening for connections explicitly and dropping connections that you're not expecting isn't a difficult thing to write. It's generous to even call it a protocol.

 

no but listening and not accepting or accepting is the servers responsibility not the client. LUA would be a client so filtering based on something makes more sense

Link to comment
Share on other sites

I think you guys are overcomplicating this LuaSocket stuff quite a bit. 

 

There's no need to do filtering, and adding a whitelisted host list would necessitate patching that into the client, something that would, again, overcomplicate things.

 

The only data we'd be able to send upstream somewhere would be the data we're already working with in LUA, and giving us a simple json format to pack that data into would resolve many problems (if they even wanted to do that for us). 

 

I don't even think that's necessary, I'd rather just have the ability to open an outbound socket so that I can send whatever data I'm getting from LUA upstream into my own application to parse and render. 

 

None of this requires developing a whole new protocol, as it puts the onus on the application devs, not NQ. The only thing we'd want from the LUA side is some error handling to make sure that scripts keep working as intended even if they're not able to push data out a socket upstream somewhere for collection. Hence the handshake idea - does a host work? Great, start sending data. Does it not work? Great, skip publishing for this script.

 

NQ doesn't have to monitor outbound traffic, the scripts run client side, there's nothing for them to monitor there. 

Link to comment
Share on other sites

Just now, ShadowLordAlpha said:

 

no but listening and not accepting or accepting is the servers responsibility not the client. LUA would be a client so filtering based on something makes more sense

But what I'm getting as that e.g. Google couldn't get DDoS'd by DU scripts because they haven't installed a thing that listens for a connection. That way you wouldn't need to waste NQ resources on a screening process of IPs or domains, but rather users of the internet connecting scripts would be able to set up both ends of the connection themselves. This shifts responsibility to the users, which, if the subset of operations/features is kept nice and controlled and well-vetted, should get rid of both DDoS and security problems.

Link to comment
Share on other sites

1 minute ago, yamamushi said:

I don't even think that's necessary, I'd rather just have the ability to open an outbound socket so that I can send whatever data I'm getting from LUA upstream into my own application to parse and render. 

 

None of this requires developing a whole new protocol, as it puts the onus on the application devs, not NQ. The only thing we'd want from the LUA side is some error handling to make sure that scripts keep working as intended even if they're not able to push data out a socket upstream somewhere for collection. Hence the handshake idea - does a host work? Great, start sending data. Does it not work? Great, skip publishing for this script.

 

I'm not sure you're addressing JC's concerns there, though. If you can send traffic to any server you want, you can end up in the DDoS type of scenario he mentioned. Also, the script being client sided only limits things to a degree since presumably these same scripting functionalities could be used on scripts that propagate from player to player on constructs sold on the market or whatever.

 

Hence why I'd suggest adding some mechanism to restrict what you can send and in what amounts, e.g. lightweight, rate limited packet streams to attempt to start a connection, which a server that didn't want to accept potentially malicious script traffic could just ignore. I mean honestly well implemented web servers and the like probably ditch badly behaving traffic anyway, but I'm just mentioning this as a potential counter to JC's concerns.

Link to comment
Share on other sites

8 minutes ago, LurkNautili said:

 

I'm not sure you're addressing JC's concerns there, though. If you can send traffic to any server you want, you can end up in the DDoS type of scenario he mentioned. Also, the script being client sided only limits things to a degree since presumably these same scripting functionalities could be used on scripts that propagate from player to player on constructs sold on the market or whatever.

 

Hence why I'd suggest adding some mechanism to restrict what you can send and in what amounts, e.g. lightweight, rate limited packet streams to attempt to start a connection, which a server that didn't want to accept potentially malicious script traffic could just ignore. I mean honestly well implemented web servers and the like probably ditch badly behaving traffic anyway, but I'm just mentioning this as a potential counter to JC's concerns.

 

 

That's what the handshake is for. 

 

 

Link to comment
Share on other sites

14 minutes ago, LurkNautili said:

But what I'm getting as that e.g. Google couldn't get DDoS'd by DU scripts because they haven't installed a thing that listens for a connection. That way you wouldn't need to waste NQ resources on a screening process of IPs or domains, but rather users of the internet connecting scripts would be able to set up both ends of the connection themselves. This shifts responsibility to the users, which, if the subset of operations/features is kept nice and controlled and well-vetted, should get rid of both DDoS and security problems.

 

You don't need to install something to listen for a connection its there with a socket.

 

Just now, yamamushi said:

 

 

That's what the handshake is for. 

 

 

 

handshakes can be used to DDoS just like ping packets can be

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...