Share this post

Link to post

Share on other sites

* a file dump or read-only api that is hosted separately from the game servers causes no concern over hacks or injection or denial of service to affect the game. Since its hosted separately, the machine is also not likely a vector past a game labs firewall.

The point that apis should be secured is a fair one, but is just part of being a software professional and doesn't deserve mention here unless we notice a mistake.

* making information about the state of the game world readily available CAN lead to unintended consequences in terms of advantages to players, new exploits, etc. That's what we're here to discuss in specifcs. And, it's one reason to start with a simple data export like this one instead of a functional API.

An example of the second case might have been revealing a player's location by including player shop contracts by name. But, that's not there so we're ok on that one.

Share this post

Link to post

Share on other sites

I could be wrong, but once npc port resources are eliminated, prices and quantities will be set by players. I am guessing that trade hubs will be set up. Players won't want to spread out across the map because not all ports can have guaranteed protection. [...]

I still don't think shop data should be published, but I don't think it will really matter. Once npcs are taken out of production, I think the economy will drastically change and this tool won't really affect it or give a major advantage. It will mostly show where the trading hubs are.

Good points. I agree that without NPC production the ability to make lots of money on small price spreads will diminish, especially if NPC shops don't buy anything either.

Share this post

Link to post

Share on other sites

Thanks for the data, makes things a lot easier for me, instead of relying on people to enter data into my site, I can use this.

Although, I'd prefer pure JSON as opposed to JSONP, as not everyone uses pure Javascript, and I dislike the need to sanitize the data before parsing (also, if I was being picky, its not even truly valid JSONP anyway).

JSONP:

Callback(
{
"Hello": "World"
}
);

Also, the UTF-8 issue is a bit of a deal breaker so far for me.

Once at least the UTF-8 issue is resolved, I'll integrate this into my tool (https://www.navalaction.tools), and make a public API available for all data.

In-case anyone is curious, I'm planning on creating a file hash from the json files, storing those in reddis and then on subsequent imports, creating a new hash and comparing to see if anything has changed before attempting a full blown import, and then using a differential algorithm to work out exactly what changed, so I have no need to perform a full loop of the data, although I'll have to create a benchmark to see the differences in time to see if the differential is necessary.

Share this post

Link to post

Share on other sites

Thanks for the data, makes things a lot easier for me, instead of relying on people to enter data into my site, I can use this.

Although, I'd prefer pure JSON as opposed to JSONP, as not everyone uses pure Javascript, and I dislike the need to sanitize the data before parsing (also, if I was being picky, its not even truly valid JSONP anyway).

JSONP:

Callback(
{
"Hello": "World"
}
);

Also, the UTF-8 issue is a bit of a deal breaker so far for me.

Once at least the UTF-8 issue is resolved, I'll integrate this into my tool (https://www.navalaction.tools), and make a public API available for all data.

In-case anyone is curious, I'm planning on creating a file hash from the json files, storing those in reddis and then on subsequent imports, creating a new hash and comparing to see if anything has changed before attempting a full blown import, and then using a differential algorithm to work out exactly what changed, so I have no need to perform a full loop of the data, although I'll have to create a benchmark to see the differences in time to see if the differential is necessary.

i'm having the same problem with UTF-8 and the JSONP (would also prefere JSON). Also it would be nice to have a timestamp to know when the data was created.

Share this post

Link to post

Share on other sites

It's a problem if you are using a language other than JavaScript. My JSON parser is written in C# and while it was able to parse JSON value expressions it was not able to parse JavaScript statements that assign JSON value expressions to variables.

Share this post

Link to post

Share on other sites

It's a problem if you are using a language other than JavaScript. My JSON parser is written in C# and while it was able to parse JSON value expressions it was not able to parse JavaScript statements that assign JSON value expressions to variables.

I use regexp to extract the json. Like

/\W+=\s+([{\[].*?[\]}]);/

Share this post

Link to post

Share on other sites

That would work if you read the entire JSON into an in-memory string before you start parsing any of it, but my parser reads directly from the incoming http stream to avoid needless use of memory. By the time I read the last few bytes of the http response, the bulk of the JSON has already been parsed. I suppose it's possible to apply regex scanning to a stream rather than an in-memory string but I wouldn't call it easy.

Of course this idea of using a regex may still be a useful tactic for others, even if it does not help me. I don't want to give the wrong impression that I think this is a bad idea, it just happens that it doesn't help in my particular situation.

Incidentally, I have already solved the problem for myself although I didn't go into details on how because I didn't think anyone would be interested. What I did was amend the expression parser to also recognize a limited class of statements (specifically "var identifier = json-value-expression" but I can add others such as callback function definitions).

Share this post

Link to post

Share on other sites

This is neither JSON nor JSONP. It is Javascript. Why they name the files .json is beyond me

JSON is data in a javascript object notation format.

JSONP is data passed to a callback function which is named according to a request query parameter (commonly called "callback" for simplicity). In reality, JSONP is essentially Javascript that has been twisted into a data-loading method to circumvent CORS.