Originally by:Meissa AnunthielMy argument is, in essence, that longer trail still requires manual checking, shorter trail does away with that need. The barrier between people without those tools and those with it will increase the shorter the trail

Well, ignoring the fact that the market history is much less useful for finding "trade opportunities" than order data is...

The problem here is that this is not about "giving them the data" vs. "not giving them the data." The data is already available. It takes about 4h to do a full market dump for a region, and having 3 accounts to do that during your work day isn't particularly difficult. Come home, work with current data.

You are, in essence, arguing that the advantage of this data should be available only to a few. This move from CCP makes that data available to more people. Making the data readily available will also make it easier to incorporate into websites that help everyone (e.g. eve-central).

Giving out this data in a more structured form helps newer (or "less focused") players, as opposed to harm them.

(Oh, and just to make this clear, I'm talking about daily data, not sub-daily data. If you go below 1-day resolution, you start favoring people who can play at any time of the day as opposed to people who happen to have to go to work and such.)

Posted - 2011.08.31 11:32:00 -
[152]
I would like to make it possible to download only part of the data : we have to choose the item and the range of the data to download.ex : tritanium, from 1 may 2010 to 26 june 2011.like this, it would be possible to analyse more simply in spreadsheets.Here, the files is just too big... My version of open office open just the first million of line in .CSV and nothing in .SQL!

Originally by:Meissa AnunthielMy argument is, in essence, that longer trail still requires manual checking, shorter trail does away with that need. The barrier between people without those tools and those with it will increase the shorter the trail

Well, ignoring the fact that the market history is much less useful for finding "trade opportunities" than order data is...

The problem here is that this is not about "giving them the data" vs. "not giving them the data." The data is already available. It takes about 4h to do a full market dump for a region, and having 3 accounts to do that during your work day isn't particularly difficult. Come home, work with current data.

You are, in essence, arguing that the advantage of this data should be available only to a few. This move from CCP makes that data available to more people. Making the data readily available will also make it easier to incorporate into websites that help everyone (e.g. eve-central).

Giving out this data in a more structured form helps newer (or "less focused") players, as opposed to harm them.

(Oh, and just to make this clear, I'm talking about daily data, not sub-daily data. If you go below 1-day resolution, you start favoring people who can play at any time of the day as opposed to people who happen to have to go to work and such.)

It takes you 4 hours to dump the data for a region, you have tools to parse the cache, the more casual traders don't. I know you already played with manufacturing and market many years ago, remember how you did things then, and realize that the less devoted and/or newer players do it like that now. That play style needs to be preserved as well, and if something can foster the other ones without needlessly harming the casual players, then absolutely.

I'm arguing that the advantage provided by this data will be available to a certain group (they're not few, they're just the dedicated ones), not that it should. Eve Central is regularly unreliable, they provide indications that need to be investigated, just like I argue this dump should.

Strictly speaking, my position would require that no dump exist, but the added value is too great to pass up on, now I'm arguing for some mitigation. We can start with 3 days easily, see what comes out of that, see the impact it has on the smaller traders, and based on that lower it further. It's easier to go from 7 days to 3 to 1 to sub-1 than it is to revert from sub-1 to 7. My point is caution...

As an individual, I'd rather have instant dump of anything I request because I can develop anything needed to take advantage of it, but I'm arguing for the numerous less dedicated players (who most likely won't even post here).

Edit:And if you wonder why I spend time arguing against my own preference, it's because it's my job as member of the CSM... I really understand and agree with your arguments.-----Member of CSM 2, 3, 4, 5 and 6. My blog

Originally by:CCP StillmanI'd personally love to provide this sort of data live because it satisfies my interest in the data. But it's not feasible because of the severe performance impact it would have.

In the real-world, I'm actually a high-level technical expert for a major data consulting firm, so I have experience drawing reporting and migration dataloads off of high-activity-volume multi-terabyte systems. Depending on the backend architecture you're using for your live data, there may be a way to get the near-live (or at least sub-60-seconds) feed you're looking for. Whether or not you're looking to publish a feed at that resolution, I imagine it might help out CCP's internal analysis by yourself and CCP Dr.EyjoG.

If you're interested in having a feasibility conversation on using some of the latest techniques available to get that kind of information in that kind of time, send me a message on EVEGate and we'll take that discussion offline.

Originally by:Meissa AnunthielAs an individual, I'd rather have instant dump of anything I request because I can develop anything needed to take advantage of it, but I'm arguing for the numerous less dedicated players (who most likely won't even post here).

I am not arguing for myself.

I already have this data. I do not need it.

I find it bad game design that the ability to write webpages that browse the market and programs that read cache files (my thanks to Entity here) and upload them to server gives me a tangible in-game advantage.

Hence, I would like to mitigate the advantage I can get from being able to use all this arcane stuff and have CCP provide the data I can get anyhow. So that it's not just the geeks with no life get to actually benefit from this data.

1-day old data is great. 2-day old data, ok, too. Older than that and the values become slowly useless for our use and we'll just keep using our existing toolchain - meaning we "dedicated users" keep having an advantage over the "less dedicated" user, instead of losing that advantage.

Quote:I know you already played with manufacturing and market many years ago, remember how you did things then

We used to use the QUANT Corp 7-day index (QTC-7), which was generated by some poor sod flying around the three Minmatar regions every day and typing down market values by hand. I think he later switched to the "market export" button.

When that poor sod burned out from that work, we switched to EVE Central, who were (and are) uploading order data from the "export" in-game button. Later still we switched to EVE Metrics until they shut down, and then back to EVE Central. As EVE Central is quite unreliable and does not offer market history data, only order data, we looked for a replacement. eve-marketdata.com looked good, but we got tired of being dependent on a 3rd party site, so we implemented our own cache reader and uploader instead.

Originally by:Meissa AnunthielProvide the exact same information in game without a person having to travel between regions and I'd agree. Requiring people to implement databases, API calls and other such things is unreasonable. Personally I have no issue with it, I'm a developper, but not everyone is, and those people matter too...

This API/export differs from other APIs in this that it provides information that isn't readily available to people in-game, that's a big difference.

Being a programmer and market dude myself, I can think of numerous ways to gain an edge off this data. But then again, as pointed out, there's already ways of scraping very similar data in-game, though less efficiently.

So I'd be very interested in hearing any arguments against provide data every, say 24 hours. Would that ruin it for the small guys, who can't do these sorts of tools?

If you release this data, you will be ruining someone's (EVE) life somewhere and sometime. This is REGARDLESS of the time delay involved,

That being said, the more timely this data is, the more useful it is. Knowledge is power, and this sort of data will help grease the wheels of economy. Yes, it may put the squeeze on profits, but it may also help orders to fill faster.

Anyways, the requirement to use third-party tools is already pretty much standard from the industrial aspect, and I bet that most traders use third-party tools already anyways. I imagine that it won't be too long after you make this data available that developers will include the option of using this data in their tools.----------

Originally by:Dr Fighter"how do you know when youve had a repro accident"

Posted - 2011.08.31 13:33:00 -
[159]
Is the averagePrice calculated correctly by weighting each transaction price according to quantity?________________________CCP: Where fixing bugs is a luxury, not an obligation.

Originally by:Meissa AnunthielProvide the exact same information in game without a person having to travel between regions and I'd agree. Requiring people to implement databases, API calls and other such things is unreasonable. Personally I have no issue with it, I'm a developper, but not everyone is, and those people matter too...

This API/export differs from other APIs in this that it provides information that isn't readily available to people in-game, that's a big difference.

Meissa I agree with your concerns. EVE is sometimes called "spreadsheets online." Its clear many people in this thread are fine with that, but I am not sure it is a great selling point for most. Its definitely not for me.

I don't want to be forced to run multiple out of game apps or analyze a bunch of out of game spread sheets in order to be competitive in eve. Thats where this is heading.

Give information in game and in english.-Cearain

Make fw pvp not pvehttp://www.eveonline.com/ingameboard.asp?a=topic&threadID=1329906&page=1

Originally by:Meissa AnunthielMy argument is, in essence, that longer trail still requires manual checking, shorter trail does away with that need. The barrier between people without those tools and those with it will increase the shorter the trail

Well, ignoring the fact that the market history is much less useful for finding "trade opportunities" than order data is...

The problem here is that this is not about "giving them the data" vs. "not giving them the data." The data is already available. It takes about 4h to do a full market dump for a region, and having 3 accounts to do that during your work day isn't particularly difficult. Come home, work with current data.

You are, in essence, arguing that the advantage of this data should be available only to a few. This move from CCP makes that data available to more people. Making the data readily available will also make it easier to incorporate into websites that help everyone (e.g. eve-central).

Giving out this data in a more structured form helps newer (or "less focused") players, as opposed to harm them.

(Oh, and just to make this clear, I'm talking about daily data, not sub-daily data. If you go below 1-day resolution, you start favoring people who can play at any time of the day as opposed to people who happen to have to go to work and such.)

It takes you 4 hours to dump the data for a region, you have tools to parse the cache, the more casual traders don't. I know you already played with manufacturing and market many years ago, remember how you did things then, and realize that the less devoted and/or newer players do it like that now. That play style needs to be preserved as well, and if something can foster the other ones without needlessly harming the casual players, then absolutely.

I'm arguing that the advantage provided by this data will be available to a certain group (they're not few, they're just the dedicated ones), not that it should. Eve Central is regularly unreliable, they provide indications that need to be investigated, just like I argue this dump should.

Strictly speaking, my position would require that no dump exist, but the added value is too great to pass up on, now I'm arguing for some mitigation. We can start with 3 days easily, see what comes out of that, see the impact it has on the smaller traders, and based on that lower it further. It's easier to go from 7 days to 3 to 1 to sub-1 than it is to revert from sub-1 to 7. My point is caution...

As an individual, I'd rather have instant dump of anything I request because I can develop anything needed to take advantage of it, but I'm arguing for the numerous less dedicated players (who most likely won't even post here).

Edit:And if you wonder why I spend time arguing against my own preference, it's because it's my job as member of the CSM... I really understand and agree with your arguments.

You keep saying that the smaller trader either won't be dedicated enough to mine the data or won't have the technical know how. You haven't taken into account in your counter argument the creation of 3rd party software to help the less dedicated and non-technical players. I can assure you that someone will create tools to help players mine this data. Therefore I propose we stick with the 1d resolution ;-)

Originally by:Dierdra VaalAs such, is it possible that this will be upgraded to an API call? Where, similar to eve-central, I can query one or more itemIds (as well as location parameters, time parameters, etc) and get their up-to-date price statistics back?

Originally by:Arkady SadikHave you considered providing an API interface instead of file dumps?

It's no coincidence that the API team has been working with Dr. EyjoG and his team on this. Our ultimate goal is to expose this through an API with more up to date data. This release is to gather feedback and find out how you guys can best benefit from a such potential API.

Definitely csv until it gets worked out with the new API key going live.

Posted - 2011.08.31 15:48:00 -
[163]On data delay:I don't believe up to date data will only benefit hardcore traders since easy to access 3rd party tools will spring up quickly. However, the point that up to date prices may overly benefit players with a lot of play time on their hands is well made. With a 24h delay these players could still gain their deserved advantage using in-game checked eve-central data correlated with the delayed CCP data, without leaving less hardcore traders too far behind. With this in mind, a 24h delay may well be the perfect compromise.

Also remember that since CCP data does not include outstanding buy and sell orders, they good-deal-between-regions trader play style is more or less preserved as is.

A 24h delay would be good enough to spot larger market trends as they happen. 24h is also perfectly suitable for item price checks and killboards. However, already at a 48h delay, the value for these uses are greatly diminished.

On data formats and delivery:A daily csv dump is fine for market data mining tools and trading desktop applications. However, for mobile applications, a 1mb load over a shaky 3G or edge connection may take a very very long time, even 100kb will introduce a large delay. And to use price data directly in interfaces, a quick API call per item (or several items) is needed. If API server spamming is an issue, cashing should help and even rate limiting calls to the API could work depending on implementation (say maximum 10 calls per whole minute with maximum 100 total items requested, this would allow for priced item listings without enabling to load prices for "all items" via the API).

Posted - 2011.08.31 17:34:00 -
[165]Edited by: Barakach on 31/08/2011 18:04:35Edited by: Barakach on 31/08/2011 17:37:39Is it possible to get this data also grouped on buy/sell? I'm sure the average buy and quite different from the average sell.

Side note: Do people not realize that you can get MS SQL Express for free? SQL is very powerful.

Edit: could we possibly get a Standard Deviation with that? I would love to be able to recreate a bell curve showing the relationship to price and volume.

Originally by:Arkady SadikParsing CSV (especially as this dump primarily contains numbers only, no complex strings) is trivial and fault-free in comparison.

Originally by:CCP StillmanOn the format discussion, I want to propose a suggestion. It might be silly, but hear me out:

CSV is a very simple format. I know that at least the database engine I work with on a daily basis(MSSQL), can import a CSV file into a table in about a small line of SQL. And I'd be surprised if other database engines couldn't.

Until you happen to use a locale where "CSV" is quite different. I work for an international company and have the "pleasure" to deal with exchanged CSV files. You know, in Germany in a "CSV" file, columns are actually delimited by ";" due to the fact that the comma is our decimal separator. Now try making sense out of a "real" CSV file.

More people will be able to use CSV (MS Excel and Google Spreadsheet both know how to use it) whereas the SQL dump many people won't know. Now, the people who could use the SQL dump very well will also be knowledgeable enough to import the CSV into their own SQL (or otherwise) DB.

Originally by:Hel O'WeenUntil you happen to use a locale where "CSV" is quite different. I work for an international company and have the "pleasure" to deal with exchanged CSV files. You know, in Germany in a "CSV" file, columns are actually delimited by ";" due to the fact that the comma is our decimal separator. Now try making sense out of a "real" CSV file.

Not exactly true. This is only an annoyance when you have to work with a lot of undocumented CSV files in different formats. If you know what you can expect it isn't very difficult to make sense of it. Nor an annoyance to work with. And I just do not imagine CCP will change the format every time they make a dump ---

Originally by:Hel O'WeenAs I said: for a text-based format almost anything is better than CSV.

I'd say SQL is actually worse, considering the differences between the various SQL dialects. Converting a nice CAST(0x96040474 AS SmallDateTime) to a DATETIME value that other SQL servers can understand takes quite some gymnastics (that's from an MS SQL export). And it's just a very obvious extreme example.

Telling your csv library to use "," as a column separator (usually the default) and "\n" as a row separator (usually the default), with no escpaing necessary, is trivial in comparison, really.

(Again, as soon as you have commas or newlines in your records, you really want something else than csv, but this is not the case here. So pretty much any halfway sensible format works with very little overhead, compared to the .BAK file.)

Originally by:Arkady SadikParsing CSV (especially as this dump primarily contains numbers only, no complex strings) is trivial and fault-free in comparison.

Originally by:CCP StillmanOn the format discussion, I want to propose a suggestion. It might be silly, but hear me out:

CSV is a very simple format. I know that at least the database engine I work with on a daily basis(MSSQL), can import a CSV file into a table in about a small line of SQL. And I'd be surprised if other database engines couldn't.

Until you happen to use a locale where "CSV" is quite different. I work for an international company and have the "pleasure" to deal with exchanged CSV files. You know, in Germany in a "CSV" file, columns are actually delimited by ";" due to the fact that the comma is our decimal separator. Now try making sense out of a "real" CSV file.

As I said: for a text-based format almost anything is better than CSV.

You do know that databases that do support importing CSV files also support specifying custom delimiters, line endings, quotation and escape characters right?

Posted - 2011.08.31 21:07:00 -
[174]
I suspect this has already been captured by someone else already, but it is worth reiterating. Please can we have Avg./Median price differentiated by BUY and SELL orders seperately. Thanks.

Oh, and another thing, I think the optimum delay is something around 5 working days. Why? Well we don't want the market to be too efficient, as otherwise, as any good economist will tell you, prices will (in theory) collapse to the base cost value. If that happens, then no-one will be making any money. I suspect for 95% of items, prices won't move very much in 5 days anyway. For the remaining 5%, that's what EVE-Central is for.

Posted - 2011.08.31 21:39:00 -
[175]
I suspect buy/sell cannot be differentiated here because of the internal implementation (the transaction is always two orders meeting, the outstanding one and the immediate one, so that's a Buy and a Sell at the same time, afaik), so we won't have such kind of details.

As for the data format: CSV for the dump as it's broadly compatible. I'm not sure about the necessity of the API calls, because if we're talking about the delayed data (a week, a month or so), then we're not really interested in the latest two days of the data, but the whole data chunk. This is not much useful for short-term planning as had been already mentioned. In my opinion, it's all about pinpointing mid/large economic cycles or something analogous to real-world economy.

On the other hand, maybe it's possible to implement these data collection routines DT-time and collect the aggregate of the finished "day"? I don't really know about the perfomance cost here, but it seems justified - there's no transactions going on, the previous financial day is closed, so why not spend two minutes on gathering the data? That way the data is really actual, and if we jack in the API here, we'll get a pretty distilled version of Awesome.

CCP hasn't said anything more aside from the "The $99 `is coming` for using our information to protect our IP" yet, the developers go to the players for useful information? Sounds to me like they're trying to put out more and more tools (Useful or not) to make that cost seem more worth while. I'm not paying $99 when I ask for nothing back, or, a small in-game currency donation.

Don't you guys have accountants that sit in development meetings to see how much more can be milked out of your players? Couldn't you just take them aside and say "Hey, what would you find useful in a historical price dump?" then shove it down our throats like pretty much everything else that has been dumped into this game over the past year or so COMPLETELY unannounced?

The data you get from eve-central alone, including the archived data, can be derived into what CCP is "willing" to offer, not to mention E-Cs data is more "real time" and has a hell of a lot more history, however not 100% accurate and up to date on all infrequently used items since its user supplied data, but I'm sure there are regular updates to at least the basic ores. The data from E-C can be massaged into different manners to suit what is required for the job.

The *ONLY* issue I have with E-Cs data, and its not their fault really, is that you're never told when 100% of an order is sold out. Aside from that, I cannot fathom how this additional tool from CCP is going to be any better than what is already derived out there.

*DEV*Even with the above mentioned, I'm still going to give my dev advice.

As for the commentary about CSVs... International or otherwise, since you're working purely with numbers, and two delimeters, if you're space and download size conscious, you can work the data into 4-bit fields and still have room for delimiters. But, since nothing will easily read a 4-bit character file, 8-bit characters will do, its standard, and its been around since the first 8-bit computers were made. Forget uni-code, forget double-character notation, deal with CHR(0) to CHR(255), one byte per character, and you eliminate problems converting to and from chinese, russian, english, french, spanish and every other different character type notation.

Stay away from using SQL structured language as the primary dump source. Any recent DBMS will easily read in the delimited file, and drop it into a table. Hell, even regular programming languages can deal with the file, line by line, split up the data due to the delimiters, and process the data accordingly. There shouldn't be any question about this.

Posted - 2011.09.01 02:49:00 -
[177]
"I'd say SQL is actually worse, considering the differences between the various SQL dialects. Converting a nice CAST(0x96040474 AS SmallDateTime) to a DATETIME value that other SQL servers can understand takes quite some gymnastics (that's from an MS SQL export). And it's just a very obvious extreme example."

Of course casting raw binary data into a value type that is stored differently amoung SQL implementations will give different values for different implementations.

Posted - 2011.09.01 02:52:00 -
[178]
"I suspect buy/sell cannot be differentiated here because of the internal implementation (the transaction is always two orders meeting, the outstanding one and the immediate one, so that's a Buy and a Sell at the same time, afaik), so we won't have such kind of details."

Would be sad if they can't even track info that simple about an order.

Posted - 2011.09.01 03:28:00 -
[179]
Just a few things here....first and foremost...this would have helped me a lot more several years ago...but, it's about time ;)

Second, XML is the way to go on this. Not CSV nor SQL. Standard XML rowsets would be fine and dandy.

Third, API calls, yes, please. Previous day's generated during downtime....all others available on a per call basis (they should be cached results, so cheap and easy...) pass in a date, and optionally a region. Probably the best would be another API call, that you could pass in an itemID, date range and optionally a regionID, to obtain only the items you are interested in....just brainstorming here...----------

COPYRIGHT NOTICEEVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. CCP hf. has granted permission to EVE-Search.com to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Search.com. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this website.