A bit over two months since I undertook the WorldTrends project, I'm happy to announce the completion of WorldTrends 1.0. For those unfamiliar with the project, it captures headlines and numerical indices for most major news sources, both niche and mass media, as well as financial market data and headlines. WorldTrends provides a web-based viewing platform for browsing this data, and also offers realtime tools to examine the data in cloud tagging views, charts, and graphs. WorldTrends 1.0 is based on a Web 1.0 model - server-generated images produced by PHP, with page refreshing to update data as it is collected by the server. In addition, I have begun WorldTrends 2.0, built on the Web 2.0 paradigm, to AJAXify and modernize the WorldTrends experience. In honor of the completion of WorldTrends 1.0, I'd like to take a look at some of the data it has uncovered in the past two months.

WorldTrends recorded 2,133,121 instances of 'iPhone' yet only 1,812,815 mentions of 'Vista', though the iPhone results cluster around a central maximum, while the Vista results are more evenly distributed [VistaiPhone]
'Iraq' appears 6,761,922 times, while 'Bush' appeared a comparable 5,238,684 times with much the same peak distribution [IraqBush]
'Democrat' showed up 1,550,929 times, while 'Republican' was only mentioned 440,659 times. [DemocratRepublican]
In a staggering imbalance, 'war' showed up 4,708,975 times, while a mere 386,911 instances of 'peace' were uncovered. [warpeace]

Without further ado, go check out WT1, drop in on WorldTrends 2 Beta and see what you think, and explore! Feel free to comment on other trends or comparisons you discover.

Are there any plans to distribute an API or source code for world trends? Is there any documentation available currently? Will you open this up for developer contributions? I'd love to help since I think this is a cool idea.

Thanks, and definitely! There is already a kinda-API, but no documentation on it yet. Currently it allows you to specify a time frame, source(s), keywords (if applicable), and number of data points to return. What else would you want?

I guess I'm looking mainly for more documentation-- I'm not sure exactly what I'm seeing in the output image result (ie. color code meaning, methodology, etc.) Also interested in how you are caching previous data results, or are you querying the data sources for historical data as well?
Thanks

What is WorldTrends?
WorldTrends Beta is an attempt to quantify the ebb and flow of activity in the world today, trends revealed in some part through the number of articles available at any given time in the Google News and Digg main RSS feeds. By summing the number of articles once per minute, taking ten readings at each minute mark to get a decent average for the randomly-generated feed, WT is able to gain a rough approximation of the activity in the world. More weight per article is given to topics with a large number of articles about that particular topic. You can view the results of this data culling and analysis using the tool you are currently browsing. Defaults have been set for you, but you can choose a complex (includes counts and timestamps) or simple (bars only) view, optimal colors to suit your personal preference, and the time increment the graph should display. Larger time increments tend to show more reliable trends, but come at the price of less data. Some views, such as monthly and yearly, many not yet be available due to the paucity of data collected thus far. WT Beta is still under development; new features that may be added soon include more in-depth statistics and analysis as well as a larger collection of graph views to choose from.

Reading the Charts
The Composite, Google, Digg, and Slashdot news sources reflect world activity as the height of the graph. Higher bars (or points on the line graph closer to the top) reflect a larger measure of activity on the associated news sources. For the three market indices, the opposite is true: the lower the values reach, the more dire the situation, while tall bars or values near the maximum indicate a more strongly positive outlook.

It reads RSS feeds and also parse s pages like finance.google.com to derive its data; the resulting data is stored in a set of MySQL tables that are used to generate all feeds, pages, and images. Because it caches all data, it doesn't have to recontact the original sources every time a person requests data.

Have your own thoughts to add to this or any other topic? Want to ask a question, offer a suggestion, share your own programs and projects, upload a file to the file archives, get help with calculator and computer programming, or simply chat with like-minded coders and tech and calculator enthusiasts via the site-wide AJAX SAX widget? Registration for a free Cemetech account only takes a minute.