Technologies

CoffeescriptMongodbPostgresSassSolrPythonDjangoAngular

PROJECT TEAM

Vladimir Sidorenko

Igor Tokarev

Pavel Volkov

Rostyslav Stekh

Mikhail Andreev

Online and offline media monitoring system

LookSMI is an advanced automated monitoring and analytics system. On a 24/7 basis, it provides users with the up-to-date information about their public reputation and the overall situation on the market. With LookSMI, companies can assess the quality of their PR campaigns, quickly discover the negative or adverse information (for example, during the crisis), run competitor intelligence and follow the latest industry news. This is the sole system on Ukrainian market that accommodates information from both traditional (newspapers, magazines, TV, radio) and the new (the Internet and social networks) media.

Apart from data collection and monitoring, the main LookSMI's gimmick is a convenient content visualization and analysis provided for the users.

The Problem and the Goal

The problem with media monitoring these days is the colossal volume of data originating from a huge amount of diverse media sources. Before, one had to manually collect different data streams, compile it using different software and then send it to the client 1-2 times a day. However, today information disseminating happens very fast, and if you care about your reputation, you must keep track of what's going on in real time.

Thus, LookSMI turned to Gearheart, setting the task to create the universal platform to collect and analyze information from all available open sources. Our task was:

to develop the system that would be capable to unify all media types in one format;

arrange information uploading from over 10.000 various media sources into one database;

create straightforward and user-friendly interface for the clients;

automate report compiling.

The client's requirement was to accomplish these tasks and launch system prototype as soon as in 1 month.

The Solution

To speed up system release to the market, Gearheart offered a solution that prioritized creating the interface for user's end. Back-end development and system's fine-tuning was attributed to the second phase.

Under these circumstances, LookSMI would at first fill the system manually, but the information on the user's side would be presented via convenient interface. This approach would allow LookSMI put their idea to the test right away and see how it works out.

The Execution

Design

To save time and budget, we offered LookSMI several ready-to-use design templates with the broad selection of ready-made visual modules. The client embraced the idea and chose the template that fully complied with his likings and requirements. Several visual themes are available to choose from.

For data visualization, we used ECharts.

Of course, today the interface of the system is optimized for mobile.

Technical side

At first, we tested Elastic Search to deal with search engine indexes. But we quickly saw that it's query language is not broad enough for the task. So we moved to Solr, which is used in the system ever since.

The main technical problem of the project was the search index load as LookSMI system is all about simultaneous reading and writing the vast volumes of data. It's simple enough to create a system that doesn't load servers too much by focusing either on writing or on reading. But creating a system that wouldn't bring servers down while performing intensive concurrent writing and reading operations is much more complicated task.

Nevertheless, after bringing in Solr into the system, thoroughly exploring the problem and fine-tuning the algorithms, we managed to find the right balance and walk the thin line between reading and writing, speed, and load. Thanks to this, now LookSMI exactly knows when they can get along by optimizing configurations or when they need to add servers.

We finished up the first development stage with creating the user interface – a dynamic single-page application on Angular JS. It requires sending less data and thus, reduces server load.

On the second stage, we automated the mechanism of filling the system with data by developing a unique programmatic solution. Our software constantly processes thousands of websites and captures new articles within just 15 minutes after they are published.

Eventually, to make the system work quickly and consistently, we created two databases:

The first database was meant to store all data from all media. We used MongoDB for that - it's ideal when you need to store huge volumes of data without doing something with it frequently.

The second database was intended to work with the users' portals and let users edit data. For that, we used PostgreSQL. The data that the client needs is imported to this database from Mongo – and then the user can work with it. He can also download the data/reports in Excel, PDF, HTML, ZIP.

Finally, we created custom on-site search engine based on Scrapy framework. It can handle complex search queries taking in consideration multiple sophisticated rules.

As a result, we developed a stable, secure, and fast system that can process huge volumes of data for many concurrent users. In total, creating the working prototype that could be sold to clients took 1 months – as was requested by LookSMI.

We constantly track new tools, solutions, and technologies that can come in handy, and implement it to improve the system performance.

Gearheartsupports LookSMI from 2012/2013. During this period, we made 3 system versions with redesigns and dozens of new features; the 4th version is on the final stage of development.

Results

We created media monitoring system that can process data from both online and offline sources. The system works with over 10.000 sources with daily increment around 300.000 inputs.

Thanks to minimalistic design and implemented technical solutions, LookSMI is very simple to navigate for users.

The major conclusion of Gearheart's LookSMI experience is that even 1 month could be enough to create a complex and comprehensive media monitoring system. The key to success here is choosing the right technology and knowing how to fine-tune it to perfection.

Testimonial

Vitalii Sydorenko,CEO, LookSMI

We're working with Gearheart for over 3 years now, and we're completely satisfied. One of the best things about them is that they don't just do what the client says them to do but get to the roots of the idea/problem and advise on how it can be done better conceptually or technically. They do the job with a view to the future – peak loads, new features/functions, etc. Hence, when you need to change or upgrade some part of the project, you don't have to change the basics.We are delighted that Gearheart's views on project development and design were very close to ours.