Published

Newscloud is dead

I’m sorry, but it didn’t work out. As of 4th August 2017 we have shut Newscloud down. We don’t keep any user data so there is nothing for users to migrate.

You can follow me on Twitter if you want updates on my next experiment or add your name to the launch list here

Background on Newscloud

The premise behind Newscloud was to give near real time updates and search across a million websites sorted by topic and theme.

At its peak, Newscloud was crawling 250k sites every hour or so and retrieving 250,000 stories a day.

We wanted to let a user choose a set of topics and get real time streams of news and articles from around the world.

Think of it like Google News meets Twitter in a tool like Google Reader. We built a mobile app, desktop app and the web platform but frankly due to freelancer problems the web app was the only one that was successful.

Newscloud was launched in 2015 as a side-project for Fernando Vega and myself. We probably hired 25-30 freelancers for various components, some worked out well, others less so.

Fernando had to pull out late 2016 due to work demands and I worked with a team of freelancers to try to get this into something we could bill for. After 9 months of frustration, culminating in a broken app that refused to come back to life I decided to pull the plug on Newscloud.

Newscloud had a lot of technical debt that was slowing us down. We just couldn’t fix things fast enough, lots of broken parts that really required a complete rewrite.

I will probably launch another version of Newscloud but it will be different, faster, lighter, more socially connected, more mobile and desktop and less web.

Tech Stack

Newscloud was built on MongoDB, SailsJS, Express, NodeJS, Sockets, Elasticsearch. Various prototyping efforts were attempted with Redis, Tensorflow, various machine learning methods.

It originally launched on Firebase but it nearly emptied our bank account overnight when a developer made a mistake and transferred a terabyte of data, (in the end we were crawling 5tb a month) we quickly realised that any sort of real customer usage would kill us (this was back in the early days of Firebase pre acquisition by Google).

Big thanks to some help from the guys at Softlayer and AWS for helping out with some cloud credits on their startup programs.

Every story was summarised using the Stanford NLP library, topic tagged using DBPedia and the sentiment was analysed using machine learning.

Real time streaming from Twitter picked up new stories and an algorithm decided what sites were trustworthy enough to include in the index based on social signals. Users could select dozens of stories and share this via their Twitter account. Users could also pull down their last 1000 Tweets and Newscloud would build you a topic list based on your interests.

If you think that sounds complex you would be right, it was way too complex, too many moving parts for a small team with no funding to maintain.

At last count, it had about 10 software repositories for all of the microservices needed to run it.

We had about 100,000 visitors to Newscloud over the 18 months it was live but to be frank we just didn’t see enough return visitors, those users that loved the product that came back every day.

A Few Lessons

If I had to do it again;

Be very careful about your freelancers and co-founders. It’s near impossible to be successful unless you get this right.

I would not use SailsJS

Wherever possible use native database drivers, ORMs are never as good as they claim.

Stick to common development frameworks and languages and avoid niche software, very difficult to get team members if you use something esoteric.

I would use more Redis and less MongoDB or even one of the new scalable databases from Azure, AWS or Google Cloud.

On the topic of databases, be very careful about your schema’s. Bad structuring of ours made it impossible to fix query performance problems without major rewrites.

I would probably use GraphQL to provide access to databases and provide an API for apps/website.

Probably build with Electron or React Native from the start (these can be converted to web apps relatively easily).

I would insist on one coding and connection standard across all the microservices (various freelancers doing their own thing resulted in 4 different ways to connect to Mongo which was a shitfight)

Documentation is critical, keeping it up to date even more so.

While we launched really early, we didn’t get close enough to our users and we did not run enough experiments with them to work out what they wanted.

There is a fine balance between your vision for what you want to build and understanding if your target market really cares enough about it to bother using or paying, so we would probably do more of that. (As I like to tell startups, no one ever went wrong talking to a customer).

Thanks to the users who actually liked it. If you want it back I’m open to discussion but it would have to be a very compelling use case.

You can follow me on Twitter if you want updates on my next experiment or add your name to the launch list here