Category: Meldbot

In July of 2013 I did some business related research regarding the type of usage made in Zapier. The research involved scraping the list of zaps that’s publicly available on the Zapier website, dumping them into a database, and fetching a few queries with different intersections on the dataset. The business I was planning at the time is now acient history, so I thought I’d share my results with the world, in case anyone might find it useful.

Imagine a scenario where you launch a new project management tool, and it’s really the best one out there, but no one wants to use it. This could happen for several reasons, one of which would be the lack of integrations with other types of software your potential users are using. They might need integration with Google Calendar, or with a variety of CRMs, or ticketing systems, or bug tracking systems, or version control services, and so on and so forth. Eventually, you’re forced to start providing built-in integrations with any and every other tool out there. With each new integration offered, the potential user base of your app grows bigger, which is good, but this scenario has it’s disadvantages:

There are infinite potential integrations, so the work is unending.

This diverts the attention of your development staff from what they do well (project management, in the case described above) to other types of work.

Bugs are usually hard to control because you’re working against systems that you don’t host, and whose code you can’t see.

The need for integration tools aimed at software vendors derives from the scenario described just now, which I like to call the integration pitfall. Application providers are faced with pressure, having to offer built-in integrations with a growing subset of related products, so that their own product would be useful to new target audiences. Some tools were developed with the aim of assisting these software providers, instead of targeting their end users directly.

Elastic.io is one such tool. It lets you connect with a variety of third party services and map fields from these data sources to your own application with a simple drag and drop interface. In sum, Elastic.io is meant for non-technical users that want to extend the application that they offer with built-in integrations. A very different approach is that of Nectil, a French service provider that developed their own software for web application integrations, and that offer small businesses tailored solutions on top of that framework.

Most other tools aim at the developer crowd, trying to help them consume APIs with greater ease and speed. Temboo provides SDKs for server side API consumption. It supports a very extensive amount of APIs and provides access to them through a unified interface. The service is SaaS based, though cheap, and meant for low levels of consumption (anything really big would require custom pricing). Webshell is an API that lets you integrate different APIs from third party services and create new functionality. It’s a JS library, which handles authentication for you and lets you construct your custom APIs using an editor. DERI Pipes is a software that lets you create automated processes that transform and mash together web content (much like Yahoo! Pipes, but more technically complex).

Cumula is a PHP application framework that takes the idea even further. Based on the assumption that modern web applications no longer sit on top of a single on-premise database, but instead connect with a multiplicity of services, they’ve contrived a framework that lets developers build applications with a unified interface for local and remote data sources. It’s modular, with the use of components for describing parts of the application. Each component has its own routing and templating. A collection of DataStore and DataService packages are available as dependencies, included in the same Github repo. As of now, Github’s popularity metrics indicate very low acceptance rates, and the same goes for the associated Google Group. Nevertheless, this is a relatively new project, and they’ve already received some media coverage.

Other platforms aim to help developers find and consume APIs, without altering the way they work or requiring the utilization of a specific framework. APIHub (renamed to the anypoint portal after being acquired by Mulesoft) and the ProgrammableWeb projects are API explorers, within which one can quickly search and learn about new APIs. Mashape does the same, although focused more on developer APIs and less on connecting with third party applications. Mashape acts as a middleman between providers and consumers, relieving API consumers from having to maintain separate authentication processes with a multiplicity of remote services, and thus enabling the proliferation of usage of such services.

Edit: Since the time of writing this, a new contestant has emerged, called CloudElements, that provides a uniform API for accessing multiple cloud based applications.

Process automation tools, meant for either small businesses and consumers, or for Enterprises, allow non-programmers to combine services from different vendors into a new piece of functionality. Enterprise and consumer tools differ from one another in the level of complexity and technical know-how that they require. Most tools meant for consumers can be configured with no more than a few simple steps. They allow people that don’t write code to define automated behaviors between cloud-based services, such as sending yourself an email when this or that happens, adding a new contact to your mailing list or CRM when a form is submitted somewhere, or sending new content directly to your Dropbox folder. These can be regarded as productivity micro-tools, and are very limited in the scope of possibilities that they offer.

Zapier (YCombinator alumni) is a prominent tool, meant for very small businesses and consumers, which offers an extensive list of third party services to integrate with. IFTTT (a Betaworks profile company) is another prominent contestant, meant mostly for consumers and free of charge. Cloudwork (sister company of GetApp) covers the same space as well, and are working on a set of SDKs, and a visual editor, to allow anyone to create new integration types. Yahoo! Pipes is a tool to aggregate, manipulate and mash up feeds from third party services, that offers lots of useful functions for processing the data and is all assembled visually. Other alternatives in this sphere include RunMyProcess, Foxweave & WeWiredWeb.

BPM software for Enterprise customers are the other end of the spectrum. They normally provide form designers, visual process editors, rule engines and customizable dashboards for monitoring workflows. These solutions are relatively complex to operate, and usually require a learning period to adapt to. Their pricing levels are clearly meant for the Enterprise world. Competitors in this sphere include Appian, Skelta, Rollbase, Iceberg and ProcessMaker, to name a few.

The tools surveyed here offer an integration of data into a single platform, but do so only in one direction. By this, I mean that these solutions don’t offer the possibility of sending updates back to the applications from which the data originally came from. One such tool is Sumall, which offers a marketing-focused set of metrics based on third party data, consumed via APIs. A competitor in the same vector is Geckoboard, offering an API itself for end users to add their own services on top of those provided out of the box.

A slightly more configurable tool is a Chartio, which connects with what are mostly DBaaS providers, to allow one to create his own customizable business dashboards. Chartio’s platform allows non-technical end users to author their own graphs by dragging and dropping fields from the third party data sources they’re connected with. Combined with that, is a rules engine, based on a set of dropdown menus, for setting up the logic behind the presented graphs. A more technical option is that presented by Dashku, offering a platform on top of which frontend developers can build widgets that connect with third party data, and that combined, are gathered into a fully self-authored dashboard.

Every BI software provider offers some level of dashboard building capabilities. Notwithstanding, these solutions are usually out of reach for small businesses. Jaspersoft, for instance, offers a community edition of its solution as an open source project. However, that open source version lacks the dashboard capabilities that paying users are offered.

A subset of dashboard providers is that of analytics solutions. These are less likely to integrate data from different sources and are therefore of relatively minor significance for this overview. In spite of that, some tools are still worth noting. The first of which is Chartbeat, which gathers data from the users’ websites, but integrates with social media data as well, to generate a more insightful perspective, all without requiring the end users themselves to do the heavy lifting data work. Another tool takes that approach slightly further – ThinkUp pulls together data from users’ social networks, but doesn’t offer graphs and dashboards at all. Instead of pouring immense amounts of data onto the screen, ThinkUp presents a feed, much like that of most social networks, that highlights actionable insights based on the sum of the data that was gathered.

Integration for SMBs

OneSaas is a data integration application that automates common tasks for specific business users, mostly in eCommerce and online marketing. It features a variety of integrated services and a simple interface to connect to these third party user accounts. Then, there are a few options for each type of data, mainly, whether or not to add and delete items according to changes in other systems that handle the same data type – Magento with eBay, Google Contacts with a CRM, etc. It also has an interface that displays all the data from the third party vendors in tables.

Cazoomi SyncApps is an integration cloud meant for small businesses. It features a visual studio for composing integrations, much like Enterprise solutions do, but are aimed at SMBs, both with regard to pricing and available integrations. Cazoomi’s website reports them having 10,000 customers, and a team of 13 members (at the time of writing). It appears that Cazoomi’s solution is meant more towards application vendors that want to create integrations than it is for end users themselves.
Another type of tools are web page scrapers meant to ease extraction of data from websites without the use of APIs that sometimes aren’t available. The prominent tool in this respect is Dapper, and a newer contestant is import.io.

UPDATE: Since this text was written, a new service came out that scrapes websites automatically, and let’s you build apps out of the data it extracts, called Kimono

Industry Specific Integration Tools

Each industry has an array of specific tools, either meant for integrating data, or that integrate data as a byproduct of their main function (dashboards, process automation, etc.). These solutions are thus somewhat parallel in their function to data integration software, and could suffice in any scenario in which integration needs are within this specific set of third party applications. However, a common scenario among small businesses is an overdependence on one such specific tool, and an inability to make that tool work with services meant for other aspects of running a business. For instance, a business in eCommerce might utilize an application that optimizes and automates some of the aspects of selling online. Such a tool would surely connect seamlessly with any and every marketplace, comparison shopping engine, shopping cart software, payment gateway and shipping service. However, the ability to make this tool work with a newsletter management system, or a PPC ad platform, is of limited capacity.

Enterprise Integration Tools

The Enterprise world covers a very wide array of solutions at this point in time. As part of this overview, only SaaS products meant at Enterprise customers will be examined and not hybrid and on-premise solutions. A key character of the following solutions is that they’re meant for IT departments to deploy, and not for the end users themselves.

Informatica Cloud is a high end service for Enterprise clients that offers a large set of tools for integrations and data processing. Jitterbit offers an open source platform for the creation and management of integrations, both legacy and SOA based. They make use of reusable templates for common integration needs so that developers have less work. Extol offers a design tool for building integrations between Enterprise applications. JackBe Presto is a visualization and BI tool that connects to external sources – data warehouses, spreadsheets, feeds, existing BI systems, etc., using a graphical assembly tool. SnapLogic is an Enterprise-grade solution for data integration that makes use of modular pieces of connectivity or functionality called ‘snaps’. Snaps can connect to third party services, local files or anything else, and can provide atomic bits of functionality used for ETL tasks. They have a marketplace for snaps with high end pricing. Mulesoft offers a set of products meant for data integration. Their visual editing tools are meant to ease the work of developers, not to provide a EUP solution. It offers a variety of pre-built connectors and integration apps, the ability to develop new ones, and visual tools for integrating services and for building data transformation processes. Dell Boomi is an Enterprise API integration tool that uses a flowchart interface and offers both pre-built integration types as well as the ability to create new ones or have Dell do the customization for the end users. Scribesoft offers the same standard set of features, as well as the ability to connect SaaS products with on premise software. Other solutions in this sphere include IBM Infosphere DataStage, IBM CastIron, and Talend’s Integration Suite.

These platforms are meant to allow non-technical end users to define data structures, according to their specific needs, and then make use of an online database that stores data according to those definitions. They usually also allow one to build business apps, which are small productivity apps for web or mobile, meant to ease tasks such as data entry and reporting.

The challenge they all face is the technical complexities involved with data modeling, which in this case is left to non-technical end users. The traditional way of solving this is the route taken by MS Access, that is, leaving it to the end user to plan and manually build table schemas and relations. Once the user completes this process, the system can automatically create forms for data entry, based on the schemas he supplied. A more modern approach to solve the same problem goes the exact opposite path, by allowing the user to build the form itself, with a drag and drop interface, then automatically generating a matching schema behind the scenes.

In general, there was a surge of app authoring tools in ’06-’07, and some of those attempts to empower non-programmers to build tools for and by themselves were halted by the ‘08 crisis. Under these circumstances, DabbleDB was acquired by Twitter in ‘10 and shut down in ‘11, Lazybase disappeared in thin air, Teqlo got shut down, and Coghead became part of SAP’s River platform.

The main tool used today for authoring small business apps is Caspio Bridge. It allows one to create databases, web forms, and apps, all without writing any code. With regard to the UI, it works similarly to MS Excel and Access, but is online, with integration support through a SOAP API, POST requests, Datahub, a JS library and a plugin for MS office. Another common solution is Intuit’s QuickBase, which allows the same standard app creation, as well as managing data through a spreadsheet UI. They differentiate by offering a large set of database templates for common industry-specific needs. Both these tools have pricing levels meant for SMBs (but not micro businesses), meaning their plans start at $250-300/mo. Zoho Creator is a third alternative with lower pricing (for small usage, whereas a high usage would cost a lot more than what’s offered by other services). It’s an online database and business apps service, with a form creation wizard, business rules engine, customizable reports, multiple view types (including an editable grid), alerting, rebranding and styles, collaboration, backups, APIs and data exports.

Other contestants in this field would be TrackVia, LongJump, FileMaker Pro and Viravis. They all offer database and app creators meant for non-programmers, as well as custom reporting and database templates. LongJump provides an SDK meant for adding functionality to the system by code. FileMaker Pro offers the ability to build apps for tablets and mobile. Eccentex is another solution, that provides a platform on top of which developers can build productivity apps, and end users can make use of them.

A newer generation of tools is emerging for the past two years or so. Apart from Dreamface, which is aimed at IBM BPM software customers, all of these solutions are meant for micro to small businesses as well as the early adopter consumer market, and are priced accordingly. Ragic! Builder is an app authoring tool that offers a spreadsheet UI meant to allow the end user to define the applications’ form fields. It then offers reporting, full text search, a query builder, embedding results onto one’s website, importing and exporting, access control, versioning, customized scripting and an API. Knack is another solution with really easy data management, user management, and that’s customizable with an open API as well as with CSS and JS editors. SodaDB is a donationware product that offers a simple and customizable database, importing and exporting, a form builder, full text search, and the ability to work without signing in.

Another brand of solutions is that of online spreadsheets, which sit at a crossroad between accounting software and online databases. These are meant to compete with MS Excel on specific vectors, and offer features that make them more attractive for these specific target audiences. Smartsheet is a paid-only online spreadsheet, with a modern UI, mobile versions, and an emphasis on collaboration. Zoho Sheet is a MS Excel clone that mostly fits existing customers of Zoho. Glide Crunch is a desktop spreadsheet application meant for large spreadsheets (the kind that won’t fit into an online tool), and that syncs automatically with your storage on Glide’s cloud office suite. AirXcell is a web based spreadsheet tool meant for scientific and statistical heavy-load work. It uses R language syntax for formulas and functions, and comes built in with a few financial applications. Flextory is a web application that acts as a sort of administration panel for your own data. It’s looks and acts like a standard web admin panel, with item editing forms, filtering, sorting and choosing table fields to view, but the data is really the kind of data you’d use MS Excel to manage. SecureSheet is an online spreadsheet that offers collaboration and security. BinaryThumb is a spreadsheet for iPhones and iPads that contains not only numbers but any kind of media (slightly pivoted since this text was written). Sumwise is a spreadsheet solution that offers smart features like grouping, reusable cells, etc. CollateBox offers collaboration on lists based on MS Excel data. Other solutions would include Editgrid and Skysheet.

UPDATE: Since the time of writing this text in 2012, a new online database solution emerged in the scene, called Team Desk.