Category: Engineering

Organizations all over the world use Telerivet for mission critical communication. We take system reliability very seriously, and we know that our customers rely on Telerivet to be available all the time – 24 hours a day, 365 (or 366) days a year.

For the past 3 years, Telerivet has built an excellent record and reputation for reliability, in large part because of our significant work to build systems and processes for redundancy, monitoring, alerting, and automatic failover – and perhaps in small part because of good luck.

However, Telerivet’s good luck took a break on Saturday, January 31, as our servers experienced three separate hardware problems that coincided with additional problems in our automated failover systems to cause more than an hour and a half of downtime.

Telerivet has had a basic status blog since 2012 – which we have rarely ever needed to update. As we responded to all the support requests related to the downtime on January 31, however, we realized we needed a better way to communicate with customers about service interruptions.

Today, we’re happy to launch our new status page at status.telerivet.com. It’s hosted outside of Telerivet’s own infrastructure, so that the status page will likely remain online even if Telerivet’s servers experience any issues in the future.

On the status page, our system administrators will post updates in real time about any significant problems detected with Telerivet’s service. We’ll also publish detailed incident reports (postmortems) explaining the root cause of outages as well as actions we take to improve our systems.

Importantly, the new status page makes it easy for you to subscribe to get updates on future incidents with Telerivet, which can be delivered via email, SMS, or webhook.

In addition, we have published several real-time metrics on our status page to give you additional insight into how Telerivet’s service is performing.

For Telerivet’s web app and API, the public system metrics on status.telerivet.com show our servers’ uptime as determined by an external monitoring service. In addition, it shows the average response time and error rate for all HTTP requests, as computed from our server log files.

With these public metrics, it’s now easier for Telerivet customers to see how well our service has performed – both now and in the past – and hold us accountable.

As the first entry on our new status page, we have completed a detailed incident report of the service interruptions on January 31, where you can read about the root causes and contributing factors of the downtime, as well as all the actions Telerivet has already taken in the past few weeks to fix the underlying issues and improve our reliability in the future. Read the incident report here.

We hope you find our new status page helpful. If you want to receive notifications of any future incidents, just visit status.telerivet.com and click the “subscribe to updates” button.

In many developing countries, where most business transactions have historically been conducted with cash, mobile payment systems like M-Pesa have rapidly transformed the way goods and services are bought and sold.

Instead of carrying cash everywhere, you can now pay with your mobile phone – and unlike Google Wallet or Apple Pay, you don’t even need a smartphone. To complete a transaction, you simply use your phone’s keypad to enter the business’ phone number or ID, the amount of money to send, and your PIN.

When a transaction is completed, the mobile network sends a SMS receipt to both the sender (buyer) and recipient (seller). If you’re the seller, you can see the SMS receipt and know that you’ve been paid.

This payment flow works for in-person payments – but what about businesses that operate online? Developer APIs provided by services like Stripe and PayPal make it easy for online services to accept payments via credit cards and bank accounts, but not via mobile money systems. Some mobile money systems may offer their own APIs, while others don’t have developer APIs at all, or have APIs that are only available for large businesses.

If you want to accept mobile payments and there’s no API available, don’t give up – just scrape the SMS receipts! There are just three easy steps:

Forward SMS receipts from your phone to your server

Extract the relevant data from the SMS receipt

Match the payment with the correct user and credit their account

At Telerivet, we’ve been using this method to accept payments from Tanzania via M-Pesa and Tigo Pesa for over a year already – allowing us to receive millions of shillings in revenue from customers who would have been unable to pay via credit cards or PayPal.

If you want to accept mobile money payments in your own online service, read on to learn how to do it:

1. Forward SMS receipts from your phone to your server

When anyone sends you a mobile money payment, the mobile network will send your phone a SMS receipt. Your online service needs this receipt to know that someone has paid you.

If you don’t have an Android phone already, get an Android phone and install the SIM card associated with your mobile money account. Then install the Telerivet Android app on your phone. Finally, configure the Webhook API to forward incoming SMS messages to a URL on your server.

Of course, this method requires your business to have a presence in the country where you’re accepting payments – it doesn’t help accept or transmit mobile payments internationally, since your business still needs to register for a mobile money account, and the funds stay in the mobile money account until you withdraw them.

2. Extract the relevant data from the SMS receipt

Each mobile money system uses a different format for their SMS receipts.

In order for your online service to process a user’s payment, the most important information is the amount received and the transaction ID.

Extracting this information will likely involve writing regular expressions to match the particular format of SMS receipts generated by the mobile money service. This can get a little tricky.

In the case of receiving funds via Tigo Pesa, we first make sure to only consider SMS messages from the sender ID “Tigopesa”.

(You may want to test if your mobile network allows other people to send spoofed SMS messages with the same the sender ID as their mobile money system. If so, you could prevent spoofed SMS receipts by keeping track of your current balance, and verifying that the balance in the SMS receipt is correct.)

To extract the amount received, we can use this regular expression:

UmepokeaW+TshW*([d,.]*d)

Note: You may also want to take special care to avoid letting someone falsify their payment amount by changing their registered name to “Umepokea Tsh 1,000,000”.

To extract the transaction ID, we can use this regular expression:

kumbukumbu noW+(w{8}.d{4}.w{5,}b)

While SMS receipt formats are typically relatively stable, mobile money systems sometimes change the format or wording of the receipts.

At this point, your service still doesn’t know which user account should be credited for the payment. (You might be able to figure it out from the payee name and phone number, but that’s unreliable.)

Instead, just store the transaction ID and payment amount in your database, so that it’s available in the next step.

3. Match the payment with the correct user and credit their account

Finally, you need to match the receipt with the correct user of your online service, so you can credit the correct account.

Some mobile money systems, including Tigo Pesa, have systems for businesses that allow customers to enter a reference number for their account when sending a bill payment. But this type of bill payment system is sometimes only available to large businesses. When someone sends money to your phone number, they probably won’t be able to send a reference number.

In this case, you can simply prompt the user to enter the transaction ID in your application, like in the example below:

The customer knows this transaction ID because the mobile money system sent the customer their own SMS receipt containing the same transaction ID:

When the customer enters a transaction ID in your application, your application can check your database for that transaction ID. If it matches, then credit the user with the payment amount associated with that transaction ID. (Be sure to prevent people from reusing the same transaction ID multiple times.)

If the customer enters a transaction ID that doesn’t match your database, it could be because you haven’t received the SMS receipt yet, or because the customer made a typo (often it can be hard to tell the difference between the digit zero and the letter O, or the letters I and L and the digit 1).

In this case, we suggest storing the transaction ID entered by the customer in your database and notifying someone on your team to look into the transaction. If there was a typo, you can record the payment manually; if the receipt was delayed, your system can process the payment manually whenever the receipt is received.

Scraping data from SMS receipts may not be the cleanest solution for accepting mobile money payments, but sometimes it’s the only solution.

Fortunately, at least one mobile money system soon plans to make this easier: Safaricom M-PESA in Kenya currently plans to open their developer API in April 2015 . But there’s no need to wait for someone else to release an API — just start scraping SMS receipts and letting your customers pay you today.

Like many online services, we at Telerivet spent much of last week responding to the OpenSSL Heartbleed vulnerability (see our post-mortem here).

While the lessons of Heartbleed are still fresh in our minds, now is the time for the software community to consider how we can improve our collective security and prepare for the next zero-day vulnerability. Whether the next vulnerability is in an open-source library, operating system, firmware, one’s own software, or a third-party service, many of the same lessons apply.

Our top 4 recommendations for online services to limit the fallout from the next vulnerability like Heartbleed:

1. Improve your SSL configuration

Heartbleed is the perfect advertisement for Perfect Forward Secrecy. Thousands of web services all realized this week that their SSL private keys may have been compromised. Those without PFS enabled also realized that someone who had been recording encrypted sessions could have used their private keys to decrypt up to two years worth of previous traffic to their servers.

But don’t just stop at enabling PFS — why not take this opportunity to enable HSTS in order to prevent against SSL stripping too?

SSL configuration is hard to get right. Qualys SSL Labs makes it easy to test your server’s SSL setup and make sure it’s both secure and compatible with all the browsers out there. Try running their SSL server test on your own server, and follow their suggestions to improve your SSL config until you get an A or A+.

2. Subscribe to security announcement lists

Many of us were lucky to learn about Heartbleed very quickly on Monday when it quickly reached #1 on Hacker News. But system administrators shouldn’t rely on visiting the right website at the right time in order to react quickly to major vulnerabilities.

Fortunately, most major projects have an email list to alert users of security vulnerabilities.

Since it can be impractical to subscribe to announcement lists for every library and software package you depend on (like openssl-announce, where Heartbleed was first publicly announced), joining the security announcement list for your server operating system would be good enough for almost everyone. Here are a few:

When the next big zero-day vulnerability happens, don’t leave it up to chance whether you find out about it immediately or after it's already too late.

3. Limit your exposure when using third-party services

Many online services use various third-party services for functionality like customer support, CRM, analytics, payments, and ads that they rely on to safeguard private data about their customers. Many of these services are integrated via JavaScript snippets within a web app, while others are integrated on the server side.

Vulnerabilities in any of these third-party services could compromise customer data just as easily as vulnerabilities in the main service itself. Yet most services’ Heartbleed post-mortems haven't discussed the risk of vulnerabilities in the third party services they use.

Commonly used services like Google Analytics, Mixpanel, KISSmetrics, Optimizely, Intercom, Stripe, and others would likely be a highly attractive target during a vulnerability like Heartbleed — either for intercepting information of millions of end users, or (worse) MITMing these services and injecting JavaScript code into the websites that use them.

In the case of Heartbleed, all of the services above were affected for at least some time before being patched. It’s hard to say with certainty that nobody used Heartbleed to extract user data from any of these services. And now that we know for sure that attackers can use Heartbleed to steal SSL private keys, it’s hard to say that nobody MITMed them either, though that would be more difficult to do on a large scale.

When assessing the security of your own application, keep in mind that your security is essentially no better than the least secure service whose JavaScript you embed on your site. Each third-party JavaScript you add increases the risk that a vulnerability in a system you have no control over could destroy the foundation of your web app’s security.

I say this as a former JavaScript embed developer myself. In a previous role, it would have been technically possible for me, acting entirely on my own, to inject arbitrary JavaScript code into thousands of websites. Today, thousands of engineers and system administrators have similar or greater abilities. In addition, the ability to inject arbitrary code is now a feature of A/B testing services such as Optimizely. In this case, your web app’s overall security is no stronger than the least secure Optimizely login password of any of your employees.

To prepare for the next zero-day vulnerability, online services should evaluate and limit their exposure to third-party services, especially services embedded via JavaScript. Self-hosting JavaScript files, when possible, can often limit the potential of compromised third-party services to inject malicious JavaScript code into your own website.

In addition, online services can prepare themselves by making it easy to immediately disable third-party JavaScripts that are vulnerable or not known to be safe when a vulnerability is announced.

Likewise, services that provide JavaScript snippets should prepare a way to disable their own service if they know that they are exploitable but are unable to patch themselves (as happened this week when everyone was waiting on Amazon to update their ELBs).

4. Design your systems with defense in depth

The Heartbleed vulnerability was surprising in some ways — how was such a critical part of the world’s security infrastructure broken for so long with so little review? But it isn’t as surprising as one might think. There is an unmeasurably massive amount of code in the wild that almost everyone online relies on every day. And in any of those obscure lines of code there might be one simple programming mistake that was overlooked in code review, missed by a static analyzer, and deployed into production. It has happened before and it will surely happen again.

As a developer, this means that it’s necessary to design your systems with multiple levels of security to mitigate the damage if one part of your system’s security fails, whether because of a vulnerability in your own code or someone else’s code.

In the case of Heartbleed, helpful mitigation methods would have included:

In general, imagine that an attacker was able to get through one level of your security, and think about how you can still prevent them from doing something even worse.

For example, imagine that an attacker somehow gained read-access to your database through a SQL injection vulnerability, or somehow gained access to your database backups. Then it would help if sensitive fields in your database were encrypted with a key stored outside the database.

Or imagine one of your employees’ laptops was stolen. It’d help if any customer data or secret keys on it were encrypted in a TrueCrypt volume.

Or imagine that an attacker already had root access to one of your database servers. Then you're in big trouble, but it would still help if you used Tarsnap with a write-only key so that they couldn’t just delete all your data along with all your backups.

And although “security by obscurity” is a bad foundation for a cryptosystem, obscurity is a perfectly valid additional layer of “swiss cheese” in a security system. Simply making things harder to find — like choosing a non-default port for your SSH servers — can at least buy you valuable time when the next zero-day is being exploited.

Since it is inevitable that another “Heartbleed” will happen again, defense in depth is the best way to make sure your users stay as protected as possible.

The two most secure types of software are the software that never ships and the software that does nothing at all. Unfortunately, neither type is very useful to anyone. In practice, we all have to make tradeoffs between competing priorities like shipping, adding features, and improving security. While the lessons of Heartbleed are still fresh, let’s all at least take a moment to make sure we’re well prepared for the next big zero-day vulnerability.

What do you think the software community should do to prepare for the next Heartbleed?

In the coming weeks/months/years, we'll be using this space to introduce new features and service updates, share tips and tutorials, profile organizations using Telerivet in interesting ways, and discuss our thoughts on the mobile ecosystem in general.

To kick off our new blog, let's introduce how Telerivet fits into the huge ecosystem of SMS tools and services, as we tell the story of why we created Telerivet in the first place.

In 2010, we wanted to set up a SMS service so that grassroots non-profit organizations in Tanzania could publish news to envaya.org, interact with other organizations, and receive notifications, even if they didn't have internet access.

Here's what we tried, and how we ended up building Telerivet instead.

Attempt #1: Online SMS APIs

If we had wanted to set up this SMS service in North America or Europe, instead of East Africa, it would have been simple.

There are a lot of great online SMS gatewayAPI services out there, like Twilio, Clickatell, Nexmo, and Tropo, which allow software developers to easily send and receive SMS messages from their own applications.

However, most of these services only offer phone numbers in a few countries, such as the U.S., U.K., and Canada.

This meant that our users in Tanzania would have to pay extra to send international SMS messages, instead of paying cheaper domestic SMS rates using one of the SMS bundles offered by their mobile network.

And although many of these services were able to send SMS to other countries, we found that international SMS was often unreliable and sometimes wouldn't reach users on particular mobile networks in East Africa.

Attempt #2: Buy a SMS shortcode

To avoid the problem of sending international SMS, we tried to get a phone number in Tanzania that would let us send and receive SMS domestically.

Unfortunately, in Tanzania there aren't yet any companies like Twilio where you can get an inexpensive "longcode" virtual mobile number with an API for sending and receiving SMS.

However, there are a number of mobile networks and resellers that offer SMS shortcodes.

Often these companies do not offer transparent pricing, and instead negotiate private contracts.

We negotiated with one of them, and they offered a year-long contract amounting to over $5,000, in addition to about 3 cents for each message sent or received.

The non-profit organization we were volunteering for couldn't justify spending that much on a SMS shortcode.

To reduce the cost of SMS shortcodes, many companies offer shared shortcodes, which allow multiple users to build services on top of the same shortcode and reserve different keywords for their applications.

For some uses, shared shortcodes are often an economical option. However, shared shortcodes have the limitation that each incoming SMS message must begin with the user's unique keyword, and we decided they wouldn't work for us.

Attempt #3: Download software to send and receive SMS with a GSM modem

Free downloadable software tools like FrontlineSMS and RapidSMS have proven popular among many non-profits in developing countries.

With these tools, you connect a GSM modem or a basic feature phone to a computer running the software, turning your computer into a SMS gateway.

Tools like FrontlineSMS are indeed useful in some scenarios, especially because you can use them in remote areas where you don't have internet access.

But as we explored using them for our SMS service in Tanzania, it soon became clear that they would prove expensive, unreliable, and unmaintainable in practice.

Connecting a phone or GSM modem to our computer to send and receive SMS was the first challenge.

These software tools interact with phones and GSM modems by using a protocol called the Hayes command set (also known as AT commands), essentially the same way computers interacted with analog modems in the 1990s and earlier, complete with archaic settings like baud rate and COM ports. However, each phone and GSM modem implements a different subset of the protocol, and only a small fraction of current devices are compatible with the software.

None of the phones we already had proved to be compatible with the software, so we ordered one of the devices known to be compatible. But when it arrived and we tried to connect it, the device drivers wouldn't install correctly on our computer.

As we struggled with connecting the GSM modem, we realized that even if we got it to work correctly, running a SMS gateway on a PC in Tanzania was not going to be maintainable.

In order to provide a SMS service 24/7, we'd need to dedicate a computer to it, make sure the software is always running and doesn't crash, and make sure the computer always has electricity – no easy task in developing countries like Tanzania.

And although software like FrontlineSMS and RapidSMS don't require internet access themselves, we would need reliable internet access in order to integrate with envaya.org. Since the non-profit's office in Dar es Salaam often has power outages that take the network down, this did not seem promising.

And if we wanted to access this computer from elsewhere, even just to send SMS notifications, we would need to host it somewhere where it would have a public IP address.

It was clear that the model of using PC and GSM modem as an SMS gateway was not going to work.

Attempt #4: Use a phone as an SMS gateway

We realized that since phones can send and receive both SMS messages and internet packets, that we could use a phone as a standalone SMS gateway without needing to be connected to a PC.

We started by exploring the possibility of building a J2ME (mobile Java) app that would work on the cheap feature phones common in countries like Tanzania. However, it didn't take long to discover that it simply wasn't possible to build apps that access the necessary SMS functionality on cheap feature phones.

We realized that using an Android phone as an SMS gateway was a great solution: sub-$100 Android phones such as the Huawei IDEOS are rapidly proliferating throughout developing countries (and are much less expensive than a computer and GSM modem); they're easy for non-technical people to maintain, and have built-in battery backup and redundant networking (Wi-Fi and mobile data) to stay operational in challenging environments.

Unfortunately, our testing showed that the existing Android apps were quite limited and not very robust. For example, all were limited by Android's restriction of 100 outgoing SMS messages per hour per app, and would freeze if you attempted to send more than 100 SMS per hour. Also, they all struggled with the problem of how to send outgoing SMS — since the Android phone generally does not have a public IP address, it has to constantly poll the server for outgoing messages, using bandwidth and battery life. They also didn't make it easy to figure out what was wrong if the phone was having problems.

Since none of the other Android SMS gateway apps were good enough for us, we built a vastly improved SMS gateway app ourselves, deployed it in Tanzania to power Envaya's SMS interface in October 2011, and shared it with other software developers.

Attempt #5: Start a company

Soon, other people began using our Android SMS gateway app, and non-developers started asking for our help because they didn't have the technical skills to set up servers, or to implement the complex developer API provided by the Android app.

Then we realized our opportunity: to provide a service to make it easy for anyone to build their own services and applications on top of SMS anywhere in the world, even if they didn't have the technical expertise (or didn't want to spend the time, money, or effort) to implement a complex API or run their own servers.

By building a service that combined an Android app with our own servers (aka "the cloud"), we could build something that non-technical users could use to send and receive SMS in developing countries, without all the issues of using a GSM modem with software on a PC.

A couple months of hacking later, we quietly released Telerivet in beta on Leap Day 2012. Telerivet lets people easily create and deploy custom mobile services anywhere in the world, letting them focus on the high-level aspects of creating and running mobile services without getting bogged down by technical barriers.

The response has been amazing. Spreading mostly by word-of-mouth, leading non-profits and businesses are already using Telerivet to send SMS in many countries around the world, such as:

Kiva to facilitate communication between borrowers in Kenya and lenders around the world

myAgro (2012 Echoing Green winner) to allow farmers in Mali to save for agricultural supplies (even throughout a coup-d'etat)

Even if you work in a place with a bit more stability than Mali or Somalia, you can use Telerivet to build automated SMS services like opt-in subscriptions, polls, and informational services, or just send and receive SMS in your browser.