Outdated card details are a big problem for online businesses. If your customers get a new card from their banks (or the number or expiry date changes), they have to manually re-add it to every service or the service stops working. It’s frustrating for customers, and loses paying customers for businesses needlessly.

We’ve rolled out support for handling new cards nicely. Now, when you save a customer with Stripe, their card will continue to work even if the physical card gets replaced by the bank. Stripe works directly with card networks so that your customers can continue using your service without interruption.

There’s no extra work required, and this feature works with most MasterCard, Discover, and Visa cards—without this improvement, over half of the cards stored with Stripe in the last year would stop working by 2016 if they weren’t updated.

The saved card only stays working as long as the credit or debit card account stays open. Your customers won’t have to worry about being billed after they’ve canceled their subscription or after they’ve closed a credit card account.

We hope this makes life easier for you and your users alike. If you have any questions or feedback, get in touch!

Using data from across the Stripe network, we’ve developed a machine learning system that evaluates charges in real-time and blocks those that are almost certainly fraudulent. By analyzing hundreds of different characteristics pertaining to each payment, these algorithms have already shielded businesses on Stripe from millions of attempted fraudulent charges.

Starting today, you can help improve these models. By letting us know when we’ve missed a charge that you believe to be fraudulent, or declined one that you think is legitimate, you can help train a model that’s optimized for your business. This ensures that the protection you receive will get better over time.

In practice, it’s very simple. In the dashboard, you can now:

Report and refund charges you believe to be fraudulent—just follow the “Report fraudulent payment” link when viewing a charge.

Mark as safe charges that Stripe blocked as suspected fraud. Charges we believe to be fraudulent will appear with a status of “Blocked” in the payments list and have a prominent message on the payment detail page. If you mark one of these charges as safe, you can retry it—we will not attempt to block it again.

There’s also equivalent functionality available in the API—check out the documentation for more details.

Over time, businesses on Stripe should not have to think about fraud and disputes. By enabling this feedback loop—between you and other businesses built on Stripe on the one hand, and our fraud detection infrastructure on the other—we’re confident that we’ll be able to deliver significant improvements in the future.

If you have any feedback or suggestions on these tools, or other thoughts about how we could better help you block fraud, please drop me a line.

Disputed payments—"chargebacks"—can be awkward and time-consuming. We’re taking advantage of the data we’ve seen across millions of transactions to make handling disputes easier and quicker, and to maximize the chance that you win disputes.

NewEnumerated evidence fields

Previously, you could only upload evidence for disputes as a string or a single file. We’ve learned that separating the type of evidence can come in handy, and you can now use typed fields such as product_description, shipping_documentation, or customer_communication in the API. (You can use the Dashboard to submit this data as well.)

NewFile uploads through the API

Before now, we only accepted evidence documents as a PDF through our dashboard. Based on your feedback, you can now upload PDFs, JPGs, or PNGs via the API or in the Dashboard.

UpdateDisputes in the Dashboard

We’ve also updated the evidence form in the Dashboard to take full advantage of these API changes. The new interface adapts to help you submit the most relevant evidence to win the dispute. For example, if a customer claims a product wasn’t received, the form will ask for shipping info if it’s a physical product or server access logs if it’s a digital service.

As much as we’d like these changes to be helpful if you face a dispute, we’re working to reduce disputes for all our users—we actually hope you have to use these tools as little as possible. If you’ve got any feedback, let us know.

We’ve added two new tools to the Dashboard to help you keep track of activity on your account.

First, it’s now easier to see activity on your account (and monitor for potentially unauthorized actions) with the security history overview. Your overview will show recent actions from all of your team members—from password and bank account changes to suspicious logins or data exports.

Second, you can now see in your logs which of your team members performed any given action through the Dashboard. This should both increase security and reduce confusion when multiple team members are managing an account.

As always, please contact us if you’re ever worried that your account has been compromised. And of course, if you have any feedback, feel free to reach out!

Starting at 10am Pacific Time this morning, you can watch The Interview at SeeTheInterview.com, powered by Stripe. The Interview was originally intended to be broadly available in theaters, but its release was curtailed after a number of distribution partners backed out.

When we announced the Open Source retreat,
we'd pictured it primarily as giving people the opportunity to work on
projects they'd already been meaning to do. However, the environment
we provided also became a place for people to come up with new ideas
and give them a try. One of these ideas, Libscore, is launching publicly today.

Top libraries used across the web.

Libscore, built by Julian
Shapiro with support from both us and Digital Ocean,
makes it possible for frontend developers to see where their work is
being used. The service periodically crawls the top million
websites, determines the JavaScript libraries in use on each, and
makes that data publicly queriable.

For example, wondering about MVC framework popularity? Backbone is
used on about 8,000 of the top million sites while Ember appears on only 185. You
can also query which libraries are used on your favorite site, or
view some precompiled aggregates.

We were attracted to Libscore because it sounded like internet
infrastructure that should exist. Sometimes—as with our support
for Alipay—we get to
build such components directly; sometimes, it seems better to support
something external—as with Stellar. If
you have other ideas, please let us know (or work on them here!).

In February 2013, we blogged about email
transparency at Stripe. Since then a number of other companies
have implemented their own versions of it (which a fewhavetalked
about publicly). We often get asked whether email transparency is
still around, and if so, how we've scaled it.

Email transparency continues to be one important tool for state
transfer at Stripe. The vast majority of Stripe email (excluding particularly
sensitive classes of email or threads where a participant has a strong
expectation of privacy) remains publicly available through the
company.

Today we're publishing two key components that have allowed us to
scale it this far: our list manager tool and updated internal
documentation reflecting what we've learned over the past year and a
half. Hopefully these will make it easier for others to run email
transparency at their own organizations.

Gaps

In the time since our first post, we've grown our mailing list
count almost linearly with headcount: from 40 employees and 119
mailing lists in February 2013 to now 164 people and 428 lists. A
plurality are project lists (sys@, sys-archive@, sys-bots@, sys-ask@),
but there's also a long tail on topics ranging from
country operations (australia@) to ideas
for things Stripe should try (crazyideas@).

We use Google Groups for our email list infrastructure. Today
we're releasing the web interface we've built on Google's APIs to make
managing many list subscriptions (and associated filters) easy. This interface, called Gaps, lets you do things like:

Quickly subscribe to or unsubscribe from a list.

View your organization's lists (categorized by topic), and which
you're subscribed to (including indirect subscriptions through other
lists).

Updated internal documentation

Scaling email transparency has required active cultural effort and
adaptation. As our team grew, we'd notice that formerly good patterns
could turn sour. For example, at first email transparency would
improve many conversations by letting people drop in with helpful
tidbits. But with a larger team, having many people jumping into a
conversation would instead grind the thread to a halt.

As we've identified cases where email transparency didn't scale
well, we've made changes to our culture. Below is our updated internal
documentation on how we approach email transparency. It embodies what
we've learned about how to make email transparency work at an
organization of our size:

Email transparency (from our internal wiki)

One of Stripe's core strategies is hiring
great people
and then making sure they have enough information to make good local
decisions. Email transparency is one system that has helped make
this possible. As with any rule at Stripe, you should consider the
recommendations in this document to be strong defaults,
which you should just override if they don't make sense in a
particular circumstance.

How it works

Email transparency is fairly simple: make your emails transparent
by CCing a list, and make it easy for others to be transparent by
observing the responsibilities below.

The main mechanisms of email transparency are the
specially-designated archive lists, to
which you should CC all mail that would normally be off-list, but only
because of its apparent irrelevance rather than out of any particular
desire for secrecy. The goal isn't to share things that would
otherwise be secret: it's to unlock the wealth of information that
would otherwise be accidentally locked up in a few people's inboxes.

In general, if you are debating including an archive list, you
should include it. This includes internal P2P email which you would
normally leave off a list, emails to vendors, and scheduling
email. Don't be afraid to send "boring" email to an archive list —
people have specifically chosen to subscribe to that list. You should
expect most people will autoarchive this list traffic (hence the
name!), and then dip into it as they prefer.

If you're new to it, email transparency always feels a bit weird
at first, but it doesn't take long to get used to it.

What's the point?

Email transparency is something few other organizations try to
do. It's correspondingly on us to make sure we have really good
indicators for how it's valuable. Here's a sample of things people
have found useful about email transparency:

Provides the full history on interactions that are relevant to
you. If you're pulled into something, you can always pull up the
relevant state. This is especially useful for external
communications with users or vendors.

Provides a way for serendipitous interactions to happen — someone
who has more state on something may notice what's happening and
jump in to help (subject to the limitations about jumping in).

Lets you keep up with things going on at various other parts of
Stripe, at whatever granularity you want. This reduces siloing,
makes it easier to function as a remote (and even just know what
we're working on), and generally increases the feeling of
connectedness.

Requires ~no additional effort from the sender.

Makes conversations persistent and linkable, which is particularly
useful for new hires.

Forces us to think about how we're segmenting information — if
you're tempted to send something off-list, you should think
through why.

Makes spin-up easier by immersing yourself in examples of Stripe
tone and culture, and enabling you to answer your own questions
via the archives.

Helps you learn how different parts of the business work.

Reader responsibilities

Email transparency cuts two ways. Being able to see the raw feed
of happenings at Stripe as they unfold is awesome, but it also
implies an obligation to consume responsibly. Overall, threads on an
archive list merit a level of civil inattention
— you should feel free to read it, but be careful about adding your
own contributions.

Talk to people rather than silently judging. If you see something on an email list that rubs you the wrong way or that you think doesn't make sense (e.g. "why are we working on that?", "that email seems overly harsh/un-Stripelike"), you should talk to that person directly (or their manager, if there's a reason you can't talk to them about it). Remember that we hire smart people, and if something seems off you're likely missing context or a view of the larger picture. No one wants their choice to send email on-list to result in a bunch of people making judgements without telling them, or chattering behind their back — if that can happen, then people will be less likely to CC a list in the future.

Avoid jumping in. A conversation on an archive list should be considered a private conversation between the participants. When people jump into the thread, it often grinds to a halt and nothing gets done. There will be some very rare occasions (e.g. if you have some factual knowledge the participants probably don't) where it's ok to lurk in to the thread, but in practice these should be very rare. By convention, the people on the thread may ignore your email; don't take it personally — it's just a way of making sure that email transparency doesn't accidentally make email communication harder. Knowing when to jump in is an art, and when in doubt, don't.

Don't penalize people for choosing to CC a list. Ideally, people are writing their emails exactly as they would if they were off-list. So be cognizant about creating additional overhead for people because they chose to CC the list. There may be typos or things that you're wondering about or don't make sense. If you're *concerned* about something being actively bad, then you should talk to the person, but if it's something small (e.g. "there's a typo", "this tone isn't Stripelike", "this conversation seems like a waste of time"), you should trust that there's either a reason, or the person's manager will be on the lookout to help them (especially if they're new).

Help others live by the above responsibilities. The only way we can preserve email transparency is by collectively nudging each other onto the right course. Whether it's poking someone to CC a list, or telling someone to stop venting about an email but just go talk to the author, the person responsible for fixing the shortfalls you see is the same as the one responsible for your availability.

Common scenarios/FAQs

I don't mind people being able to read this boring scheduling email, but I don't think it's worth anyone's time to read. You should still send it to an archive list! Archive lists are intended to be the feed of everything going on within a particular team — let the people who are subscribing decide if it's worth their time or not.

I have a small joke on this thread. Should I CC it to the list, or just send it to one person (or a small set of people)? Small jokes are good! The main cost is potentially derailing the relevant thread. So generally, if it's a productive, focused thread, just send your joke off-list, but if it's already fairly broad, then you should feel free to send the joke publicly.

I feel like I need to write my email for the broad audience that might be reading it, rather than the one person it's actually meant for. The only change between how you write emails for email transparency and how you would write them privately to other Stripes should be that one has a CC. That is, if you feel a need to rewrite your emails for the audience, then that likely indicates a bug in the organization we should fix. If you notice yourself having this tendency, talk to gdb — we should be able to shift the norms of the organization so this isn't a problem.

How do we make sure this respects outside people's expectations? In many ways, email transparency is just a more extreme version of what happens at other organizations — since it's opt-in, all of the emails are human-vetted to be shareable. Email transparency is mostly about changing the default thresholds. As a corollary, if someone requests that their email not be shared, then certainly respect their request.

Common exceptions

Like any tool, email transparency has its limitations. Since it's
in many ways a one-way communication system, email transparency is bad
for sensitive situations where people may react strongly. It's also
important to preserve people's privacy. The following is a description
of the classes of things which you may not see on an archive list.

Anything personnel related (e.g. performance).

Some recruiting conversations, especially during closing or when people are confidentially looking around. People's decision-making process at that stage is usually quite personal, and even if people have a hard time picking Stripe, we want to make sure that they start with a blank slate.

Communications of mixed personal and professional nature (e.g. recruiting a friend).

Early stage discussions about topics that will affect Stripes personally (e.g. changing our approach to compensation).

Some particularly sensitive partnerships.

As we said in the original email transparency post, it's hard to
know how far it will scale. That doesn't bother us much: we continue
to do unscalable things
until they break down. The general sentiment at Stripe is that email
transparency adds a lot of value, and it seems we'll keep being able
to find tweaks to keep it going.

Hopefully these components will help you with email transparency
in your own organization. If you end up implementing something
similar, I'd love to hear about it!

We’re open-sourcing the tool we use to collect and analyze on-call data from PagerDuty. We use pd2pg to improve the on-call experience for engineers at Stripe, and we think it’ll be useful for your teams too.

PagerDuty data in Postgres

PagerDuty is an important source of data about how services behave in production and the on-call load experienced by engineers. This data has been instrumental for managing and evolving our on-call rotations: over five months, we’ve reduced on-call load for our systems team by about 75%.

We import data from the PagerDuty API into a Postgres database using pd2pg, where we can use the full power of Postgres’ SQL queries.

The script incrementally updates existing data, so it’s trivial to refresh your database periodically. (It also fetches historical data from your account, so you can get started with long-term analysis right away.)

Querying PagerDuty data with SQL

You can start analyzing and exploring your PagerDuty data once it’s in the database with psql:

How we use pd2pg at Stripe

Weekly team report: Our sys team reviews a detailed on-call report each week. It covers all alerts sent by either a team-owned service or fielded by an engineer (which can include escalations from other team’s services). This detailed report helps us understand the types of incidents we’re seeing so we can prevent or respond to them better.

Per-service incident counts: Aggregates like per-service incident counts help give us a high-level overview. (They’re not actionable results in themselves, but do show us high-load services we should review further.)

Interrupted hours metric: A common way to measure on-call load is counting the number of incidents over a period a time. Sometimes, this over-represents issues that cause several related alerts to fire at the same time (which aren’t actually more costly than a single alert firing). To get a more accurate view of on-call load, we calculate an "interrupted hours" metric that counts the intervals in which an engineer receives one or more pages. This metric provides pretty good insight into real on-call load by suppressing noise from issues that result in multiple pages and more heavily weighting incidents with escalations.

On-hours vs. off-hours alerts: Pages during the work day are less costly than ones that wake an engineer up at 3am on a Sunday. So, we look at the metrics discussed above broken down by on-hours vs. off-hours incidents.

Escalation rate analysis: Frequent or repeated escalations may indicate that either that responders aren’t able to get to a computer, or they aren’t prepared to deal with the issue. Some escalations are expected, but keeping an eye on escalation rates across services helps us keep an eye out for organizational bugs.

Individual on-call load: Being primary on-call is a major
responsibility, and high on-call load can cause burnout in engineers.
To help understand on-call load at the individual level, we can perform user-specific variants of the above queries.

We’d love to hear how you use pd2pg. If you’ve got any feedback, please get in touch or send us a PR.

Timberlake is a dashboard that gives you insight into the Hadoop jobs running on your cluster. Jeff built it as a replacement for the web interfaces currently provided by YARN’s ResourceManager and MRv2’s JobHistory server, and it has some features we’ve found useful:

Avi wrote a Scala framework for distributed learning of ensemble decision tree models called Brushfire. It’s inspired by Google’s PLANET, but built on Hadoop and Scalding. Designed to be highly generic, Brushfire can build and validate random forests and similar models from very large amounts of training data.

Sequins is a static database for serving data in Hadoop’s SequenceFile format. I wrote it to provide low-latency access to key/value aggregates generated by Hadoop. For example, we use it to give our API access to historical fraud modeling features, without adding an online dependency on HDFS.

At Stripe, we use Parquet extensively, especially in tandem with Cloudera Impala. Danielle, Jeff, and Avi wrote Herringbone (a collection of small command-line utilities) to make working with Parquet and Impala easier.

If you’re interested in trying out these projects, there’s more info on how to use them (and how they were designed) in the READMEs. If you’ve got feedback, please get in touch or send us a PR.