Risk of an uncompleted payload

So far so good. We have an excellent API that we can use for storing anything we want. And here magic starts.

We decide to store some analytics results, that are used by other APIs to grant access to some super essential and expensive business information™.

It doesn’t matter what the results are. All we need to know from our perspective, is the fact, that it will fit into memory. So, we hand out our API client code to other developers; we run our server and… in the middle of the night the phone rings:

Data that is supposed to be deleted is still available. We constantly run the DEL command but nothing disappears! We need to revoke all the access ASAP!

How can it be!? This service has been running for months now, and everything was good. There was a recent update in Ruby, but even after that specs were passing and the service has been running for at least two weeks.

And this is the moment when this bug presents itself in all the glory. For big enough payload, Ruby is trimming data that is being sent, and unfortunately for us, it trims last three letters, that is the full DEL command. When we run an ADD and DEL on a given string, we expect it not to be in the results anymore, however…

Note: the dots from the payload below aren’t usual dots but Unicode middle dots – that is important.

The data is still there! Because the data consists multibyte characters, the payload got trimmed, and we’ve ended up with a non-direct GET operation (DATA,) instead of a DEL. We had three multibyte characters in the data, and because of that, Ruby removed three last characters from the string before sending it to the server.

Patching things up

As a temporary patch you can use the body_stream instead of using body combined with Ruby StringIO:

module NetHttpFaradayPatch
def create_request(env)
super.tap do |request|
if env[:body].respond_to?(:read)
request.content_length = env[:body].size
end
end
end
end
Faraday::Adapter::NetHttp.prepend(NetHttpFaradayPatch)

Summary

It’s a rather unpleasant bug, and I’m quite surprised that despite being fixed, new Ruby version hasn’t had been released yetRuby 2.6.1 has been released and it fixes the issue. For now, if my patches work for you, that’s great but anyhow I would advise you to downgrade to Ruby 2.5.3 . It’s hard to be sure, that there aren’t other scenarios in which this bug may become even more problematic.