Random stuff from a software developer

Tag Archives: http

Graphical HTTP Client 1.0.7 was approved on the App Store this week. Here are some of the changes that came with it:

1. Save to File… functionality fixed

2. Support for PATCH requests

3. New cookie functionality

This included a bunch of changes, so I’ll talk a little more about it. In previous versions, cookies were automatically added to the request from the persistent cookie store on your machine. While this was desirable in most cases, it wasn’t very useful when you were trying to test cookie-related functionality. I’ve tried to make changes that preserve the automatic behavior when it is useful to you, while making it possible to customize. Here are the changes:

a. If you add a ‘Cookie’ header, the use of the persistent cookie store is disabled.

b. You can also click on the ‘Cookies’ button to pull up this dialog:

This lets you see all the cookies for a given URL/path that would be sent with the request. You can do the following here:

a. Turn off use of the persistent cookie store for this request.

b. Add cookies to or delete them from the persistent cookie store.

c. Manually select cookies you want to include in the request, and then click ‘Use selected cookies as request header’ to have them put into a Cookie header, which is automatically added to your request.

Hopefully this remains easy to use, but allows more flexibility with how you send cookies. If you have any problems or suggestions to make it easier to use, please let me know.

Next Version

I’ve already made a few small changes for 1.0.8, which should be released later this month. A small, but useful, change will be to use UTF8 for decoding response bodies if the response doesn’t include an explicit encoding. This should make things a lot better for users who work with services that use non-latin encodings and don’t specify an encoding in the response.

Windows Version

I’ve had a handful of requests for a Windows version over the past year or so, and have been considering this a little bit more lately. If you or someone you work with is interested in a Windows version, please fill out this short survey so I can have a little bit more information about interest in a Windows version.

I’ve had an enjoyable time getting down into a lot of gritty details about HTTP over the past year or so as I’ve worked on some webservices and a HTTP client tool for OS X. Every month, it seems like, I learn a little more about it based on the way my users use my app.

This past month, over the course of a few weeks, I got several tickets stating:

DELETEs are broken – they show as a POST on my server.

I was a little concerned when the first one came in – something as simple as sending the correct verb shouldn’t be broken, but if it were, that was pretty bad. A quick test showed that DELETEs were, in fact, working. After requesting a saved .httpreq file, I quickly pinpointed the problem – DELETEs normally worked, but when a request body was present, they were magically turned into POSTs.

Further investigation revealed the problem to be in the library I use to actually assemble and send the HTTP requests (this behavior is actually present in a number of HTTP libraries, it turns out).W hile fixing the problem (version 1.0.6 contains the fix and should be available on the app store soon), I wondered – Is it valid to include a body with a DELETE request?

I spent a bunch of time reading through the HTTP specification, and saw nothing that would indicate this situation was explicitly forbidden. In practice, however, a number of HTTP clients and servers seem to not approve – as I mentioned, a number of clients silently turn them into POSTs, and per this Stack Overflow thread, a number of servers silently discard the body of a DELETE request.

Regardless of whether it is allowed, or how it is implemented by popular clients and serveres, I think it is a bad idea in general. If you are consuming a service, you don’t really have a lot of choice in the matter, but if you are creating one, I think you shouldn’t create DELETE services that require bodies. This is due to how inconsistently it is implemented, and the fact that it doesn’t make a lot of sense to include a body with a delete in the first place – a body is supposed to represent the entity, but you are deleting it, so why do you need to send it?

There are a few reasons I think folks may try to develop services that rely on a body being sent with the DELETE request, and I think there are better ways to accomplish them:

1. To identify WHAT is to be deleted.

This should be specified by the URI itself.

2. To specify metadata about the delete request itself – for example, who deleted it or a comment related to the action.

In most cases, a header is much more appropriate for these metadata fields.

“The POST body isn’t working correctly! I add in something like name=spencer&age=29, but the server doesn’t get the parameters. Please fix your program!”

After a few support requests and 1 one-star review related to this issue, I figured it was at the very least worth a deeper explanation. If you’ve googled this and are just looking for the answer here it is:

You need to set the Content-Type header to ‘application/x-www-form-urlencoded’.

That said, I think it is worth spending a few minutes to learn about HTTP messages and why this is required. Removing levels of abstraction and learning more about what is going on underneath makes you a better developer and is usually a lot of fun, too. This is a big part of the reason I’m giving up my evenings and weekends to go back to school for a CS degree.

Anyway, here is a quick primer on HTTP messages and why you have to add the header above…

The HTTP message

When you make an HTTP request, you’re connecting to an HTTP server at a given IP on a certain port and sending it a message. It then processes that message and gives you some sort of response.

What does that message look like? It consists of 3 parts:

Request Line. This tells the server what resource you’re looking for and what HTTP method (ie, one of GET, POST, PUT, DELETE, etc..) you are using. It also specifies the version of the HTTP protocol to use.Headers. These give metadata about the message, such as the Content-Type, the Host, what type of content you’ll accept, and a whole bunch of other things.[CRLF]Optional message body.

So, for example, if you came to the homepage of my blog, your browser sent a message to my server that looked something like this:

GET / HTTP/1.1
Host: www.spenceruresk.com
[CRLF]

Pretty simple, right? Let’s look at a slightly more complex example. Let’s say I was hosting a Twitter-like service that allowed you to post status updates with simple HTTP POSTs. A request you’d send might look something like this:

POST /twitterClone/updateStatus HTTP/1.1
Host: www.spenceruresk.com
Content-Type: text/plain
[CRLF]
Check out this crappy cellphone pic I took at a concert!

In this case, we’re POSTing some data and telling the server what kind of content it is – in this case, it is just plain text. Pretty simple, right?

Content-Type and message bodies

One header in particular – Content-Type – is relevant to the issue and is worth talking about. When you send a message body, it can represent any number of types of data. For example, it could be plain text, a GIF, form data, or a JSON document. In order for the server to properly decode the body, you have to tell it what it is you’re sending it. The Content-Type header is how you do this. For a list of common MIME types, this Wikipedia article is useful.

Submitting forms

So, what HTTP message is generated and sent to the server when a user submits a form in their browser? The answer is that it depends. If the form is submitted via GET request, the message would look something like this:

The Content-Type line is critical – it tells the server “I’m sending you URL-encoded form data in the message body.” Most/All web frameworks see that, then decode the body and make each key/value pair available as request parameters.

If you don’t tell it that you are sending URL-encoded form data, it doesn’t decode the body and make the parameters available.

Conclusion

Hope that helps make it at least a little bit clearer. In the next version of the app, I’m thinking about adding some sort of Content-Type autodetection (that you can turn off) to help avoid people getting confused and frustrated in situations like this.