Monday, April 30, 2012

Well that might be a stretch, but it was pretty cool. I sent a SPDY SETTINGS frame to Chrome indicating that the data transfer window was very low, which ensured that Chrome would use flow control. I then sent back the necessary WINDOW_UPDATE to Chrome as data was POSTed from browser to server, allowing data to flow.

Tonight, I would like to see what happens in the opposite direction. What happens when the data being sent from a spdy/3 server (flow control is new in spdy/3) is too large for the data window?

For that, I need a large image. One of the scanned documents from The SPDY Book should suffice:

I add the image to the homepage of my simple spdy-ized express.js, restart the server and load the page up:

That broken image icon is not the result of HTML-gone-wrong. At first, it starts to load, but quickly goes broken after that. Checking the SPDY tab of Chrome's chrome://net-internals, I see that the image is, indeed requested. Indeed, transfer of the data seems to start out OK:

The WINDOW_UPDATE from the browser is ignored and the server keeps right on sending data, which is what the browser wants anyway.

Another 31 data frames are sent, each sized 1300 bytes. There is also a stray 660 byte frame, for a grand total of 40960 bytes. And then Chrome issues another WINDOW_UPDATE indicating that all of the data received so far has been processed:

This is followed by another 25 full-sized frames and one 660 byte frame, along with a WINDOW_UPDATE for delta = 33160. The same pattern is repeated once more, resulting in another WINDOW_UPDATE for delta = 33160.

After each of these WINDOW_UPDATE frames, the internal data transfer window that the server should have is 64kb, the default data transfer window in SPDY. The server sends 33kb or so of data, the window goes down, then the browser signals that is has read all pending data and the window goes back to 64kb.

And then something interesting happens.

Not 29, 25, or 31 DATA frames are sent. This time, 60 full DATA frames are sent, plus two 660 sized DATA frames. This is a total of 79320 bytes that are sent from the server to the browser, which is well past the 64kb transfer window.

At this point, Chrome does the right thing by telling the server that something has gone wrong:

Sunday, April 29, 2012

I ended yesterday in an argument with Chrome over SPDY version 3 WINDOW_UPDATE frames. I thought that sending them immediately after connection should impact the browser's data window. Chrome disagreed.

Upon further reading of the specification, it turns out that I may, in fact, have been mistaken. I have a hard time reading networking specifications sometimes, but I now think that WINDOW_UPDATE is only sent after too much data has already been received.

The scenario from the specification is that the window size is originally the default 64KB, but the recipient immediately replies on connection that it can only handle 16KB. Unlike what I did yesterday, the specification states that the recipient communicates this information with a SETTINGS frame, not a WINDOW_UPDATE. The sender has already sent along the default 64KB, so it now realizes that it has sent 48KB too much. Assuming the recipient does not terminate the session (which is a possibility), then the sender needs to wait for enough WINDOW_UPDATE frames to be sent back to add up to 48kb at which point data can again start flowing.

That seems like a hard scenario to establish when the server is the recipient. The form that I use to submit the data will already have established SETTINGS:

So a race condition of 64KB vs. 16KB seems hard to establish. Besides, I could be completely wrong about this whole thing (again).

So, instead, I specify an absurdly small window size, 1024 bytes, that the server can handle in the initial SETTINGS:

After the browser sends the 1024B window size that the browser and server agreed upon, its window size should be 0. In other words, it should pause until it gets back a WINDOW_UPDATE saying that all or part of the original 1024B has been processed. At this point, the browser's window size will equal what WINDOW_UPDATE says has been processed (the 512B that I lie about).

So I fire up the browser, fill out the form with well over 1024 characters and submit:

As you might imagine, it takes a number of round trips to complete the conversation. This example is intentionally contrived to see flow control in action. In real-life scenarios, a SPDY stream might pause a much larger window because of a slow DB write or unresponsive cluster node. The nice thing about the per-stream flow control in SPDY is that other streams can proceed like normal and regular TCP/IP flow control can also do its thing for then entire connection.

It would be nice for node-spdy to detect situations in which WINDOW_UPDATE frames are needed. Hard coding them like this is no solution. I also need to get the reverse case working. That is, if the server needs to send a 150KB image back to the browser, the server should wait until receiving WINDOW_UPDATEs. Grist for tomorrow, at least.

Saturday, April 28, 2012

Today, I am going to try to get my node-spdy-powered express.js site to force Chrome to honor the SPDY v3 WINDOW_UPDATE frame. I think it more likely that, in real life, the browser would send the server WINDOW_UPDATE packets to indicate that the server should slow down. I am unsure how to coerce Chrome into doing this, but I can send WINDOW_UPDATEs from node-spdy.

In fact, I was able to get node-spdy to send WINDOW_UPDATEs last night, but they had no effect. This was because the browser has no more data to send—it merely sent HTTP GET requests and was then done. The browser can send data—if it is POSTing a form. So I create a jade template to POST a form:

The first four octets are version and type, the second 4 are the length (which is always 8 for WINDOW_UPDATE), the next four are for the stream ID, and the last four are the amount of data that can be sent beyond the initial window size (established in a SETTINGS frame). Actually, take my definition of delta with a grain of salt—I really do not have a firm grasp on the concept yet, which is why I am experimenting.

I think my first experiment will cause protocol errors. Even so, it seems the smallest step I can take. I am going to send a WINDOW_UPDATE immediately before my first SYN_REPLY. I think that this is wrong because the recipient of the data (the browser) should be sending WINDOW_UPDATEs per the spec, not the server. If I get a protocol error, then I can hope that I somewhat understand this and can build on it tomorrow. If I do not get a protocol error, then I can go back to the drawing board.

So, back in the server, I manually write a WINDOW_UPDATE after initializing the first SPDY stream:

The browser gets, and seemingly accepts, my WINDOW_UPDATE of a delta of 8 bytes. The rest of the SPDY conversation takes place as if this frame were not present. Specifically, the server issues a SYN_REPLY with more than 8 bytes in the DATA frame and no errors are raised:

I am not surprised that the SYN_REPLY and subsequent DATA are unchanged—I have not done anything in the server to alter their behavior. I am also not surprised that the browser does not barf on receiving them—the WINDOW_UPDATE is sent by the recipient, but, in this case, the recipient is the browser which sent no WINDOW_UPDATE and should therefore be unaffected.

The only thing that surprises me is that the client accepted the WINDOW_UPDATE in the first place. Then again, SPDY conversations are bi-directional, so maybe that should not surprise me. Technically, the server could be a recipient of DATA (e.g. a POST body).

Satisfied that, if nothing else, my WINDOW_UPDATE is crafted well, I call it a night here. Tomorrow, I plan to start by POSTing data after a WINDOW_UPDATE to see if it has any effect. Then the fun begins...

Thursday, April 26, 2012

Yesterday I finally got a version 3 flavored SPDY conversation to take place. I had enabled spdy/3 in Chrome's about:flags the other day, but the struggled through conversion of node-spdy's version 2 implementation to account for incompatibilities between the two versions and then a dumb mistake in assembling the headers.

Today, I clean up.

First, I remove every console.log() statement. There were a lot.

Then I make sure that everything is still working. Stranger things have happened than everything breaking after removing console.log() statements. Thankfully that is not the case this time. I still have my spdy/3 session:

There is still much work to go, but this good enough to create a branch on the node-spdy repository.

The whole reason that I started down the spdy/3 road was to experiment with flow control. But there is at least one thing that I have to do before I move onto that. I need to re-enable spdy/2. Nearly all clients still use spdy/2, but right now I only have the spdy/3 zlib dictionary (used for blindingly fast header compression) hard-coded in lib/spdy/utils.js. I need to move that out of utils.js and into the protocol version definitions. I also need to dynamically select the appropriate dictionary at connection time.

Thankfully, the node-spdy module makes the value of the Next Protocol Negotation (an extension of SSL) available on the socket. So, when the zlib compression contexts are established on the connection, I can supply the protocol version:

With that, I have my SPDY version three connection working again. This time, however, the dictionary comes from the correct protocol version library. After doing the same for spdy/2, node-spdy is now capable of serving up both spdy/3 and spdy2 from the same server:

That is a good stopping point for now. Tomorrow, I will hopefully pick back up with flow control.

Wednesday, April 25, 2012

I continue to struggle with implementing version 3 of the SPDY specification in node-spdy. I was finally able to get a response from inbound spdy/3 requests, but Chrome does not like the format of the response.

In fact, I have gotten to the point that Chrome actually recognizes that I am trying to speak spdy/3:

I have gone through the actual construction of the head a couple of times by hand and it looks OK. Of course, it's all writing bytes and lengths and things like that so it would be easy to make a mistake. Then there is the the Zlib compression...

I know that I have the Zlib inflation working because I can parse the inbound SPDY headers from Chrome. So a logical next step would be to attempt to parse the SPDY headers that I am generating. But before that, could it be as simple as missing those colon headers?

When I first parsed the inbound headers, I noticed some headers that started with colons:

I mistakenly thought something had gone wrong, but it turns out that they are part of the spec. And there are also colon headers that I am not including outbound: :status and :version. So I convert those headers over to be leading colon:

This ends up logging what looks like the correct opening series of bytes, but not the correct ending.

Sigh.... it seems that I will be assembling headers manually tomorrow.

Update: I figured it out! It was a simple matter of not including the correct number of octets / bytes for the headers. When I converted from spdy/2 to spdy/3, I accounted for the increase from 16 bit to 32 bit header length values everywhere except when totalling the length. All that I needed was an 8 (4 octets for the keys and 4 for the values) every time that I added a new key value pair to the header:

Fuuuuuu.... I can't even leave it at that. I look at so many amazing people in the Ruby, Javascript, and other communities that actually are amazing and I feel like I haven't done anything. But even so, looking back at the 366 days of the last year, what I did was, well... amazing.

I wrote three books on very different technologies that I knew nothing about.

And it worked. Every night, I ask a question to which I don't know the answer and I try my damnedest to answer it.

Every time I do this, I learn. The daily deadline forces me to learn. Blogging about it challenges my assumptions and makes me learn even more.

And then, doing it again the next day reinforces the learning. As does writing the book. And the second edition.

I am proud that I didn't let this get in the way of what's important. I still took vacations with the family—drove to the beach and Disney World. Birthdays, anniversary, sickness—I was there for it all.

And in the end, what did I learn? Well aside from a ton about cool technologies, I learned that...

I tricked myself into being awesome

I heard a story on RadioLab about a lady named Zelda. She tricked herself into quitting smoking by swearing that she would donate $5,000 to the KKK if she ever smoked another cigarette. And she never did. Would she have really donated that money if she had given in? Maybe not, but it was enough for her to have convinced herself that it would happen.

And, in the end, I did the same. Would the world have ended if I missed a day? Of course not. Very few, if any people would have noticed. But I would have noticed because I committed to doing this. And, after 366 days, I have more than not smoking to show for it. I have three books, the last of which is being published by The Pragmatic Programmers.

The Results

Tuesday, April 24, 2012

Up tonight, I continue in my effort to get node-spdy working with SPDY version 3. I have Chrome speaking SPDY v3 by virtue of about:flags settings. I have begun SPDY v3 implementation in the node-spdy server by copying the v2 implementation to v3, updating the compression dictionary, and making a change or two to the header parsing.

As of last night, I am able to parse headers from the incoming session, but that is about it. No response is ever sent and I can see no evidence that the request listener ever sees the request.

Now that I think about it, I have no idea why some of those headers begin with a colon (e.g. ':host'). That seems like it must be coming from Chrome. If it were a zlib inflation error, the values would have additional characters as well. Besides, a bit or two offset would cause the whole thing to crash.

The spdy/2 version of the headers definitely does not include the prepended colon:

Unfortunately, this does not quite work as a server error still arises. But at least it is a different error this time. Tantalizingly close, I have to call it a night.

Update: The colon headers are, of course, part of the spec. Folding them back into a new url header is necessary, but removing the colons was not. It seems that the response also needs colon headers—hopefully that accounts for most of my remaining problems.

Monday, April 23, 2012

Yesterday, I was able to get my simple express.js prepped and ready for SPDY v3 with an assist from the latest and greatest node-spdy. I even have Chrome speaking SPDY v3 thanks to an about:flag setting. Only the SPDY conversation quickly dies.

This is almost certainly because I do not actually have v3 well defined in node-spdy (in fact, I only have an exact copy of v2). I begin my investigation with Wireshark.... only everything looks OK. I already know that the browser is SPDY v3 capable and Wireshark confirms that the server also thinks it is capable of talking SPDY v3:

Unfortunately, node-spdy hard-codes the dictionary in lib/utils.js rather than in a protocol version location. For now, I simply aim to get this working, not architected well, so I copy the dictionary from the spec into utils.js:

Now, when I reload everything... I still get the same net::ERR_SPDY_PROTOCOL_ERROR error in the browser. On the plus side, the server is no longer crashing, so there's that!

The next bit of cleanup that I perform is to convert the header parsing from 16 bit to 32 bit. In SPDY version 2, the key value pairs (think HTTP headers) had been marked by 16 bit integers. Version 3 ups that to 32bit integers, mostly for consistency. At any rate, this involves replacing things like readUInt16BE with readUInt32BE and offsets of 2 with 4. Eventually, I am able to parse the incoming headers again:

Sunday, April 22, 2012

Up tonight, I would like to experiment a bit with SPDY version 3. Much of the research that went into SPDY Book was based on the draft for version 3, but all of the code was written for version 2.

I seem to recall hearing mention that someone had started work on version 3 support in node-spdy, but I see nothing in the fork graph in GitHub. So I start on my own. First up, I link my locally cloned node-spdy so that npm will use it rather than trying to download it from the network:

Saturday, April 21, 2012

I would very much like to move on to exploring flow control in SPDY, but first feel like it is a good idea to address a small bug in express-spdy. Express-spdy will be going away very soon since recent versions of node-spdy proper support express.js integration out of the box. But that will not be ready until the next node.js stable release. Since that may not be in time for the impending 1.1 edition of SPDY Book, I should get my express-spdy house in order.

The particular bug is that, after the initial SPDY connection goes through, the follow error occurs:

The browser sends a PING to the server, but the server fails to respond. In fact, the server is incapable of responding because the older version of node-spdy being used is PING-ignorant. Naturally, this all works in the recent node-spdy, but that does not help.

In the core SPDY server, the SPDY connection is piped through to an instance of a SPDY parser. I add a check for PINGs to the end of that parser:

The old node-spdy defined methods like createRstFrame and createSettingsFrame but, since it was PING ignorant, there is not a corresponding createPingFrame. I will eventually create one, but for now, I use the lower level createControlFrame() method to respond to the first PING:

Friday, April 20, 2012

I remain undecided about how I should approach node-spdy in the updated edition of SPDY Book, so tonight I take a break. Instead I am going to kick the tires on mod_spdy. When I first got started on this current kick of writing, I looked at mod_spdy and came to the conclusion that it was not worth expending time. A year later, that is no longer the case.

First up, I grab the 64 Ubuntu package (I do not miss compiling that gclient source code):

I follow those instructions (restarting apache), then access my local SSL server at https://jaynestown.local, and...

It works. I access my old SSL site and, checking out the SPDY tab in chrome://net-internals, I actually have a legitimate SPDY session:

Wow. That was incredibly easy. When I tried this a year ago, I despaired that writing The SPDY Book might well prove impossible on the first day I researched it, thanks to mod_spdy. Now, I can install a Deb package and I am running immediately. Amazing.

Some important notes: I benefit from running Ubuntu 12.04, which comes with a version of openssl that is capable of Next Protocol Negotiation (NPN). Without that, the browser would have no way of knowing that Apache was SPDY-enabled. I spent some time last year configuring my SSL server, some of the highlights of which are:

The bottom line is that SSL configuration is pretty easy, but not needing to install an edge version of openssl makes any pain of using a Beta Ubuntu more than worth it.

As for the mod_spdy module itself, aside from working, there is not much else to explore. Work it does, but there are relatively few configuration options. I am already eagerly waiting for SPDY server push configuration settings in a forthcoming release.

Thursday, April 19, 2012

I am slowly coming to the conclusion that this may not be the best time to issue a 1.1 edition of SPDY Book. Much of the new hotness that is node-spdy is in limbo until the unstable 0.7 node.js becomes stable 0.8. I almost have a handle on that except for express.js, the pleasant HTTP resource-oriented application server. The current 2.0 series will work with edge node-spdy, but not 0.7/0.8 node.js. For 0.8 node.js, the 3.0 alpha of express.js is imminent—only it does not play well with node-spdy.

All of these moving parts have me thinking that a more modest SPDY Book update might be the better part of valor. But I am not quite ready to give up. First, I would like to have a go at getting edge node-spdy and edge express.js working together.

The latest node-spdy does a rather elegant thing when creating SPDY-sized servers—it accepts an optional class argument that can be used to decorate server-like objects. This is specifically used to create express.js instances as I found last night:

var app = spdy.createServer(express.HTTPSServer, options);

The problem presented by edge express.js is that there is no express.HTTPSServer class anymore. In particular, the new express.application is more of a mixin than a proper class.

The answer (hopefully) is that node-spdy also works with HTTP server objects, which is what the express() function ought to return in 3.0. I have my forks of express.js and connect.js (for middleware) checked out locally. I have to modify the package.json for each to be compatible with 0.7:

Aw, man! That is nice. I was really worried that the new express.js would be incompatible, but clearly my worries were completely unfounded. Not only does it work, but it is really easy to configure and get running. Once again, kudos to Fedor Indunty for the very slick implementation.

Despite the ease with which I got all of this up and running, I am still in a bit of a bind with regards to including edge stuff like this in the updated book or sticking with express-spdy, which is clearly no longer needed. The former is a moving target and the latter is likely to be obsolete days after I put out the next edition of The SPDY Book. I think this suggests that minimal changes for now might be the best course of action, but I will sleep on it before making a decision.

Wednesday, April 18, 2012

As of last night, I know how to SPDY-ize express.js applications with only node-spdy—no express-spdy required. This requires a bit of futzing (switching down to express.js 2.x because node-spdy won't work with 3.x and modifying express to run on unstable node.js 0.7). I will worry about compatibility another day. Today, I would like to get SPDY server push working with this configuration.

The node-spdy documentation suggests that a push stream can be initiated with (this was corrected):

That will not work for my case because this is a dumb server that responds to all requests with nothing more than a document containing: <script src="/main.js"></script> (the res.end() line at the end). This will not give express.js a chance to kick in and do its thing. Also, I think that the res.send() line is meant to be res.push().

This is easy enough to try out, I save the dumb server, fire it up, and load the page in Chrome. This promptly results in a crash:

Man, I love SPDY server push. Chrome makes a request, to which the dumb server always replies "<script src="/main.js"></script>". Without SPDY server push, this would result in the browser making a second request of the server for the /main.js resource. But, using SPDY server push, we have already pushed the contents of /main.js into browser cache—in this case the alert() dialog. By the time the browser realizes it needs to request /main.js, it is already in browser cache so no request is actually made and the content is served directly from cache.

Checking things over in the SPDY tab of chrome://net-internals, I see that there is, indeed, a SPDY push stream taking place:

The only problem with the above SPDY server push is that the entire contents of the push are sent before the page response. In SPDY, data and headers are separate—even in push. It would likely be more proper to send just the headers in the form of SPDY_SESSION_PUSHED_SYN_STREAM, then complete the SYN_REPLY and send all data associated with the initial request along on stream #1 before sending the pushed data. That is a minor quibble (and one that can be addressed another day). For now, it is pretty cool that SPDY push works so well.

With that, I can return to my SPDY-ized express site. To perform a SPDY push in express, I need to modify my index route:

With that, I get the alert() pushed into cache. Once I click OK, the normal express page loads:

Kudos to Fedor Indutny on that API—I think that is an improvement on the original SPDY push that I had hacked into node-spdy. It is easy and makes sense. This is why I write the books, not the original code.

Tuesday, April 17, 2012

I am back on edge node-spdy and the view is wonderful. To be sure there are drawbacks suck as only running on edge node.js—but that is a also a benefit since edge node.js bundles a SPDY-friendly version of openssl. One of the other features of edge node-spdy is that it supports express.js out of the box. Tonight I am going to try to see how that works.

My first instinct is to install edge express.js (along with edge connect.js) in order to get this working:

I am not 100% clear if this will work because express.js has changed in preparation for the 0.8 series of node.js. The new 3.0 series of express.js features, among other things, a server object created with the express() method rather than the current createServer(). I have no idea how deep such changes go. Breaking things is one way to find out...

I require express, spdy and a couple other necessary modules. Then I create an app server instance via node-spdys' createServer() constructor. The first argument passed into createServer() is the base class used to create the application server instance—node-spdy should just decorate this instance with a few tweaks of its own.

I would not expect https.Server to respond to configure(). That is an express.js method whereas https.Server comes from node.js core.

So I give up, for now, on trying to get this working with the 3.0 series of express.js. Instead, I switch my local fork of express.js back to the 2.x series. Specifically, my fork includes the provision that this will work with the unstable version of node.js in package.json:

Monday, April 16, 2012

Last night I was able to confirm that express-spdy does not work with node.js installs that are built against older versions of openssl (i.e. those that lack NPN / Next Protocol Negotiation). I set off down the rabbit hole of trying to get exress-spdy to not install if it detected an older openssl, but was not able to get that working.

Tonight, I am abandoning that effort and instead following the advice of Fedor Indutny, node.js coder extraordinaire and author of the very excellent node-spdy package. He suggested that I drop my efforts in the node.js stable branch in favor of the unstable 0.7.x series. The 0.7.x bundles an NPN capable openssl. It has the additional benefit of node-spdy 1.0. I had planned on playing with that eventually. Now seems a good time.

Watching the conversation in Wireshark, I note that there is, indeed, no NPN (next protocol negotiation) taking place in the server hello portion of the SSL handshake:

Contrast this, for instance, with the SSL handshake from the 12.04 server:

I am hard pressed to decide if this is proper behavior for express-spdy. It makes no sense to use express-spdy without a version of openssl capable of NPN (per the openssl changelog, that means openssl version 1.0.1 or higher).

The only thing that actually compiles C code in the entire express-spdy dependency tree is the zlib compesssion code, which is not a place to be checking for the proper SSL versions. What if I use the node extension toolset, but only to check prerequisites? In that case, the package.json might add a script section like:

I do not know enough about WAF or node-waf to understand why this would not fail outright. It seems like it ought to halt. Somewhat frustrated, I call it a night here. I will bone up on my WAF tomorrow and hopefully come up with a way of getting this to fail.

Saturday, April 14, 2012

I now have express-spdy working with node.js—both the stable 0.6.x release and, as of last night, the unstable 0.7.x series. For both stable and unstable, I have a working express-spdy generator capable of generating and spinning up a SPDY server. The SPDY server has all of the goodness express.js but with working SPDY server push and other SPDY goodness.

My goal for the next couple of weeks is to get the unstable version working with the latest versions of node-spdy (express-spdy currently pegs to last year's version). Last year, I was able to keep things working against node.js unstable with crazy packages with which I polluted npm. This year, I found that I can develop this all on GitHub, provided that I use unstable branches for each of my express.js fork, my connect.js fork, express-spdy, and connect-spdy. Each of those unstable branches has a package.json that lists the other unstable forks as dependencies:

It is all one, big, happy, very unstable family. But it should be good enough to let me do what I need to do.

But, before moving on, I hope to remove another section from the install instructions for express-spdy. Specifically, installing edge-openssl is a pain. I recently upgraded to Ubuntu 12.04, which is rumored to include a version of openssl that supports NPN (next protocol generation). NPN is how SPDY clients are able to identify servers capable of speaking SPDY. Without it, SPDY will not work.

The current version of node that I have installed is SPDY capable by virtue of linking to the edge-openssl that I have installed locally:

After installing the system linked node.js, I npm install express-spdy globally (to get the executable generator), run the generator to build a sample server, install the packages necessary for the server, and fire up the server: