ColdFusion CFHttp To Query Much Faster Than Java Buffered Reader

In response to some posts on CF-Talk that have discussed parsing large files, I thought I would do some investigation of my own. I have never parsed in a large file, so this was new an exciting. I was going to try two different approaches: Using the CFHttp to read in a file to a query and using the Java Buffered File Reader to read in a line of a file at a time and build a query based on the records. From my other experiences, I had assumed that the Java method was going to clean the floor with the CFHttp tag in terms of speed.

I was absolutely floored by the speed of CFHttp! Not only did it parse data files like no body's business, it did it so much faster than the Java method. When the test cases got too big, the Java method would often timeout.

NOTE: I am not a great Java programmer, so my methods could have been horrible.

As you can see, the data file was made up of 8 distinct column: last_name, first_name, hair_color, height, weight, body_type, pet, and has_nice_smile. The delimiter was a tab character. If you don't realize that the textqualifier attribute defaults to double-quotes, you are gonna pull your hair out trying to figure out why it can't find all the columns (and throws errors) if you use quotes in your field values!

To test my Java theory, I set up this test that read in a single line at a time:

In this test, I am creating a buffered file reader that is reading in one line at a time. For each line, it breaks it up into an array and adds those values to the query.

When I ran the test on 490,000 rows of data, CFHttp executed in about 25 seconds! I can't give you Java file buffer speeds as the page kept timing out. I must be doing something horribly wrong in my methodology as Java is what is really going on underneath any way. I chalk that up to my inexperience.

This is crazy! I am floored at how fast that is. So the question is, why are people still having problems reading in files. For starters, this only works with structured query-esque data. You can't read in an XML file object like this. Secondly, it requires a URL to work, so if you need to read a file that is outside the webroot, you are out of luck. That would be the benefit of the Java methodology (though clearly one better than the one I put together), you can read in files from anywhere in the file system.

To be honest, I am not sure what is making it faster. I assume that the parsing in the ColdFusion engine is just much faster and more efficient than anything that I was doing.

As far as the different for the file IO and the buffered file reader, the Buffered File read is actually a decorator that wraps around the file IO object and adds additionally functionality, like reading in large chunks at a time rather than one huge read or a ton of smaller reads. It just allows for more efficient file reading when you want to examine parts of the file content at a time.