Introduction

Recently I wrote a program in which I wanted to generate a CRC for a given file. I did some checking on the web for sample CRC code, but found very few algorithms to help me. So I decided to learn more about CRCs and write my own code. This article describes what a CRC is, how to generate them, what they can be used for, and lastly source code showing how it's done.

What is a CRC

CRC is an acronym for Cyclic Redundancy Checksum or Cyclic Redundancy Check (depending on who you ask). A CRC is a "digital signature" representing data. The most common CRC is CRC32, in which the "digital signature" is a 32-bit number. The "data" that is being CRC'ed can be any data of any length; from a file, to a string, or even a block of memory. As long as the data can be represented as a series of bytes, it can be CRC'ed. There is no single CRC algorithm, there can be as many algorithms as there are programmers. The ideal CRC algorithm has several characteristics about it. First, if you CRC the same data more than once, you must get the same CRC every time. Secondly, if you CRC two different pieces of data you should get two very different CRC values. If you CRC the same data twice, you get the same digital signature. But if you CRC data that differs (even by a single byte) then you should get two very different digital signatures. With a 32-bit CRC there are over 4 billion possible CRC values. To be exact that's 232 or 4,294,967,296. With that many CRC values it's not difficult for every piece of data being CRC'ed to get a unique CRC value. However, it is possible for spurious hits to happen. In other words two completely different pieces of data can have the same CRC. This is rare, but not so rare that it won't happen.

Why use CRCs

Most of the time CRCs are used to compare data as an integrity check. Suppose there are two files that need to be compared to determine if they are identical. The first file is on Machine A and the other file is on Machine B. Each file is a rather large file (say 500 MB), and there is no network connection between the two machines. How do you compare the two files? The answer is CRC. You CRC each of the two files, which gives you two 32-bit numbers. You then compare those 32-bit numbers to see if they are identical. If the CRC values are different, then you can be 100% guaranteed that the files are not the same. If the CRC values are the same, then you can be 99% sure that the files are the same. Remember, because spurious hits can happen you cannot be positive that the two files are identical. The only way to be positive they are the same is to break down and do a comparison one byte at a time. But CRCs offer a quick way to be reasonably certain that two files are identical.

How to generate CRCs

Generating CRCs is a lot like cryptography in that involves a lot of mathematical theories. Since I don't fully understand it myself, I won't go into a lot of those details here. Instead I'll focus on how to program a CRC algorithm. Once you know how the algorithm works you should be able to write a CRC algorithm in any language on any platform. The first part of generating CRCs is the CRC lookup table. In CRC32 this is a table of 256 specific CRC numbers. These numbers are generated by a polynomial (the computation of these numbers and what polynomial to use are part of that math stuff I'm avoiding). The next part is a CRC lookup function. This function takes two things, a single byte of data to be CRC'ed and the current CRC value. It does a lookup in the CRC table according to the byte provided, and then does some math to apply that lookup value to the given CRC value resulting in a new CRC value. The last piece needed is the actual data that is to be CRC'ed. The CRC algorithm reads the first byte of data and calls the CRC lookup function which returns the CRC value for that single byte. It then calls the CRC lookup function with the next byte of data and passes the previous CRC value. After the second call, the CRC value represents the CRC of the first two bytes. You continuously call the CRC lookup function until all the bytes of the data have been processed. The resulting value is the CRC for the whole data.

Code Details

In this sample program I wanted to show that there are many different ways of generating CRCs. There are over 8 different CRC functions, all based on the above steps for generating CRCs. Each function differs slightly in it's intended use or optimization. There are four main CRC functions, each described below. There are also two separate CRC classes, but more on that later. And lastly there are a few helper functions that CRC strings.

C++ Streams: The first function represents the simplest CRC function. The file is opened using the C++ stream classes
(ifstream). This function uses nothing but standard C++ calls, so this function should compile and run using any C++ compiler on any OS.

Win32 I/O: This function is more optimized in that it uses the Win32 API for file I/O;
CreateFile, and ReadFile. This will speed up the processing, but by using the Win32 API the code is no longer platform independent.

Filemaps: This function uses memory mapped files to process the file. Filemaps can be used to greatly increase the speed with which files are accessed. They allow the contents of a file to be accessed as if it were in memory. No longer does the programmer need to call
ReadFile and WriteFile.

Assembly: The final CRC function is one that is optimized using Intel Assembly. By hand writing the assembly code the algorithm can be optimized for speed, although at the sacrifice of being easy to read and understand.

Those are the four main CRC functions. But there are actually two versions of each function. There are two classes,
CCrc32Dynamic and CCrc32Static, each of which have the above four functions for a total of eight. The only difference between the static and dynamic classes is the CRC table. With the static class the CRC table and all the functions in the class are static. The trade off is simple. The static class is simpler to use, but the dynamic class uses memory more efficiently because the CRC table (1K in size) is only allocated when needed.

// Using the static class is as easy as one line of code
dwErrorCode = CCrc32Static::FileCrc32Assembly(m_strFilename, dwCrc32);
// Whereas there is more involved when using the dynamic class
CCrc32Dynamic *pobCrc32Dynamic = new CCrc32Dynamic;
pobCrc32Dynamic->Init();
dwErrorCode = pobCrc32Dynamic->FileCrc32Assembly(m_strFilename, dwCrc32);
pobCrc32Dynamic->Free();
delete pobCrc32Dynamic;

Whenever you calculate a CRC you need to take into account the speed of the algorithm. Generating CRCs for files is both a CPU and a disk intensive task. Here is a table showing the time it took to CRC three different files. The columns are the different file sizes, the rows are the different CRC functions, and the table entries are in seconds. The system these numbers were captured on is a dual Pentium III at 1 GHz with a 10,000 RPM SCSI Ultra160 hard drive.

44 Kb

34 Mb

5 Gb

C++ Streams

0.0013

0.80

125

Win32 I/O

0.0009

0.60

85

Filemaps

0.0010

0.60

87

Assembly

0.0006

0.35

49

As expected the C++ streams is the slowest function followed by the Win32 I/O. However, I was very surprised to see the filemaps were not faster than the Win32 I/O, in fact they are slower. After I thought about it some, I realized memory mapped files are designed to provide fast random access to files. But when you CRC you access the file sequentially. Thus filemaps are not faster, and the extra overhead of creating the "views" of the file are why it's slower. Filemaps do have one advantage that none of the other functions have. Memory mapped files are guaranteed to be able to access files up to the maximum file size in NT which is 264 or 18 exabytes. Although the Win32 I/O may handle files of this size, none of the documentation confirms this. [Note: The largest file I have CRC'ed is 40 GB, which all eight functions successfully CRC'ed, but took over 10 minutes each.]

If anyone who reads this article knows a way to improve the speed even more, please post the code or email me. Especially if you know of a speed improvement for the assembly code. I will bet there are further optimizations that can be made to the assembly code. After all I don't know Intel Assembly very well, therefore I'm sure there are optimizations I don't know about.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

The old iostream library has now been removed from VS 2003 (it was flagged as deprecated in vs 2002).

This means that the FileCrc32Streams function will not compile, however if the FileCrc32Streams function and header definition are commented out, the rest will compile fine so no problem if you don't use FileCrc32Streams.

I really like this article and found it very useful, I encourage you to take a look at re-writing FileCrc32Streams to use the new Standard C++ iostream library.

The code I provided was never meant to be used as an SDK if you will. The article is meant to teach people about CRCs and give them code showing them how it's done. It is up to the other developers to use my code as a guide, or copy and paste the necessary sections into their projects. Thus I doubt I'll take the time to update the code to the new compiler.

As #include is deprecated in VC7, the code compile, but if you use it in a MFC application, it couldn't link because you have a "delete" function redefinition in the .lib containaing file stream functions ...
The solution to use it in VC7 with MFC is to comment
//#include
and to comment FileCrc32Streams function in both .cpp and .h files...

sorry ... i post the message without check the box "Display this message as-is (no HTML)" ... and thus, it is not complete ... Here is the message:

As #include <fstream.h> is deprecated in VC7, the code compile, but if you use it in a MFC application, it couldn't link because you have a "delete" function redefinition in the .lib containaing file stream functions ...The solution to use it in VC7 with MFC is to comment//#include <fstream.h>and to comment FileCrc32Streams function in both .cpp and .h files...

In the new Standard C++ iostream library
"open" functions do not take a third parameter (the protection parameter),
and some elements of the old iostream library are not elements
of the new iostream library, among them is "nocreate".

First of all, you must change the
#include <fstream.h>

to:#include <fstream>

Then add the string:
using namespace std;

You should also make some changes in the function "open" to
comply with the new stream library implementation of VC7,
and it will just work fine.

I decided to try my hand at the assembler, poked around the link listed by Ante above. My result is below. In my case, I "init" the CRC in a separate call, this routine works a series of buffers as they're received until I reach the end. My table is a global variable, not a static member of the object, but otherwise it's like yours. I noted that on the site Ante links to, they claimed 158 Mbytes per second on a T'bird 1.4 Ghz. On my 1.5Ghz AMD XP I get 275 Mbytes per second, not sure if the T'bird should be that far behind or not. For me, the punch came from loading a quad word at a time, not a byte at a time.

I'm using your awesome crc generator. I have a question though: is it possible that a crc would be returns as 0, and how probable is this? I am assuming that I don't have a crc if it is equal to 0, so I'm wondering how safe is this of an assumption.

Cheers,

swinefeaster

Check out Aephid Photokeeper, the powerful digital
photo album solution at www.aephid.com.

A CRC of 0 is entirely possible. The most likely case for a CRC of 0 is in the case of a zero byte file. But it is also possible for a non-zero byte file to have a CRC of 0. Basically all CRC32s from 0 to 0xFFFFFFFF are possible. Thus you cannot rely on the CRC return value to determine if the CRC was successful. That's way all of my CRC functions take the CRC as a parameter and return an error code from the function. You must use that error code to check for errors, not the CRC value.

???
Special? There's nothing special with the CRC32 usind in PNG, it's just the standard CRC-32 polynomial. What made you think there was anything special about it?

Can somebody show me som code for this, which doesn't use a LUT?!?

You almost always use a lookup table. The difference might be that some implementations provide it explicitly while others generate it "on the fly". For an example of the latter you might want to Google just one before asking. Try "CRC-32 generate".

On NT platforms it may be possible to increase the performance very slightly by using 2 buffers & overlapped IO, so that the OS is loading your next buffer whilst you are calculating the CRC of the current buffer. Does anyone with a good grasp of overlapped IO care to try it?

Thanks for the idea. I don't know why I didn't use overlapped IO in the first place. The results of my tests are overlapped IO is and isn't faster. Let me explain. I created several new CRC32 functions that were identical to the assembly functions, except the file reading was done using overlapped IO. When I ran the tests sometimes overlapped IO is faster, but most of the time it's not. The results were all over the place, especially in the "static" class. Sometimes the overlapped IO function was faster than assembly function, but most of the time it was slower. The results in the "dynamic" class were more consistent. The assembly function was always faster than the overlapped IO. To be honest, I don't even know why there is a speed difference between the dynamic and static classes in the first place. However, one thing is true from my testing, the dynamic assembly function is still the fastest. Faster than overlapped IO and definitely faster than filemaps.

Again, these test results come as a surprise to me. I originally expected the filemap function to be the fastest, but that proved wrong do to the sequential versus random file access. But I would have expected overlapped IO to be faster. The only reason a programmer would want to use overlapped IO is because there is a lot of processing that could be done while the file is being read. My guess right now is the assembly CRC code is so efficient that it completes the CRC'ing of the previous buffer so fast, that it ends up waiting for the next buffer just as if it were synchronous IO. But the extra overhead associated with overlapped IO (creation of multiple buffers and copying the data from one buffer to the other) is enough to cause the code to run slower.

Nonetheless, I'll try and get my overlapped IO code posted just in case someone else can tweak the code so that it runs faster.

I am building an SFV-checker in MFC and found a small ASM routine to calculate the CRC32 just like yours. I am trying to get overlapped I/O but havent got that working correctly as beeing faster. Would you mind pasting your code. Maybe I can do some improvments. The code I have today is fairly fast Intel but very slow on AMD. Still I/O seems to be the bottleneck rather than the CPU.

Could you elaborate a bit on what was done in the assembly version of the algorithm ? I am kind of curious.

The math is actually rather interesting. It involves an ex-or for each byte followed by an and, shift, and another ex-or for each bit of each byte. This makes it very computationally expensive. This is one reason that lookup tables were generated for it. I have worked with the CRC-16 algorithm but not CRC-32. I suppose that they are not significantly different. Oddly enough, I have run across a few PLC communication protocols that rely on CRC-16 for error checking.

Another odd story: back in the early 80s DEC VAXs were popular for factory automation applications. We developed a driver for the RTU protocol that utilized CRC-16 for error checking. This would absolutely swamp a VAX-750. We had a service rep come out and modify the wire-wrapped motherboard and install new firmware to support a new CRC-16 instruction performed by the CPU hardware. This sped things up quite a bit but I was shocked by what he had to go through. It was something else seeing a CPU that occupied a motherboard approximately 50x50cm and it was wire wrapped.

Well, all CRCs are based off of the same theory of polynomial arithmetic (most of which I don't understand.) The size of the polynomial is where the name comes from. However, early on people started calling very specific CRCs based on the polynomial size. Thus we have CRCs called CRC16 and CRC32. These names don't even come close to describing all the elements of a CRC.

Here is all the different parameters for a CRC:

1. What is the polynomial?
2. Are the input bytes reflected?
3. What is the initial value of the CRC?
4. Is the final value XORed?
5. Is the polynomial reversed?

1. The polynomial specifies the mask of the CRC and the size. A polynomial such as x^16 + x^12 + x^5 + 1 has a mask of 0x1021. The X^16 isn't included in the mask. A 32 bit CRC would have an x^32 element.

2. When each bit is processed in a CRC, you can either start from the high bit and move to the low bit, or the low bit and move to the high. The latter case is know as reflected. Now, the slow way of doing a reflected CRC is to reflect the input values as they are being processed. The fast way is to reflected the whole bloody algorithm including the polynomial mask. Thus making the reflected CRC just as fast as the non-reflected. In my experience, it is more common to find reflected algorithms than non-reflected. If your CRC is being shifted left, that is a non-reflected CRC. If your CRC is being shifted to the right, that is a reflected CRC.

3. The initial value of a CRC is very important. This value is usually 0xFFFF for a 16 bit crc and 0xFFFFFFFF for a 32 bit crc. Some CRCs use a 0 initial value which isn't a very good idea since you can't tell the difference between a stream of 11 bytes of 0 and 12 bytes of zero.

4. Many algorithms XOR the final value with 0xFFFF for a 16 bit crc and 0xFFFFFFFF for a 32 bit crc. This is purely up to the whim of the programmer.

5. Well, some moron had to do it. They used a non-reversed polynomial mask in a reflected algorithm or they used a reversed polynomial mask in a non-reflected algorithm. The good news is that a reversed poly seems to work just as well.

So, to answer your question, which CRC32 are you talking about? I have seen two or three.

Thanks, that is some interesting info. The specific CRC16 I worked with was used as a checksum in the RTU communications protocol. I have seen this same protocol used with several PLCs and RTUs over the years. Regarding the issues you mentioned :

- it uses 0xFFFF as an initial value.
- it does right shifting so it is reflected.
- if the low order bit is one prior to the shift, the value is xor'ed with 0xA001
- there is no final xor done.

I had no specific CRC32 in mind. I was just curious about what the value of the xor mask was. I looked through the ziplib code that I have and I noticed that they use a lookup table also for their CRC32 so the xor mask is built in.

Well I don't know really how to elaborate on what the assembly code does. The assembly code is pretty well documented as to what each line does.

To see a C++ version of the assembly code, compare the two functions FileCrc32Win32 and FileCrc32Assembly. The functions are identical except inside the while loop. The first function has a for loop inside of which it calls the CalcCrc32 function. The assembly function does this exact same logic as this except it does it in hand written assembly. The 'crc32loop' marks the beginning of the loop, which replaces the for loop. The next 8 lines of assembly do the same computation as the CalcCrc32 function. The assembly code above and below the 'crc32loop' is housekeeping if you will, getting set up for the loop.