Introduction

Recently I wrote a program in which I wanted to generate a CRC for a given file. I did some checking on the web for sample CRC code, but found very few algorithms to help me. So I decided to learn more about CRCs and write my own code. This article describes what a CRC is, how to generate them, what they can be used for, and lastly source code showing how it's done.

What is a CRC

CRC is an acronym for Cyclic Redundancy Checksum or Cyclic Redundancy Check (depending on who you ask). A CRC is a "digital signature" representing data. The most common CRC is CRC32, in which the "digital signature" is a 32-bit number. The "data" that is being CRC'ed can be any data of any length; from a file, to a string, or even a block of memory. As long as the data can be represented as a series of bytes, it can be CRC'ed. There is no single CRC algorithm, there can be as many algorithms as there are programmers. The ideal CRC algorithm has several characteristics about it. First, if you CRC the same data more than once, you must get the same CRC every time. Secondly, if you CRC two different pieces of data you should get two very different CRC values. If you CRC the same data twice, you get the same digital signature. But if you CRC data that differs (even by a single byte) then you should get two very different digital signatures. With a 32-bit CRC there are over 4 billion possible CRC values. To be exact that's 232 or 4,294,967,296. With that many CRC values it's not difficult for every piece of data being CRC'ed to get a unique CRC value. However, it is possible for spurious hits to happen. In other words two completely different pieces of data can have the same CRC. This is rare, but not so rare that it won't happen.

Why use CRCs

Most of the time CRCs are used to compare data as an integrity check. Suppose there are two files that need to be compared to determine if they are identical. The first file is on Machine A and the other file is on Machine B. Each file is a rather large file (say 500 MB), and there is no network connection between the two machines. How do you compare the two files? The answer is CRC. You CRC each of the two files, which gives you two 32-bit numbers. You then compare those 32-bit numbers to see if they are identical. If the CRC values are different, then you can be 100% guaranteed that the files are not the same. If the CRC values are the same, then you can be 99% sure that the files are the same. Remember, because spurious hits can happen you cannot be positive that the two files are identical. The only way to be positive they are the same is to break down and do a comparison one byte at a time. But CRCs offer a quick way to be reasonably certain that two files are identical.

How to generate CRCs

Generating CRCs is a lot like cryptography in that involves a lot of mathematical theories. Since I don't fully understand it myself, I won't go into a lot of those details here. Instead I'll focus on how to program a CRC algorithm. Once you know how the algorithm works you should be able to write a CRC algorithm in any language on any platform. The first part of generating CRCs is the CRC lookup table. In CRC32 this is a table of 256 specific CRC numbers. These numbers are generated by a polynomial (the computation of these numbers and what polynomial to use are part of that math stuff I'm avoiding). The next part is a CRC lookup function. This function takes two things, a single byte of data to be CRC'ed and the current CRC value. It does a lookup in the CRC table according to the byte provided, and then does some math to apply that lookup value to the given CRC value resulting in a new CRC value. The last piece needed is the actual data that is to be CRC'ed. The CRC algorithm reads the first byte of data and calls the CRC lookup function which returns the CRC value for that single byte. It then calls the CRC lookup function with the next byte of data and passes the previous CRC value. After the second call, the CRC value represents the CRC of the first two bytes. You continuously call the CRC lookup function until all the bytes of the data have been processed. The resulting value is the CRC for the whole data.

Code Details

In this sample program I wanted to show that there are many different ways of generating CRCs. There are over 8 different CRC functions, all based on the above steps for generating CRCs. Each function differs slightly in it's intended use or optimization. There are four main CRC functions, each described below. There are also two separate CRC classes, but more on that later. And lastly there are a few helper functions that CRC strings.

C++ Streams: The first function represents the simplest CRC function. The file is opened using the C++ stream classes
(ifstream). This function uses nothing but standard C++ calls, so this function should compile and run using any C++ compiler on any OS.

Win32 I/O: This function is more optimized in that it uses the Win32 API for file I/O;
CreateFile, and ReadFile. This will speed up the processing, but by using the Win32 API the code is no longer platform independent.

Filemaps: This function uses memory mapped files to process the file. Filemaps can be used to greatly increase the speed with which files are accessed. They allow the contents of a file to be accessed as if it were in memory. No longer does the programmer need to call
ReadFile and WriteFile.

Assembly: The final CRC function is one that is optimized using Intel Assembly. By hand writing the assembly code the algorithm can be optimized for speed, although at the sacrifice of being easy to read and understand.

Those are the four main CRC functions. But there are actually two versions of each function. There are two classes,
CCrc32Dynamic and CCrc32Static, each of which have the above four functions for a total of eight. The only difference between the static and dynamic classes is the CRC table. With the static class the CRC table and all the functions in the class are static. The trade off is simple. The static class is simpler to use, but the dynamic class uses memory more efficiently because the CRC table (1K in size) is only allocated when needed.

// Using the static class is as easy as one line of code
dwErrorCode = CCrc32Static::FileCrc32Assembly(m_strFilename, dwCrc32);
// Whereas there is more involved when using the dynamic class
CCrc32Dynamic *pobCrc32Dynamic = new CCrc32Dynamic;
pobCrc32Dynamic->Init();
dwErrorCode = pobCrc32Dynamic->FileCrc32Assembly(m_strFilename, dwCrc32);
pobCrc32Dynamic->Free();
delete pobCrc32Dynamic;

Whenever you calculate a CRC you need to take into account the speed of the algorithm. Generating CRCs for files is both a CPU and a disk intensive task. Here is a table showing the time it took to CRC three different files. The columns are the different file sizes, the rows are the different CRC functions, and the table entries are in seconds. The system these numbers were captured on is a dual Pentium III at 1 GHz with a 10,000 RPM SCSI Ultra160 hard drive.

44 Kb

34 Mb

5 Gb

C++ Streams

0.0013

0.80

125

Win32 I/O

0.0009

0.60

85

Filemaps

0.0010

0.60

87

Assembly

0.0006

0.35

49

As expected the C++ streams is the slowest function followed by the Win32 I/O. However, I was very surprised to see the filemaps were not faster than the Win32 I/O, in fact they are slower. After I thought about it some, I realized memory mapped files are designed to provide fast random access to files. But when you CRC you access the file sequentially. Thus filemaps are not faster, and the extra overhead of creating the "views" of the file are why it's slower. Filemaps do have one advantage that none of the other functions have. Memory mapped files are guaranteed to be able to access files up to the maximum file size in NT which is 264 or 18 exabytes. Although the Win32 I/O may handle files of this size, none of the documentation confirms this. [Note: The largest file I have CRC'ed is 40 GB, which all eight functions successfully CRC'ed, but took over 10 minutes each.]

If anyone who reads this article knows a way to improve the speed even more, please post the code or email me. Especially if you know of a speed improvement for the assembly code. I will bet there are further optimizations that can be made to the assembly code. After all I don't know Intel Assembly very well, therefore I'm sure there are optimizations I don't know about.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

I wonder, how do I use it when serializing data to file (I wan´t to be able to validate data in file afterwards when reading it back in to my application).

In the serialisation process data is "streamed" to file. First after the file is generated, it would make sence to to the actual CRC check of the file. Another issue is if this does destroy the possibility to use serialization when reading the file back (at leasy some execptions will be thrown).

So my question is: Is there a standard method to use CRC when serializing, so data in the file can be validated afterwards (fx. before reading the file back).

I check several books that mentioned about CRC principle.
The basic theory is like as below description in coding.
====================================
for(index = 0; index <= x; index++)
{
crc = crc & 0x80000000 ? (crc << 1) ^ 0x04C11DB7 : crc << 1;
}
=====================
But, In the most of CRC32's implementation.
It used a look up table.
And both results are difference.
Why??
I am curious in CRC32's implementation method and theory.
Who is the inventor?
Why the result doesn't correspond to the basic theory?
Where can I get any related paper?
Your kind reply will be highly appreciated.

First, let me say this article was a big help. It had just the right amount of info for a person who knows very little about CRCs and simply needs the basics. However, I have a question/problem. I am working on a project that I need to have the same CRC each time it is compiled. I compile my project under VisualStudio 6.0 on WinNT and generate a CRC on the executable I just created. Then, I rebuild the entire project (using the exact same files and compiler) and do a CRC on the new executable. The two CRCs are different, even though the exes are the same. I have come to the conclusion that there is some sort of timestamp telling when the file was created. Does anyone know of an algorithm that compensates for the timestamp and ignores it? Does that make sense? If not, ask and I can clarify.
Thanks

How do you know the two files are the same? Did you do a bit-wise comparision (e.g. "fc /b"). Even though the source code didn't change, everytime a project is compiled, Visual Studio does not generate the exact same binary. You'd think it would, but it doesn't. I'm not sure why (maybe someone else knows), there might be an internally compiled timestamp which would explain the difference. But my program does not use the file system times. That's most likely why you're getting different CRCs, because the files are indeed different.

That having been said, I think there is a flaw in the way you're trying to use CRCs. Are you saying if you were to make actual changes to the code you still want to get back the same CRC? To try and force one file to have to same CRC as another file is a very difficult task, nearly impossible. The whole idea behind CRCs is the slightest difference in a file results in a significant change in the CRC value.

I opened the files in a Hex Editor, looking for differences, and noticed that there were 3 locations that a pair of entries were different. The locations of the differences are always the same, so I'm fairly sure that there is some sort of timestamp applied to the file. However, I haven't found if there is a way to control this from Visual Studio and was hoping someone might know where I could find information about this. The reason I need the CRC is to prove to certain authorities that I can recompile exactly the same executable now and six months down the road, with the exact same files (We're under verions control management).

You are correct in your conclusion. The PE file format contains multiple timestamps. The resources are also timestamped. To "prove" that two exe's are identical, you will have to parse the PE file format, crc'ing only the code sections.

Have you considered crc'ing the source code instead?

HPS HwndSpy - GUI developer's aid to visually
locate and inspect windows. For the month of August
only, use coupon code CP-81239 for 30% off.