I've seen that dr1394 has made a couple comments that he hasn't been doing much with the w6rz site recently. I would like to have a 75% gray, above white, and maybe a flashing color decoder pattern on an HD DVD disk. Can anyone give me any insight if it's possible to create the base image files with a computer graphics editor? Movie factory seems to accept either MPG or image files, so I figure getting correct images would be a good start.

From Ron's posts I've gathered that his files were created in YCbCr, not the typical RGB that most image editors use. On searches the only editor that I've noticed which is said to use that without getting into conversion of some sort is Krita on Linux. I found a comment that the latest stable version was said to use 601 and they were wanting to change things for 2.0 to make it adjustable where 709 would be a possability. I'm not sure if I really want to learn how to use Linux just to check out Krita alpha files where I have no idea if that change was implemented.

Sorry for the extremely late reply, but I had to resurrect some code to make a meaningful response.

IMHO, the major issue with creating test patterns is verifying their accuracy. After all, they are to be used for calibration, so it's imperative that they are accurate (and traceable).

For all of the patterns on my website, I used a direct to YCbCr technique. Since I'm a software guy (well, I'm really a hardware guy, but I know how to write code), I just wrote C programs to create the patterns. For example, here's the code for REC709 colorbars:

The program creates a YCbCr file (in UYVY 4:2:2 format) that can be fed directly to an uncompressed file server that I use with my hardware MPEG-2 HD encoder. The file is also compatible with AVIsynth, using the "RawSource" plugin.

Of course, most folks won't be willing to take this approach. Many don't even have a C compiler on their Windows machine. So an approach that works with an image editor is required.

When I helped out on the GetGray disc, I developed a program to convert .tga files created in Photoshop to YCbCr files. I've updated the code to do REC709 for HD images, and it can be downloaded here:

Sorry for the extremely late reply, but I had to resurrect some code to make a meaningful response.

IMHO, the major issue with creating test patterns is verifying their accuracy. After all, they are to be used for calibration, so it's imperative that they are accurate (and traceable).

For all of the patterns on my website, I used a direct to YCbCr technique. Since I'm a software guy (well, I'm really a hardware guy, but I know how to write code), I just wrote C programs to create the patterns. For example, here's the code for REC709 colorbars:

Ron,
Can you please tell how you verified your patterns? I have generated a MPEG2 pattern (red primary window) using your outline above, but every program I use to display it on my computer monitor looks different and measures different. It seems as though the final output on my computer is altered by the program I'm using to display it. How did you verify that your patterns were correct when you were done?

One thing I'm doing differently is using a software MPEG2 encoder (HcEnc and QuEnc give similar results). Both of these free encoders require YV12 as the input, so I'm having to use ConvertToYV12 in my Avisynth script. Should this cause any problems? Isn't MPEG2 on DVD stored in YV12 anyway?

Lastly, have you tried converting a 1280x720 .tga with your program? 1920x1080 works like a champ, but when I use 1280x720 as command line arguments and attempt to playback the Avisynth script (using the resulting 720 .yuv file) in VirtualDub, VirtualDub gives an error stating that the file is too small for even one frame?

This will decode the bitstream file to a sequence of .tga files with PC (0 - 255) levels using the REC709 matrix (unless the encoder had put a REC601 display extension in the bitstream). Or you can decode to YCbCr.

This will decode the bitstream file to a sequence of .tga files with PC (0 - 255) levels using the REC709 matrix (unless the encoder had put a REC601 display extension in the bitstream). Or you can decode to YCbCr.

I checked using the reference decoder and the pattern was decoded incorrectly. I think I found where the problem is. I am using a 100% red window. This was created in Photoshop with RGB values 235,16,16. I save that image in 24 bit .tga.

1) I create an Avisynth script using ImageSource to load the .tga file directly into VirtualDub. No other processing is used. The levels are correct when loaded into VirtualDub (235,16,16). This indicates Photoshop is saving it out correctly.

2) I run the .tga through rgbtouyvy. I then create another Avisynth script using only RawSource to load the newly created .yuv. I use VirtualDub to load the Avisynth script, the same as in 1. This time, the RGB comes out in VirtualDub as 255,0,0.

3) If I keep going and encode the .yuv to MPEG2, decode it using the reference decoder, and load the .tga file back into Photoshop, it shows 255,0,0. I thought Photoshop might be messing around with the RGB values, but it looks like it is working correctly.

It looks like rgbtouyvy is converting the video level to PC level. I'm sure that this is by design and I'm just using the program incorrectly. What am I doing wrong?

I checked using the reference decoder and the pattern was decoded incorrectly. I think I found where the problem is. I am using a 100% red window. This was created in Photoshop with RGB values 235,16,16. I save that image in 24 bit .tga.

1) I create an Avisynth script using ImageSource to load the .tga file directly into VirtualDub. No other processing is used. The levels are correct when loaded into VirtualDub (235,16,16). This indicates Photoshop is saving it out correctly.

2) I run the .tga through rgbtouyvy. I then create another Avisynth script using only RawSource to load the newly created .yuv. I use VirtualDub to load the Avisynth script, the same as in 1. This time, the RGB comes out in VirtualDub as 255,0,0.

3) If I keep going and encode the .yuv to MPEG2, decode it using the reference decoder, and load the .tga file back into Photoshop, it shows 255,0,0. I thought Photoshop might be messing around with the RGB values, but it looks like it is working correctly.

It looks like rgbtouyvy is converting the video level to PC level. I'm sure that this is by design and I'm just using the program incorrectly. What am I doing wrong?

Awesome. Did you get a message from alluringreality? We are working together on a new HD DVD disc and wanted to include a few of your patterns. Would that be ok? We are already going to credit you and link your site for allowing us to use rgbtouyvy, but we wanted to double check on the patterns. That is a great little program. It would have taken us forever to get that conversion right without it. I'm definetely interested in looking at the source for those patterns to learn some things.

If it is ok to use your patterns let us know. We have generated most of them ourselves, but there are some miscellanous patterns that we could use from w6rz.net that would really save us some time. Thanks!

Awesome. Did you get a message from alluringreality? We are working together on a new HD DVD disc and wanted to include a few of your patterns. Would that be ok? We are already going to credit you and link your site for allowing us to use rgbtouyvy, but we wanted to double check on the patterns. That is a great little program. It would have taken us forever to get that conversion right without it. I'm definitely interested in looking at the source for those patterns to learn some things.

If it is ok to use your patterns let us know. We have generated most of them ourselves, but there are some miscellanous patterns that we could use from w6rz.net that would really save us some time. Thanks!

No problem using any of the patterns. Here's all the .yuv and source files:

Do'h! I made a mistake with the bars709.yuv file. I uploaded an old version that I use with my hardware encoder that compensates for a bug with that encoder. YCbCr values above 128 have one added them.

The bars709.yuv files has been fixed in both the direct link and the .zip file.

Do'h! I made a mistake with the bars709.yuv file. I uploaded an old version that I use with my hardware encoder that compensates for a bug with that encoder. YCbCr values above 128 have one added them.

The bars709.yuv files has been fixed in both the direct link and the .zip file.

Ron

Ron,
We are making good progress on the new disc. Would you be willing to verify a few of our patterns? We want to get a third set of eyes on them before we beta test it. If you would be willing, I will PM you download information.
Thanks in advance,
Casey

Ron,
We are making good progress on the new disc. Would you be willing to verify a few of our patterns? We want to get a third set of eyes on them before we beta test it. If you would be willing, I will PM you download information.
Thanks in advance,
Casey

Ron,
This may be a little off topic, but I figure you will know the answer. One of the software encoders I have been using (QuEnc) has produced buffer underflows in several of the clips it has encoded. The encoder actually realizes that it is happening and produces a warning. The result is a flashing pattern that stutters.

What exactly is happening here? It is my understanding that SD and HD devices generally use two different buffer sizes (HD being larger), but what is the buffer for? Does it just store the decoded values from disc until they are ready for display? What would cause it to underflow, the bitrate being too low?

Ron,
This may be a little off topic, but I figure you will know the answer. One of the software encoders I have been using (QuEnc) has produced buffer underflows in several of the clips it has encoded. The encoder actually realizes that it is happening and produces a warning. The result is a flashing pattern that stutters.

What exactly is happening here? It is my understanding that SD and HD devices generally use two different buffer sizes (HD being larger), but what is the buffer for? Does it just store the decoded values from disc until they are ready for display? What would cause it to underflow, the bitrate being too low?

It's called the VBV (Video Buffering Verifier). It's used to allow for frame by frame variability in the bitrate. In other words, you can have big I-frames that instantaneously exceed the peak bitrate.

For SD, the size of the VBV is usually 1835008 bits. For HD, the size is typically 9781248 bits, although for ATSC, it must be less than or equal to 7995392 bits.

The VBV is emptied at the peak bitrate which is signaled in the sequence header. In the test clips I downloaded from your ftp site, the VBV is set to 1835008 and the bitrate is set to 9.8 Mbps. It should be 9781248 bits and 29.4 Mbps for HD DVD or 40 Mbps for Blu-ray.

It's called the VBV (Video Buffering Verifier). It's used to allow for frame by frame variability in the bitrate. In other words, you can have big I-frames that instantaneously exceed the peak bitrate.

For SD, the size of the VBV is usually 1835008 bits. For HD, the size is typically 9781248 bits, although for ATSC, it must be less than or equal to 7995392 bits.

The VBV is emptied at the peak bitrate which is signaled in the sequence header. In the test clips I downloaded from your ftp site, the VBV is set to 1835008 and the bitrate is set to 9.8 Mbps. It should be 9781248 bits and 29.4 Mbps for HD DVD or 40 Mbps for Blu-ray.

Ron

Ok, that makes sense. I guess QuEnc is setting the VBV and bitrate based on SD since it is primarily intended for DVD compression. It must not change them based on the input resolution. Two questions:

1) Can we change the sequence header to the correct VBV and bitrate without re-encoding (using a hex editor maybe)?
EDIT: I tried hex editing one of the files, but I can't find any good information on how the bitstream is formatted. I know what flags the sequence header contains and the size of each flag, but I don't understand how the sequence headers are placed. Is there one sequence header at the beginning of the file? From what I have read, there seems to be a sequence header for every GOP. If that is the case, then I also don't understand what the start and end codes should be to define the start and end of a sequence header so that I can find them in the stream. Can you point me in the right direction on this? I apologize for being such a moron, but I have searched and searched and can't find any conclusive information.

Ok, that makes sense. I guess QuEnc is setting the VBV and bitrate based on SD since it is primarily intended for DVD compression. It must not change them based on the input resolution. Two questions:

1) Can we change the sequence header to the correct VBV and bitrate without re-encoding (using a hex editor maybe)?
EDIT: I tried hex editing one of the files, but I can't find any good information on how the bitstream is formatted. I know what flags the sequence header contains and the size of each flag, but I don't understand how the sequence headers are placed. Is there one sequence header at the beginning of the file? From what I have read, there seems to be a sequence header for every GOP. If that is the case, then I also don't understand what the start and end codes should be to define the start and end of a sequence header so that I can find them in the stream. Can you point me in the right direction on this? I apologize for being such a moron, but I have searched and searched and can't find any conclusive information.

2) Did you find anything else wrong?

Thanks again,
Casey

It looks like you got some good help over on doom9. I believe hank315 is directly involved with HCenc, so he's a great resource for you.

At this point, it seems like re-encoding your clips would be the best bet. However, if you want to try and patch a file, use the replace all function in the hex editor with these hex strings:

It looks like you got some good help over on doom9. I believe hank315 is directly involved with HCenc, so he's a great resource for you.

At this point, it seems like re-encoding your clips would be the best bet. However, if you want to try and patch a file, use the replace all function in the hex editor with these hex strings:

Find:
00 00 01 B3 78 04 38 34 17 ED 23 80

Replace with:
00 00 01 B3 78 04 38 34 47 C7 32 A8

Otherwise, the encodes look good. YCbCr levels are accurate.

Ron

Thanks Ron, alluringreality and myself really appreciate your help. Hank has helped me tremendously. I was able to track down the flags, but you are correct, it is going to be much easier (and better in the long run) to re-encode the clips. The link to the specs helped my understanding a lot.

I'm not sure if you noticed in the doom9 posts or not, but do you know why Hank is using 24500 for the bitrate in HCenc? Is that for ATSC broadcast material or something?

Thanks Ron, alluringreality and myself really appreciate your help. Hank has helped me tremendously. I was able to track down the flags, but you are correct, it is going to be much easier (and better in the long run) to re-encode the clips. The link to the specs helped my understanding a lot.

I'm not sure if you noticed in the doom9 posts or not, but do you know why Hank is using 24500 for the bitrate in HCenc? Is that for ATSC broadcast material or something?

EDIT: Bytes 8-11 of his sequence header are as follows: 17 ED 32 A9

Thanks again,
Casey

The bitrate field in MPEG-2 is divided by 400. So 24500 is really 24500 * 400, or 9.8 Mbps.

The A9 byte versus the A8 byte is due to HCenc using a non-intra quantiser matrix.

The bitrate field in MPEG-2 is divided by 400. So 24500 is really 24500 * 400, or 9.8 Mbps.

The A9 byte versus the A8 byte is due to HCenc using a non-intra quantiser matrix.

Ron

Won't that cause the buffer to empty more often than it really needs to? For an encode of "real" material (not test patterns), it seems like it would be emptying all the time because the entire clip would most likely be over 9.8 Mbps. That's a problem isnt it?

Won't that cause the buffer to empty more often than it really needs to? For an encode of "real" material (not test patterns), it seems like it would be emptying all the time because the entire clip would most likely be over 9.8 Mbps. That's a problem isnt it?

If the max. bitrate shows 9800 then the average of the whole clip will be lower, I don't see a problem here.
For a more extensive explanation how the VBV buffer works, please read the ITU H.262 document starting at annex C, clause C.3.2 which is applicable for HCenc.

About your sequence header, this is what I can make of it:
First part of the header:
78 04 38 34 17 ED 32 A9
splitted out in decimal:
1920 1080 3 4 24500 597 0 0 1

Won't that cause the buffer to empty more often than it really needs to? For an encode of "real" material (not test patterns), it seems like it would be emptying all the time because the entire clip would most likely be over 9.8 Mbps. That's a problem isnt it?

Yes, it's too low. You always want to set the sequence header bitrate to something over the peak bitrate in the bitstream. For VBR, it really doesn't matter how much over. You'll often see HD bitstreams with sequence header bitrate values of 65 Mbps or 80 Mbps.

BTW, the VBV is modeled "upside down". Encoded bits empty the VBV and it's filled at the sequence header bitrate. An underflow is really too many bits from the encoder, while an overflow is not enough (for CBR only).

The VBV starts out full at 1835008 bits. The first frame (an I-frame) is 865880 bits, and this brings the VBV to 1835008 - 865880 = 969128 bits. Then the VBV fills at the sequence header bitrate for one frame. 9.8 Mbps / 29.97 = 326994 bits. This brings the VBV to 969128 + 326994 = 1296122 bits. And so on and so forth. Note that 9.8 Mbps / 29.97 is really 326993.66 bits, and you do have to keep track of these fractional bits.

For VBR streams, we don't care if the VBV overflows. When it does, it's just set back to full.

My apologies to hank315, I misunderstood how HCenc worked. I thought that it would switch between 9.8 Mbps and 29.4 Mbps based on input resolution. What it actually does is code whatever max bitrate you give it. For example, if I set 15 Mbps max bitrate and 6 Mbps avg. bitrate, then it codes 15 Mbps into the stream, so it is working correctly. The VBV also seems to be correct.

I encoded two clips using the same Avisynth script and as close to the same settings between HCenc and QuEnc as possible. The bad news is that the Toshiba XA2 we are testing on will not play any of the HCenc encoded files for some reason (just gives an error code and won't read the disc). On the other side, QuEnc produces buffer underflows when I mux using TMPGEnc. The resulting video stutters as the buffer drops.

Below is the output from Ron's program to analyze the sequence headers. It looks like QuEnc might be underflowing because the VBV is too small. It actually sets the bitrate the same way HCenc does. Both clips were encoded at a max bitrate of 15000, avg. of 6000.

I'm going to try using QuEnc by patching the VBV and seeing if it plays smoothly.

hank315, do you have any ideas why the HCenc encode would not play? It is being authored using Ulead Movie Factory to HD DVD, burned to DVD+R, and tested on XA2. We can get you an ISO or anything else you want if you would like to take a look. I would still like to use HCenc for the entire disc, but we can't figure out why it will not play. Thanks in advance for any help you could give, and I apologize again for giving bad information.