One could modify the decoder to output 24 bit samples where the least significant 4 bits are zero (effective resolution: 20 bits) so it'll compress better using FLAC and/or WavPack.

Since the HDCD decoder attenuates the signal at most by 13.5 dB which translates to 2.25 bits, you won't need more than 20 bits to represent the signal faithfully. So quantizing to 20 bits makes sense and won't hurt (won't effect the noise floor noticably).

Still, what's the point on decoding for lossless storage? Guys! Think about it. I'd keep the 16 bit signal as is and use a HDCD capable player instead!

QUOTE

Interesting result this, the 24bit HDCD decoded version compresses to a size even smaller than the original 16bit.

...which can only mean that your CD doesn't make use of all HDCD features (no "soft dynamics compression").

One could modify the decoder to output 24 bit samples where the least significant 4 bits are zero (effective resolution: 20 bits) so it'll compress better using FLAC and/or WavPack.

Since the HDCD decoder attenuates the signal at most by 13.5 dB which translates to 2.25 bits, you won't need more than 20 bits to represent the signal faithfully. So quantizing to 20 bits makes sense and won't hurt (won't effect the noise floor noticably).

Still, what's the point on decoding for lossless storage? Guys! Think about it. I'd keep the 16 bit signal as is and use a HDCD capable player instead!

Hi guys, great thread. I found it earlier and have been posting a lot to the Hoffman forum about my findings, using both the WMP plugin and hdcd.exe.

As for not using lossless storage, that means storing them as 16-bit WAVs; the only HDCD software player I know is player is WMP, which does not recognize wavs as HDCD. You either have to play an actual disc, or mount a virtual one, for WMP to recognize it at HDCD.

Or are you thinking of some other mechanism?

Regarding the two software decoders, I find that they produce almost identical results - I ripped a Joni Mitchell HDCD to my drive with EAC, then mounted it with Daemon tools for WMP (with the WAV plugin and 24-bit playback active), or else fed the .wavs to hdcd.exe. FOr a given track the two resulting .wav files almost null out, though there is tiny, residual signal at -88 dBFS or so, at a few spots (bit comparison with foobar says there are something like ~700 nonidentical samples, in one track, for example). I couldn't tell them apart in an ABX.

Regarding final bit depths, I find that the HDCDs which do NOT use peak extension, and which are highly compressed/limited, so far always decode to '17 bit' files , reduced by 6 dB (examples: Beck and Van Halen HDCDs). THose that use peak extend decode to 20 bit files (e.g. Yes, Joni Mitchell), though some that have no apparent peak extension, but are not super compressed dynamically, also decode to 20 bit files (e.g. King Crimson), with lesser amounts of level reduction. see this post for examples.

Lastly, I find that foobar2000 (v 0.9.4.4) cannot play decoded files made by the WMP plugin (which appear to be in 'wavex' format, and elicit a WAVEXTENSIBLE tag from flac.exe when I compress them) wheras it can play the 24 bit files produced by hdcd.exe, and flacs made from them. Unlike the older foobar, the newer one doesn't have options for setting playback bit depth, as best I can tell.

All feedback appreciated! I'm especially interested to know if the 'low resolution extension' and selectable filter options are actually used by either decoder (I've also got a hardware HDCD player and plan to compare its analog output to the digital 'rips')

QUOTE (Christopher Key @ Aug 27 2007, 16:17)

I'll have a go tomorrow at adding some more diagnostics, maybe a --verbose option, which will list which features get used after the track's been processed. I'd guess:

Peak extend enabled at some point during the audioPeak extend disabled at some point during the audioPeak extend used (|sample|>0x5980 while peak extend used)Range of gain adjustment values used.

From what I've read (e.g. John Marks' self-interview about his HDCD releases) it appears that the HDCD A/D converter adds the flag that will activate the HDCD logo even if NONE of the processing options are used during the mastering...in other words, if 'HDCD' technology us just used for its high-quality A/D conversion. So WMP would light up the HDCD logo, but simply attenuate these by 6 dB, as it does for all input when the 24 bit option is active (I recall this being a mandate from Pacific Microsonics originally).

Hi guys, great thread. I found it earlier and have been posting a lot to the Hoffman forum about my findings, using both the WMP plugin and hdcd.exe.

As for not using lossless storage, that means storing them as 16-bit WAVs; the only HDCD software player I know is player is WMP, which does not recognize wavs as HDCD. You either have to play an actual disc, or mount a virtual one, for WMP to recognize it at HDCD.

Or are you thinking of some other mechanism?

My view is that theoretically, you should only ever store the 16 bit master. That can then be played back as 16 bit or 24 bit dependent upon the listening environment. In practice however, until all my 'hifi' playback devices support hdcd, I'll probably have 16 bit and 24 bit versions in my library.

QUOTE (SiriusB @ Aug 30 2007, 03:38)

Regarding the two software decoders, I find that they produce almost identical results - I ripped a Joni Mitchell HDCD to my drive with EAC, then mounted it with Daemon tools for WMP (with the WAV plugin and 24-bit playback active), or else fed the .wavs to hdcd.exe. FOr a given track the two resulting .wav files almost null out, though there is tiny, residual signal at -88 dBFS or so, at a few spots (bit comparison with foobar says there are something like ~700 nonidentical samples, in one track, for example). I couldn't tell them apart in an ABX.

That's rather worrying. The software was designed to be bit identical to wmp, and is for my 5 test cds. I've rechecked them all using the following method:

Rip CD using EAC. (Action -> Copy Image and Create Cue Sheet -> Uncompressed)Manually edit the cue sheet to remove all but the first track entry (effectively create a single track that spans the entire CD)Run hdcd on this file (hdcd.exe -o hdcd_hdcdN.wav hdcd.wav)Run wmp on this file and capture the output with Chronotron.Compare the two.

Apart from the fact that the wmp version is exactly one frame short, the two match exactly. (Not strictly true. A few times I've had the one or two samples not match, but they've always it's always been out by a power of 2, and have matched exactly if retested. I'm inclined not to trust the memory in the machine doing the testing.)

Regarding final bit depths, I find that the HDCDs which do NOT use peak extension, and which are highly compressed/limited, so far always decode to '17 bit' files , reduced by 6 dB (examples: Beck and Van Halen HDCDs). THose that use peak extend decode to 20 bit files (e.g. Yes, Joni Mitchell), though some that have no apparent peak extension, but are not super compressed dynamically, also decode to 20 bit files (e.g. King Crimson), with lesser amounts of level reduction. see this post for examples.

Tracks not using peak extend are simply right shifted by one bit, and are padded to 24bit with 7 zeroes. The peak extend table is specified with 20 bit precision, hence these tracks show up as 20 bit. Any tracks using gain adjustment (low level resolution extension?) will also show up as 20 bit.

QUOTE (SiriusB @ Aug 30 2007, 03:38)

Lastly, I find that foobar2000 (v 0.9.4.4) cannot play decoded files made by the WMP plugin (which appear to be in 'wavex' format, and elicit a WAVEXTENSIBLE tag from flac.exe when I compress them) wheras it can play the 24 bit files produced by hdcd.exe, and flacs made from them. Unlike the older foobar, the newer one doesn't have options for setting playback bit depth, as best I can tell.

From some very brief research, it appears that wavex (audio_format = FFFE) is simply a normal WAV, with an extra 24 bytes in the header that specify mapping from the channels in the file to playback channels. If you're so inclined, this can be fixed with a hex editor.

QUOTE (SiriusB @ Aug 30 2007, 03:38)

All feedback appreciated! I'm especially interested to know if the 'low resolution extension' and selectable filter options are actually used by either decoder (I've also got a hardware HDCD player and plan to compare its analog output to the digital 'rips')

There are 3 unidentified bits in the hdcd codes that might be used to switch filters. My test cd's had these permanently set to zero, and wmp didn't appear to do anything with them. I'd be very grateful to hear from anyone who finds a CD in which they're used.

Both wmp and hdcd do allow the output level to be attenuated by up to 7.5dB (in .5dB steps). Is this what's referring to as 'low level extension'?

QUOTE (SiriusB @ Aug 30 2007, 03:38)

From what I've read (e.g. John Marks' self-interview about his HDCD releases) it appears that the HDCD A/D converter adds the flag that will activate the HDCD logo even if NONE of the processing options are used during the mastering...in other words, if 'HDCD' technology us just used for its high-quality A/D conversion. So WMP would light up the HDCD logo, but simply attenuate these by 6 dB, as it does for all input when the 24 bit option is active (I recall this being a mandate from Pacific Microsonics originally).

That certainly sounds sensible from a marketing point of view. As an end user, I'd probably be rather puzzled if the decoder didn't light up when playing material sold as hdcd encoded.

My view is that theoretically, you should only ever store the 16 bit master. That can then be played back as 16 bit or 24 bit dependent upon the listening environment. In practice however, until all my 'hifi' playback devices support hdcd, I'll probably have 16 bit and 24 bit versions in my library.

Theoretically very desirable, but practically not do-able in any easy way....so far. I'd love to see a foobar hdcd plugin. (Though 24-bit files of maybe a dozen HDCDs is not a big issue for me, storage-wise.)

QUOTE

That's rather worrying. The software was designed to be bit identical to wmp, and is for my 5 test cds. I've rechecked them all using the following method:

Rip CD using EAC. (Action -> Copy Image and Create Cue Sheet -> Uncompressed)Manually edit the cue sheet to remove all but the first track entry (effectively create a single track that spans the entire CD)Run hdcd on this file (hdcd.exe -o hdcd_hdcdN.wav hdcd.wav)Run wmp on this file and capture the output with Chronotron.Compare the two.

Apart from the fact that the wmp version is exactly one frame short, the two match exactly. (Not strictly true. A few times I've had the one or two samples not match, but they've always it's always been out by a power of 2, and have matched exactly if retested. I'm inclined not to trust the memory in the machine doing the testing.)

It was Joni Mitchell 'Mingus'. FWIW I do my EAC rips as track by track, not album image + cue -- this was done long ago, before any of the hdcd--> wav tools were around. To mount the Mingus files in Daemon Tools for WMP, I used foobar2k to 'retrospectively' generate an album .wav + cue from the set of Mingus tracks. For hdcd.exe of course all I have to do is feed it the individual .wavs directly. I can do it all again and compare the files. Also I can send you the 16-bit master file of track or tracks from the HDCD, and you can check it, if you like.

QUOTE

Tracks not using peak extend are simply right shifted by one bit, and are padded to 24bit with 7 zeroes. The peak extend table is specified with 20 bit precision, hence these tracks show up as 20 bit. Any tracks using gain adjustment (low level resolution extension?) will also show up as 20 bit.

Well, again, all I can report is that so far, the tracks that did NOT use peak extension, and show only a -6db crush, do NOT show up as 20 bit -- Audition calls them '17 bit'. The Beck 'Midnite Vultures' HDCD for example. Here's the Audition stats for track 1 , 'Sexx Laws', undecoded and hdcd.exe decoded. (please excuse the lousy formatting)

From some very brief research, it appears that wavex (audio_format = FFFE) is simply a normal WAV, with an extra 24 bytes in the header that specify mapping from the channels in the file to playback channels. If you're so inclined, this can be fixed with a hex editor.

Thanks. I'd much rather just use hdcd.exe, so long as it's creating files that are identical in all the ways that matter, to what the WMP WAV plugin generates.

QUOTE

There are 3 unidentified bits in the hdcd codes that might be used to switch filters. My test cd's had these permanently set to zero, and wmp didn't appear to do anything with them. I'd be very grateful to hear from anyone who finds a CD in which they're used.

Check out the link I made, to my Steve Hoffman forum post There I document a variety of hdcd behavior -- ranging from simple 6dB crush, all the way to ~6 dB of added 'dynamic range' (crest factor). How can I tell if the 3 bits are in use?

QUOTE

Both wmp and hdcd do allow the output level to be attenuated by up to 7.5dB (in .5dB steps). Is this what's referring to as 'low level extension'?

Beats me. I've got some technical papers on HDCD waiting to be read. Sebastian might know already.So far the HDCD options I can keep straight in my head are 1) peak extension 2) low resolution extension3)selectable filters at A/D (I have yet to determine if these are applied at decoding too.. there was a huge flamewar about it on Audio Asylum between the maker of Ayre Cd players, who is very anti-HDCD, and Chris Tham, who isn't totally reliable on digital matters). Maybe there's more options?

QUOTE

That certainly sounds sensible from a marketing point of view. As an end user, I'd probably be rather puzzled if the decoder didn't light up when playing material sold as hdcd encoded.

IMO, simply being converted from analog to digital, even by an admittedly high-quality converter, doesn't really qualify as 'hdcd encoding'. But it is good marketing for the converter, to brand these CDs as HDCD, and have them trigger the HDCD light.

Mandating that players licensed for HDCD lower all non-HDCD output by 6 dB is also good marketing, from a psychoacoustic POV -- people tend to think louder tracks sound better than quieter ones.

What's funny is that there are CDs out there that light up the HDCD decoder, but aren't marked as HDCD on their packaging. At least one of these even uses peak extension (Mahavishnu Orchestra 'Lost Trident Sessions').

My view is that theoretically, you should only ever store the 16 bit master. That can then be played back as 16 bit or 24 bit dependent upon the listening environment. In practice however, until all my 'hifi' playback devices support hdcd, I'll probably have 16 bit and 24 bit versions in my library.[/quote]

Theoretically very desirable, but practically not do-able in any easy way....so far. I'd love to see a foobar hdcd plugin. (Though 24-bit files of maybe a dozen HDCDs is not a big issue for me, storage-wise.)[/quote]

Hopefully plugins will become available. Storage isn't really the problem, HDCDs will always be a very small percentage of one's music library, it's more the incovenience of having two versions available stored, keeping them both tagged correctly, making the correct one gets selected etc. It depends on your particular set up as to whether this is a problem

[quote][quote]That's rather worrying. The software was designed to be bit identical to wmp, and is for my 5 test cds.

Which CD was it that wasn't matching for you?[/quote]

It was Joni Mitchell 'Mingus'. FWIW I do my EAC rips as track by track, not album image + cue -- this was done long ago, before any of the hdcd--> wav tools were around. To mount the Mingus files in Daemon Tools for WMP, I used foobar2k to 'retrospectively' generate an album .wav + cue from the set of Mingus tracks. For hdcd.exe of course all I have to do is feed it the individual .wavs directly. I can do it all again and compare the files. Also I can send you the 16-bit master file of track or tracks from the HDCD, and you can check it, if you like.[/quote]

If you get a chance, would you mind rechecking just one of the tracks. You may need to watch out for sample offsets, I did find that wmp rather bizarrely advanced to first 2m by 4 samples, then corrected itself. I've not been abl to get it to do it again however.

I'll see if I can acquire a copy of 'Mingus', along with a few others mentioned later to test this myself.

[quote][quote]Tracks not using peak extend are simply right shifted by one bit, and are padded to 24bit with 7 zeroes. The peak extend table is specified with 20 bit precision, hence these tracks show up as 20 bit. Any tracks using gain adjustment (low level resolution extension?) will also show up as 20 bit.[/quote]

Well, again, all I can report is that so far, the tracks that did NOT use peak extension, and show only a -6db crush, do NOT show up as 20 bit -- Audition calls them '17 bit'. The Beck 'Midnite Vultures' HDCD for example. Here's the Audition stats for track 1 , 'Sexx Laws', undecoded and hdcd.exe decoded. (please excuse the lousy formatting)

That's exactly as expected. If neither peak extend nor gain adjustment are used, the least significant seven bits of each sample will always be zero (as will the one most significant), hence the bit depth is detected as 17 bits. I don't know why the peak amplitude values aren't exactly 6dB lower in the decoded case, (0.09dB -> 6.11dB, not just the way the values are rounded), but the sample values are *exactly* halved.

[quote][quote]There are 3 unidentified bits in the hdcd codes that might be used to switch filters. My test cd's had these permanently set to zero, and wmp didn't appear to do anything with them. I'd be very grateful to hear from anyone who finds a CD in which they're used.[/quote]

Check out the link I made, to my Steve Hoffman forum post There I document a variety of hdcd behavior -- ranging from simple 6dB crush, all the way to ~6 dB of added 'dynamic range' (crest factor). How can I tell if the 3 bits are in use?[/quote]

Thanks for the link, you initial post covers the behaviour nicely. Category 3 and 4 are essentially the same from a technical point of view, they both have peak extend enabled, category 4 has larger sample value before decoding that go further up the expansion curve. Given that I've got the data, I really should produce an accurate plot of the curve to demonstrate how it works.

[quote][quote]Both wmp and hdcd do allow the output level to be attenuated by up to 7.5dB (in .5dB steps). Is this what's referring to as 'low level extension'?[/quote]

Beats me. I've got some technical papers on HDCD waiting to be read. Sebastian might know already.So far the HDCD options I can keep straight in my head are 1) peak extension 2) low resolution extension3)selectable filters at A/D (I have yet to determine if these are applied at decoding too.. there was a huge flamewar about it on Audio Asylum between the maker of Ayre Cd players, who is very anti-HDCD, and Chris Tham, who isn't totally reliable on digital matters). Maybe there's more options?[/quote]

Mentions those three, and only those three, so I suspect that those are the options. I'll try to use their terminology from now on to avoid confusion.

Interestingly, the low level range extension referred to there (20dB in 4dB steps) is much different to and stronger that used by wmp (7.5dB in 0.5 dB steps).

I did catch the AA thread but got lost fairly quickly as a result of the strength of their convictions.

[quote][quote]

That certainly sounds sensible from a marketing point of view. As an end user, I'd probably be rather puzzled if the decoder didn't light up when playing material sold as hdcd encoded.[/quote]

IMO, simply being converted from analog to digital, even by an admittedly high-quality converter, doesn't really qualify as 'hdcd encoding'. But it is good marketing for the converter, to brand these CDs as HDCD, and have them trigger the HDCD light.

Mandating that players licensed for HDCD lower all non-HDCD output by 6 dB is also good marketing, from a psychoacoustic POV -- people tend to think louder tracks sound better than quieter ones.

What's funny is that there are CDs out there that light up the HDCD decoder, but aren't marked as HDCD on their packaging. At least one of these even uses peak extension (Mahavishnu Orchestra 'Lost Trident Sessions').[/quote]

Quite agree, particularly the loudness point. You can justify the behaviour from a technial point of view, if normal CDs weren't attenuated, and an hdcd code was spuriouusly detected during playback, you'd get a sudden jump in volume. On the other hand, when they are already attenuated by 6dB, a spuriusly detected hdcd code will just give you slightly more dynamic for a short period, which is much less offensive. I doubt this is the primary motivation however!

I'll try and get a copy of those that have the unknown codes. The 0x03 means that at some point, the decoder has seen both 0b000 and 0b001 in the upper three bits of the hdcd codes. This could well be for filter switching, although I couldn't see wmp doing anything with these bits so there was no way of knowing. If it does nothing with them, then there's really not a lot I can do, unless I can see the digital output of a hardware decoder that also upscales to 88.2kHz.

[quote](is there any way to get hdcd.exe to generate a log file? I can't select and paste from the cmd window. I'm also looking for a tool that would scan my flac collection for all HDCD tracks, to see if I have 'hidden' ones.)[/quote]

Just redirect stderr to a file. Using the newer version, it should be as simple as,

hdcd.exe -o out.wav in.wav 2>mylog.txt

and with older version that only supports stdin / stdout,

hdcd.exe < in.wav > out.wav 2>mylog.txt

Also, take a look at the slimdevices thread, that's got some guidance on how to use this in a script,

I did a quick scan of my library and was interested to find that all my Mark Knopfler CDs are HDCD, despite not being marked as such, as was *one* track from a Bond themes compilation CD. I'm guessing the compiler probably didn't realise either!!!

I'll try and get a copy of those that have the unknown codes. The 0x03 means that at some point, the decoder has seen both 0b000 and 0b001 in the upper three bits of the hdcd codes. This could well be for filter switching, although I couldn't see wmp doing anything with these bits so there was no way of knowing. If it does nothing with them, then there's really not a lot I can do, unless I can see the digital output of a hardware decoder that also upscales to 88.2kHz.

I've looked again more closely at exactly what wmp does. It does in fact extract and store the third bit from the hdcd codes, but then does absolutely nothing with it. I'm hence guessing that this single bit is used to switch between two alternate filters, which seems to fit in pretty closely with the last paragraph of,

I'm not sure quite what the truth is, but am inclined to think there are only two filters, given that over the sample of tested CDs, the only currently unused bit that was ever set is the one extracted by wmp.

I'd be quite keen to add an option to do the correct upsampling as well. I've managed to extract a little bit of information; the AES paper specifically says that its a symmetric FIR filter, and another link that I've lost for now claims that it has less than 256 taps.

If anyone has access to a hardware decoder with a digital output, it should be possible to calculate the exact tap values for each filter by analysing the output from various suitable inputs. With an analogue decoder, it should be possible to approximate the filters' responses, but getting the exact values won't be possible.

I'm going to go and try to buy some of the CDs known to use this bit to see if I can at least identify which filter is which, and what might cause the encoder to change between the two.

I'm going to go and try to buy some of the CDs known to use this bit to see if I can at least identify which filter is which, and what might cause the encoder to change between the two.

Regards,

Chris

Van Halen's 1984 yields some interesting results. I set up the decoder to output just the value of the third bit from each hdcd code to the left channel, and to leave the right channel untouched. Attached is a screenshot showing the start of track 3, Panama.

This shows bit3 being turned on for a brief period spanning each drum beat during the intro. Although much less clear, a similar pattern is visible throughout the rest of the cd. bit3 is also always turned on for a brief period covering the very start of each track.

My guess is that this is exactly as described in the last paragraph of,

and that setting bit3 is supposed to switch the decoder from the standard, sharp recontruction filter to one with a more gentle roll off that better preserves transients.

Does this seem to make sense?

Chris

Edit: Just done the same experiment with Joni Mitchell - Both Sides Now, and this seems to be quite clearly what's going on. It's not used at all in tracks 4, 6, 12, which seem to feature much softer drum beats, and perfectly follows the percussion at about 2:22 into the first track.

So does this mean you think that the sound of either filter sampling to 88.2 is aren't that different from each other?

Although I don't know the exact specs of the reconstruction filters I feel I can safely say that decoding this (non-)feature isn't worth the trouble.

Indeed, it probably isn't worth the bother of decoding, but it would be nice to know. I've created four wav files, an impulse train and a 10kHz -> 22.05kHz sweep, each set to use both the normal and transient filter:

Sirius, do you think you could pipe these four through your hardware decoder and capture the output. I'd strongly recommend against feeding the decoded signal into anything other than a capture device, as I doubt speakers (amp too???) would fare very well with them!