Dial-up bitrate listening test

I agree that 32 Kbps is an interesting data point, both for modem audio streaming, and for the audio track of a video stream at moderate broadband data rates.

A few points:Might be interesting to hear LAME using its low bitrate tuned modes. No 32 Kbps preset

QDesign testing will need to be done with the full QDesign Music Pro 2 version, not the free one with QuickTime. However, that codec is largely deprecated for QuickTime use these days in favor of Apple's MPEG-4 AAC-LC codec.

For RealAudio, we should probably test both RealAudio 8 Stereo Music (aka "cook") and the new RealAudio 10 codec (AAC-LC based). Note the RA8 stereo codec is way better than the RA mono codec, even with mono source.

The "sweet spot" KHz for optimal quality might vary from codec to codec. I'll argue it's better to pick the right sample rate for each codec in order to let each codec show its best.

And if this goes well, maybe we should do another test at 16 Kbps. There's a lot of interest there for audio for modem rate video streams, and for mobile devices and streaming radio. The WMA Voice codec does nicely there.

Dial-up bitrate listening test

For RealAudio, we should probably test both RealAudio 8 Stereo Music (aka "cook") and the new RealAudio 10 codec (AAC-LC based).

The Real AAC codec doesn´t go below 64kbps, so the only codec that can be used at this bitrate (32 or 48 kbps) is cook (ok, there was dnet (DolbyNET, AC3 with low samplerate extension) once, but that codec isn´t featured anymore for years). But still there are four flavors that could be used:

Dial-up bitrate listening test

AFAIK, even though the modem has a higher bandwidth than 32kbps and the protocol is usually UDP there is still some overhead bandwidth which knocks several kbps onto the stream. (Correct me if I'm wrong)

Shoutcast/Icecast use TCP. But yes, there is still overhead either way--TCP adds a bit more.

Dial-up bitrate listening test

And if this goes well, maybe we should do another test at 16 Kbps. There's a lot of interest there for audio for modem rate video streams, and for mobile devices and streaming radio. The WMA Voice codec does nicely there.

That's exactly what I thought too. I hope this test will be done at 32 Kbps, 44 Khz stereo and it will be great if we could have another test at 16 Kbps.

Dial-up bitrate listening test

Right. I don't think anyone on his right mind would encode audio to 32kbps at 44.100Hz. I am considering either resampling everything to 22050 (I guess that's what people encoding to this bitrate would do), or letting the encoders choose the appropriate sampling rate.

Quote

Are any VBR/ABR settings going to be used at this low bitrate?

Vorbis and AAC will use VBR, unless I'm advised todo otherwise. QDesign and Real, of course, won't. I'm not sure about WMA. It seems to me it doesn't offer two-pass VBR for sampling rates other than 44.1 and 48kHz...?

Quote

If so the samples should give bitrates below 32kbps/48kbps and above 32kbps/48kbps. Ideally in the shape of a bell curve!

Dial-up bitrate listening test

First I took 18 tracks, one from each of the 18 albums I encoded for the 128 kbps multi-format test. Each of these tracks averaged about the same as the entire album did. Then I threw out highest, lowest, and cut from the top until I had 13 tracks which averaged 128 kbps using -q 4.35. The actual bitrates were:

Dial-up bitrate listening test

Okay, this started out as a 48kbps test and everybody complained that you can't actually stream 48kbps on most dial-up connections. So that is obviously the most important part of this test. (streamability on a dialup connection) So why use VBR for any of the samples then? VBR + streaming is not a good idea, especially if the person's buffer is not very large (usually not, people don't like to wait.) VBR on low bandwidth connections is an awful idea because when it peaks to 64 or 96kbps you may experience dropouts. Just my 2c.

Dial-up bitrate listening test

Okay, this started out as a 48kbps test and everybody complained that you can't actually stream 48kbps on most dial-up connections. So that is obviously the most important part of this test. (streamability on a dialup connection) So why use VBR for any of the samples then? VBR + streaming is not a good idea, especially if the person's buffer is not very large (usually not, people don't like to wait.) VBR on low bandwidth connections is an awful idea because when it peaks to 64 or 96kbps you may experience dropouts. Just my 2c.

Just exploring the bitrates and the test assumptions. Assume that there was a vorbis vbr setting that averaged 32 kbps. Would the deviation about that bitrate be small enough to allow it to be considered in a dialup test? If you subtract 6 kbps from each of the bitrates I found, you'd get 40 kbps for the highest track. Seems like it's at least arguable that it could work.

In any case, another setting needs to be found. Bitrate managed may be the only option.

Dial-up bitrate listening test

Not sure I'm upto date on all streaming codecs, but some things to consider:

1) Why not hard limit upper range of bit rate?

I think that would be a relatively fair assumption for streaming conditions. Using VBR and then calculating arbitrary averages isn't necessarily going to work under real streaming conditions. If average path is chosen, then it'll be more like a "pseudo streaming" codec test, no?

So, would CBR or hard-limited max bit rate be a better choice, considering this is a streaming test and that the results should reflect that (and not pseudo-idealized conditions)?

2) I also vouch for using at least two langauges (in speech samples) if possible.

This is not about trying to represent all the world languages, but trying to find for example languages that have widely differing fricative frequency/type and intonation. These can throw different problems for different codecs. I'd suggest Engl/Ger as minimum two myself. Spanish is much lower in frequency on certain type of hard fricatives as is French.

Excellent test idea, btw! Thanks again to rjamorim for being brave by facing the task (again!).

Dial-up bitrate listening test

Yes, please remember that at bitrates this low even a few kbps deviation from 32kbps (CBR/ABR, or average if we decide to use VBR) will make a large difference to the final output, and file size/bitrate IS very important here, as opposed to when testing files which are targeting a quality, not a bitrate.

Dial-up bitrate listening test

Dunno the case with speex, but most voice coders fail VERY badly on music and complex sound effects.

Spees does fail "VERY badly" on music, only "badly" ;-) Seriously, at CBR ~34 kbps, it's possible to have recognizable music with Speex, but the quality is definitely not on par with what you'd get for Vorbis. Probably not worth including in this test.

Quote

I think Speex belongs to a test dedicated to vocodecs. I already started discussing such test with jmvalin, hopefully he'll have time to conduce it (or someone else might become interested).

Don't have much time to organize it, but I can provide tips on what codecs/settings to use.

Dial-up bitrate listening test

I know there are already six encoders being used in this test, and even though this has been the limit in the past would it be possible to include musepack using the telephone preset (quality 2) in this test? It may not be as well supported as the others, but musepack has done suprisingly well in the 64kbps test, maybe it can do well at 32kbps too.