Synthesia Talk

Nov-30-2018 Notice: New user registrations have been temporarily disabled. The continued onslaught from spammers and the bounced registration emails they produce have been getting our support email address blacklisted. So, we need to let that cool down for a bit. There should be no interruption for already-registered users. New users: please direct your questions/feedback to support@synthesiagame.com. Sorry for the inconvenience!

Yeah, multiple note-heads with correctly stacked accidentals is on the list for Synthesia 11. It's a rather involved topic (scroll down a page or two), so I can't promise it will end up perfect. But, it should end up in a place better than "completely broken" like it is today.

Actually, 10.5 is moving along quite rapidly, especially in the last two weeks. The BASSMIDI-based replacement synth for Windows and Android has been a delight to implement. Their API is very nice and it does most of the work for you, which is the opposite of what I'm used to!

It's much faster and (hopefully once I finish working out licensing with the Voice Crystal folks) I'm hoping to get an upgraded version of the sound set we've been using on the iPad (since 2012) with a higher-quality grand piano. That's a huge upgrade for Windows and something approaching a quantum leap for Android. It's so nice, I'm even considering adding it to the Mac and iPad versions, too, to have a nice, even, high-quality sound across all four platforms.

The latency is so much better, I was hoping to actually quantify it in a repeatable way this time so that the "X% faster" bullet point in the feature list will be based on real data. As a little hobby project in the evenings, I've been cobbling together a Teensy 2.0-based latency tester that will send out a MIDI note (via USB or MIDI port) and start a timer to see how long it takes before its headphone jack picks up any sound. (Most of the work has already been done by Google, but it's still been some fun. I don't get to tinker with hardware nearly as much as I'd like.)

Choosing the software synth on the Settings screen now gives you a new "Reverb" slider which -- used with some restraint -- can add some nice extra depth to the sound. (Without restraint you can make it sound like your piano has been installed in a public restroom!) And, if the background song scanner (that populates the song list) happens to stumble across any SF2 SoundFont files in the same folders, they're shown in a list on the same settings screen and you can choose between and audition them in a single click. That means you can bring your own favorite/preferred SoundFont quite easily, even on Android.

Other work in the meantime: the constant tide of Android bugs continues to assault and be pushed back. I think I may have found and fixed the current #1 source of Android crashes, which also coincidentally resulted in a nice speed/responsiveness improvement on that platform.

Remaining work includes a final bug fix before Chromebook support is complete, automatic latency detection for the new synth on Android (I've still witnessed some drop-outs using BASS's built-in detection, so hopefully we'll be able to layer another detector on top of that or at least include a manual slider), some UI wrap-up for the new built-in synth settings screen, actually measuring some MIDI latency , and the last handful of tiny bugs that have been reported recently.

Then this eight month(?!) "emergency" bug-fix release will be out the door and we'll be able to get back to work on Synthesia 11.

I built a little latency tester prototype and ran it against most of the test hardware I have around here.

There are 1000 milliseconds (ms) in one second. On a typical computer monitor or tablet (running at 60 frames per second), each frame lasts for 1000/60 = 16.7ms.

Procedure

This testing device sends a "Note On: Middle C at max velocity" message through a MIDI cable, starts a timer immediately, and counts the time until the very first evidence of any sound at all coming in from the headphone jack. It can measure that result down to the nearest two microseconds (a thousandth of a millisecond), so we've got about 500x the resolution we need to accurately characterize things at the millisecond level. It continues to send Note On/Off pairs automatically to acquire 25 separate measurements.

Baseline

As a sanity check (and because I was curious ), I tested a few digital pianos directly. Their whole job is to produce sound quickly after a key strike, so presumably they will also respond to MIDI messages quickly. Keyboards should be the fastest things around, so these tests establish a nice baseline for the fastest we should realistically ever hope to see a software synth behave.

Keyboard average time to respond to a MIDI "Note On" message:

6.4ms (800μs std dev) - Casio LK-100

9.9ms (260μs std dev) - Yamaha EZ-200

12.4ms (780μs std dev) - Yamaha P-70

These are delightfully fast (as you'll see) with an amazingly tight distribution. Every one of them produces sound in less time (often substantially) than a single video frame.

More data points that may be interesting:

57.1ms (2.3ms std dev) - Synthesia on iPad 4

57.2ms (30μs std dev) - Synthesia on iPad 2

Results

Computer/tablet average time to respond to a MIDI "Note On" using legacy synths (Windows MME on Windows and Sonivox on Android), with the latency tester connected through an E-MU Xmidi 1x1:

Right away we can see that the legacy synths take ~15-40x longer to produce a note than a keyboard. The Windows synth at a quarter second is especially shameful. The wider spread most-likely has to do with Synthesia's own MIDI polling once per frame. (2x either of those standard deviations is close to a 16.7ms frame.)

Computer/tablet average time to respond to a MIDI "Note On" using the new BASS 2.4.13.8 based synth (WASAPI on Windows and OpenSL on Android), with the latency tester connected through an E-MU Xmidi 1x1:

For computers and "newer" Android, BASS is consistently half to one-third the time it takes the old synths! Four video frames is still inside the range of "imperceptible delay" for me (and hopefully you!), and this synth can use arbitrary SoundFonts and add effects like reverb (without impacting that latency). This is very exciting.

Tripling the frame rate (on Surface Pro 2) appears to have soaked up the "average 8ms in either direction" slop and tightened up the standard deviation (more than should have been expected).

The older the Android device, the less rosy the picture. On devices that don't have an audio "Fast Path", it's neck-and-neck with Sonivox usually winning on speed but BASS easily still winning on sound quality. I've decided to leave it up to users: both synths will be available in the Android version of Synthesia 10.5 (defaulting to the new BASS synth, which is the easy answer on Android 6 and later).

(Checking against my manual measurements from six years ago, I'm happy to see these results are very consistent with those.)

Conclusions

The only software synths that can break into the high 50ms range are Apple's and the broken/unusable UWP synth in Windows 10 at the expense of pops and crackles. So, really just Apple's.

The rule of thumb in the organ world, if I remember correctly, is that 20 mSec is the threshold of perceptible delay. Organists learn to cope with perceptible delay because the electro-pneumatic mechanism used to open a pipe valve takes some time to open from the time an organ key is depressed, a pipe takes some time to "speak" after air starts flowing into the pipe, and the pipes are usually far enough away from the organist that it takes a perceptible length of time for the sound of a pipe to reach the organist.Having a pipe 20 feet from the organist, which is close, is enough to introduce a perceptible delay. I haven't played real pipe organs enough to learn to cope with the delays. I'm told that good organists don't listen to the sound because it will throw you off.

Yeah, playing with units a bit, it looks like sound travels at 1.1 ft/ms. So that pipe at twenty feet is already enough to eat the 20ms.

I suppose when my options are 75ms or 240ms after twelve years of 240ms, I'm pretty happy to take 75!

Certainly the advice (now backed up by real data) will continue to be "use your keyboard's synth if you've got one", but in those cases where a keyboard synth isn't available, things might have reached "passable" now instead of "what do you mean a quarter second?!"

Whew! 10.5 is finally out the door. The final sound set ended up being 20 MB (instead of the 4 MB set used on the iPad). And all 16 MB of new samples are for the main Acoustic Grand Piano instrument. It sounds very nice now.

Yeah, OBS is one of the apps whose overlays seem to crash Synthesia for whatever reason. I haven't been able to track down whether it's Synthesia doing something they don't expect or them doing something Synthesia doesn't expect, but the workaround for now is "don't run them at the same time".

A couple of the things are already finished and released in past updates (see the first page in this topic).

That said, we're finally back on task now that 10.5 is out the door. There are a couple of fires to put out (the Android version always breaks in new and exciting ways) over the next few days and then I finally get to resume the "Multiple Lines of Notation" preview mentioned in the top post.

I'm sorry to interrupt here but I have some problems with 10.5. I use the built-in Midi synth now but unfortunately it causes severe crackling when the volume gets too loudor when there are many instruments. Furthermore, I use Synthesia in full screen mode but when I exit and restart, Synthesia is in window mode again. It doesn't seem to save the full screen.

The idea is that the volume slider on that particular screen is just for subtle adjustments if your particular SoundFont has its samples adjusted too loud/quiet. It's really easy to get clipping there, so the right answer is usually to leave it at 100% and adjust the system volume instead.

That said, what you're describing sounds like something different.

Try this: open the configuration window (hold Shift while launching the app), find the "Synth.BufferExtra" entry near the bottom of the list, and change it to something like 300.

That will add a lot of extra latency, but as a test does it get rid of the buzzing? If it does, you can try smaller numbers. (Even something like 20 or 50 might help stop the buzzing.) The real question will be: if it does solve the problem, why isn't BASS's built-in latency detection working?

I'm using the General User Soundfont because I find the sounds very realistic. I set the Buffer to 250 and there indeed is a latency and I'm afraid to tell youthat the crackling got even worse. But I think that it might be the soundfonts I used. Do you have any recommendations for soundfonts?

Edit: I just remembered something. The CoolSoft Midi Synth could load the soundfont into the RAM before starting a piece. Maybe that would make it better?

Ah ha, that is very interesting. (Especially since it got worse!) More interesting is that GeneralUser isn't particularly large. Maybe 20 MB or so, right?

When you switch back to the "Built-in sounds by Voice Crystal", do you still get crackling? Or is it only happening with outside SoundFonts?

And, one last thing to check: there is a much newer, unofficial version of BASS (24 revisions newer than the one Synthesia 10.5 shipped with). You can download it, here: http://www.un4seen.com/stuff/bass.zip

If you copy that bass.dll over the one in the Synthesia folder (with Synthesia closed), does that fix the crackling?

I tried the other Bassmidi version now and there's much less crackling! It only crackles when it really gets loud, like fortissimo and louder.I guess the problem is not caused by the soundfont, I had this issue with both the included soundfont as well as others.There's just another little thing: Sometimes when an instrument starts to play, it leaves out the first note. So I guess Bassmididoesn't preload the soundfont to the RAM like Coolsoft's Midi Synth.