Edit ${NSISDIR}\Contrib\UIs\modern.exe with Resource Hacker and apply with MUI_UI/ChangeUI. This will only change the UI layout. You can use SetCtlColor to change control colors but you are never going to get exactly what you have in that screenshot because that is probably a custom MSI UI.

I use UltraModernUI for NSIS, which leverages the SkinnedControls plugin, and made a modern flat look UI, with custom buttons and scrollbars etc. But it was quite a bit of work to skin almost everything (message boxes still aren't skinned).

No tutorials that I know of, but here's a quick and dirty one to get you started:

After you install UMUI, make a copy of one of the skin .nsh files in C:\Program Files (x86)\NSIS\Contrib\UltraModernUI\Skins and copy the corresponding folder of images, and move them to your installer folder. Grab Paint.NET or Gimp and change the buttons and scrollbars to a flat color, and use a slightly different color as a border. Take a screenshot of one you want to duplicate and use the eyedropper tool to get the exact color if you want. Don't bother making the background/header/bottom, etc. a single color; you can just disable those. If you use the components page, you probably want to make new checkboxes too. Copy and edit one of the ones from C:\Program Files (x86)\NSIS\Contrib\Graphics\Checks, and specify the new image with MUI_COMPONENTSPAGE_CHECKBITMAP.

!import the skin's .nsh file into your installer, and tweak it to remove the images you don't want and to change the paths to your new versions, tweak the colors as necessary. I'd suggest specifying the following in the .nsh: