webrtcH4cKS: ~ getUserMedia resolutions III – constraints unleashed

Back in October 2013, the relative early days of WebRTC, I set out to get a better understanding of the getUserMedia API and camera constraints in one of my first and most popular posts. I discovered that working with getUserMedia constraints was not all that straight forward. A year later I gave an update after the situation with Chrome was greatly improved, but Firefox at the time effectively only supported a single resolution so constraints were not much help. Specifically, I am interested in understanding what happens when you ask for a specific resolution. You might want to have a specific resolution returned by getUserMedia if you want to match the camera resolution to a specific video area to have a 1 to 1 pixel correlation, in a computer vision application where each pixel represents a distance, or if you are dealing with non-standard video devices.

Now it has been another fifteen months and a lot has changed since that last post. getUserMedia, WebRTC’s most mature API is supposedly nearing standardization. The spec gone through several updates and we have full support from Firefox and Edge. I was hoping this post would be a quick code update and simple data addition to my last post to reflect these updates. It ended up being a lengthly process process that resulted in some confusing and nonintuitive results.

getUserMedia Resolutions 3 – the constraints are back and this time they need to deal with the Foxfox and Edge

tl;dr summary

Don’t require specific resolutions in your app – use a range if you can instead and try the new ideal constraint if you really want, but don’t require something specific

If you do require specific resolutions in your app, make sure to check the video element to see if the returned video is the same size as what you asked for and only ask for standard sizes, like 1280×720, if you want your app to work will multiple browsers.

No 4K with getUserMedia – 1080p is the maximum resolution (in 16:9 aspect ratio) you can get today

Chrome 49 on desktop does a good job scaling to what ever you ask it

Unlike previous versions Chrome 47 on Android does not always give you what you ask for – this might be a bug

Use adapter.js to simplify the many differences that exist for getUserMedia and its constraints between browsers (the same is true for PeerConnection too)

Firefox only works within a set of fixed resolutions

Edge only works within a set of fixed resolutions

Device orientation matters when you are requesting fixed resolutions on a mobile

Asking for a specific square resolution will not help you much on Firefox or Edge, but (mostly) works on Chrome

Some of the results returned here are someone confusing and nonintuitive

Much has Changed

A lot has changed since my last post:

The WebRTC API specs were updated to support promises (as mentioned here)

FireFox added support for getUserMedia constraints in v38

Microsoft Edge was introduced with getUserMedia

srcObject was added as a mandatory attribute for assigning a getUserMedia stream to video element

MediaStream.stop() was deprecated in Chrome in favor of the new MediaStreamTrack object with MediaStreamTrack.stop()

New getUserMedia constraints

Since the scanner am tests to see which specific resolutions work, and which don’t, I need getUserMedia to check one resolution per call. The old way of doing this was just to set mandatory maximum and minimum constraints to the same value:

Old constraints object

1

2

3

4

5

6

7

8

9

10

11

12

varconstraints={

audio:false,

video:{

mandatory:{

sourceId:device.id,

minWidth:{candidate.width,

minHeight:candidate.height,

maxWidth:candidate.width,

maxHeight:candidate.height

}

}

};

The new way to do this is to use the exact object:

1

2

3

4

5

6

7

8

9

varconstraints={

audio:false,

video:{

deviceId:device.id?{exact:device.id}:undefined,

width:{exact:candidate.width},//new syntax

height:{exact:candidate.height}//new syntax

}

}

};

You will also notice that sourceId also changed to deviceId for specifying which device to choose. Note that addition to exact the new gUM constraint specs also allow ideal and advanced, but that is the topic for another post.

adapter.js

The biggest change by far was the addition of FireFox and Edge into my test set. Fortunately adapter.js mostly worked (more on that next). If you are not familiar with adapter.js, it is a brilliant piece of code that adapts for the various browsers and variances with the spec, so you can write your code to the latest spec and in theory it should work across Chrome, FireFox, and Edge (see more here). At the advice of frequent adapter.js contributor Fippo, I pointed to the latest version of adapter JS here: https://webrtc.github.io/adapter/adapter-latest.js

I found a bug

The program has 2 options:

Quickscan that sorts through a bunch of predefined, common resolutions

Fullscan to check every 16:9 and 4:3 ratio resolution between an entered range

I killed a couple of hours trying to figure out why the quickscan option worked but the full scan always returned a constraint error. After some debugging I discovered my full scan code was using toFixed() to figure out the width. toFixed converts the value to a string. adapter.js has a bug in Chrome where it requires the value to be number. Fippo filed a pull request to fix this, but later determined it is difficult to fix and closed it. So my code change will have to do for the time being.

Mobile improvements

I made a few other small improvements to help with usability on mobile devices, including using bootstrap to make it responsive and adding some hyperlinks to jump to the end of the table for long scans.

Also, with services like Instagram defaulting to square photos to avoid device orientations issues I decided to add 1:1, ratio test as part of the full resolution scan to see what happens.

1080p is WebRTC’s max

I did some quickscan tests of 4K (2160) resolutions in all scenarios – none of them worked. I scanned through the official webrtc.org source code, I could not find any references above 1080 in the code. 1080 appears to the max unless someone modifies the source to add higher resolutions.

[EDIT]

As Mihály indicates in the comments below, it does appear that Chrome and Opera both allow 4K resolutions on their desktop versions.

Mapping results/errors to categories

Like last time, results can fall into one of 4 categories:

Pass – the resulting stream output has the same resolution of what was asked for in the mandatory constraints

Mismatch – this is my programs own error which is tagged whenever the output resolution is different from what was asked for in the mandatory constraints. I see this as the most problematic result since you are getting something different than you asked for without any indication from the API.

Device Error – For whatever reason, the browser returned a device error. For example, Chrome would sometimes return a DevicesNotFoundError and Edge would give a SourceUnavailableError. I did not get errors like this in Firefox.

Over Constrained– I used this value whenever the browser returned an explicit error that the constraint was not satisfied. Chrome gives a ConstraintNotSatisfiedError, Firefox gives a OverconstrainedError, and Edge gives a NotFoundError.

Results

I was not able to get a condensed summary table of 45,360 datapoints to fit in this post with any reasonable readability, so I am going to provide some highlights below. I suggest you grab my summary table pdf or just check out the source Excel file.

Note that my Microsoft LifeCam is buggy on Mac, so I only used it on my Windows 10 machine.

Chrome

Desktop

Like last time, Chrome scales the returned image to whatever you ask it, up the the maximum resolution for the camera after which as ConstraintNotSatisfiedError was returned. Chrome 49 behaved consistently for me on OS X and Windows 10.

A 4:3 dimension maxed out at 1280×960 and asking for 1:1 dimension maxed out at 1024×1024 on my old Logitech Webcam 500 even though the max resolution in 16:9 dimension is 1280×720.

Chrome OS

I did a last minute scan on my Chromebook and it showed similar results to the normal desktop above. The only difference is the camera is pretty lower power so it topped out at 360p for 16:9 and 640×480 for 4:3 and 1:1 resolutions. (Note I have not added these results to the pdf table yet).

Mobile

The results on my Samsung Galaxy S5 were not very straight forward compared to desktop.

Mismatches

In my August 2013 scan Chrome 37 scaled the returned resolution down to any resolution I asked for under 1080. For example, when I asked for 1038×584 I got 2×2 as a result. I checked many of these values independently to make sure there was not some bug in my code and I got the same result.

Browser-OS-Camera

Device Max (HD)

Pass

Mismatch

chrome 47 on Android 5.0

camera 0, facing back

16:9

2160

199-360; 406-540, 595-720, 811-1080

1-198; 361-405; 541-594; 721-810

4:3

2160

265-720

1-264; 721-1080

1:1

2160

2; 289-720

1; 2-288; 721-1080

camera 1, facing front

16:9

1080

1-180; 199-360; 406-540; 595-1080

180-198; 361-405; 541-594

4:3

1080

1-240; 265-540; 595-1080

241-264; 541-594;

1:1

1080

1-240; 289-1080

241-288

As you can see in the table above, there seems to be a few seemingly random ranges that return an unusable video stream. These ranges also differ between the front and rear camera.

This seems to be a step back from the very predictable results I got last time and worthy of a bug report.

Device Orientation

Like last time, device orientation makes a difference. Rotating my GS5 vertically cases the returned resolutions to mismatch what I asked for.

Device orientation matters when asking for exact constraints on a mobile. Chrome 48 on Android

The rotation not only swapped the length and width, but also adjusted the image to be more square in many cases.

Firefox

Firefox had made a ton of improvements since last year.

Desktop

Firefox sends out a dock bounce alert on OS X after a successful getUserMedia capture if the window is not in focus. Whenever this happens it appears Firefox pauses Javascript execution until you click on Firefox again to make it the focus. This makes it tough to run the scanner in the background.

Also make sure to set permissions to “Allow” instead of “Always Ask”:

Firefox 45’s camera and microphone permission UI

Firefox acts a lot like Chrome did in my original tests in October 2013 – a fixed set of resolutions are supported and if you ask for anything else you get an over-constrained error. Firefox 45 on OSX gave me a lot more resolutions than I got out of Windows 10 with the same camera.

Mobile

The permissions scheme on Firefox mobile does not cache permissions, meaning you have to explicitly approve camera sharing every time. This does not work well when you are are calling getUserMedia 3240 times.

Unlike the desktop version, Firefox 44 mobile does not seem to provide an option to cache permissions

Fortunately there is a flag you can set to allow permission by default. Just type in about:config in the url bar, then set media.navigator.permission.disabled to true.

Much like Chrome, rotating the phone to portrait mode causes returned video element area to be reversed in cases where the – a mismatch in my methodology.

Browser

Device

Res Name

Ratio

Ask

Actual

firefox 44

Camera 0, Facing back, Orientation 90

1080p(FHD)

16:9

1920×1080

1080×1920

firefox 44

Camera 0, Facing back, Orientation 90

720p(HD)

16:9

1280×720

720×1280

firefox 44

Camera 0, Facing back, Orientation 90

VGA

4:3

640×480

480×640

firefox 44

Camera 0, Facing back, Orientation 90

CIF

4:3

352×288

480×640

firefox 44

Camera 0, Facing back, Orientation 90

QVGA

4:3

320×240

240×320

firefox 44

Camera 0, Facing back, Orientation 90

QCIF

4:3

176×144

144×176

Edge

As the new kid on the WebRTC block, how did Microsoft’s new Edge browser fare? Like Firefox, Edge acts a lot like Chrome did in the early days with only a few pre-defined resolutions working and everything else returning a NotFoundError.

Camera

Pass

NotFoundError

edge 10586 on Windows 10

camera 1

16:9

1080

360, 720, 1080

4:3

1080

120, 240, 480, 600

1:1

1080

camera 2

16:9

720

180;360;720

4:3

720

120, 240, 480, 600, 720

1:1

720

Camera 1 was my LifeCam and Camera2 was my Logitech Webcam 500. Oddly, the Logitech Webcam returned a SourceUnavailableError on 160×90 and 800×450.

With no Windows Mobile WebRTC support, this was the only test I could run.

[edit: addition below]

Try it Yourself

Want to keep up on our latest posts? Please click here to subscribe to our mailing list if you have not already. We only email post updates. You can also follow us on twitter at @webrtcHacks for blog updates.

14 comments on “getUserMedia resolutions III – constraints unleashed”

Great post! In so far as the webcams are concerned there are a few additional things that you might consider.

1 – You need some newer webcams. The MS Lifecam is a quirky beast that relies heavily upon a Windows device driver for decent performance. Logitech’s C500 is ancient. Neither are capable of 1080p.

The Logitech C920 and C930e are really the current benchmarks. Logitech seems to have rebranded the C930e as the HD Pro Webcam.

2 – Consider frame rates, especially at higher resolutions. There are a few webcams that deliver 1920 x 1080 pixel frames, but may not deliver them at rates that you might consider useful. If the application doesn’t leverage UVC1.1 (MJPEG) or UVC1.5 (H264, VP8?, SVC?) to request compressed frames the frames will be uncompressed. Thus the maximum frame rate rate will be limited by the USB 2.0 bus speed.

Since I have several webcams hereabouts a simple experiment or two has answered some of my own questions:

With respect to the Microsoft Lifecam Studio, when using the sample resolution select page that you cited, the camera can be asked for “Full HD” and returns a stream that is that dimension. However, I believe that the camera is actually delivering 720p that’s being scaled up. The result is simply not as sharp as selecting Full HD from a C920.

Further, the C920 delivers a proper 1080p stream. I suspect that GUM is using the UVC1.1 capability to deliver MJPEG encoded frame. In infer this by observing that there’s minimal latency to the video.

When I’ve used software (vMix) that has explicit control of the webcam encoding I can use the UVC1.5 capability to request H264. This always returns a stream that’s delayed around a second.

1. I am definitely in need of a new camera and will use your suggestions. I was actually considering getting a 4k camera which is what prompted me to check to see the max resolution support in the source code (where I discovered 4K would be a waste at this point).

2. I have played around with framerates in similar test as this. I ran out of time to work on that as part of this post.

3. Not that I know of or have seen anywhere. It certainly is not in the spec.

A few years ago, seeing that they were destined to become ever more important, I started to explore the state of the art in webcams. It sad really, how little progress has been made in the realm. It’s basically stagnant.

4K “webcams” are a rarity. USB 3.0 capable webcams are nearly as rare. These fact are related.

There’s more to image quality that the sheer number of pixels. Noise, performance in varying lighting and color depth are all things that could be improved, even at 720p or 1080p.

I suspect that 4K webcams won’t matter at all for another couple of years, even as 4K displays become commonplace.

If you want to pick a fight, how about we charge manufacturers with installing built-in webcams that can compete with their aged external counterparts? That’s be a noble effort.

I used also your handy camera/resolution finder tool during the tests. It worked perfectly, and so it is appreciated very much your hours you had spent developing it. Many thanks again!!

I also tested peerconnection.
To setup a peerconnection with UHD 4k media I tested with apprtc.

You could see a screenshoot about the webrtc internals page:
* googFrameWidthInput ~4k
* googFrameHeightInput ~2K
but encoded and sent video was only 2K.
* googFrameWidthSent ~2k
* googFrameHeightSent ~1k

I could grab 4k video with getusermedia, but chrome encoded and transmitted only 2K.

All in all according my experiences 4k was experimental at the beginning of 2015. Had some promising signs, partially working getusermedia, but also had many limitations.

Wow. That is really interesting.
On #1, I meant that I could not find reference to a resolution greater than 1920×1080 in the source code anywhere (thanks for sharing the search link). Your results indicate that the Chrome and Opera team must have changed this – in the desktop versions at least.

I was hoping to see getUserMedia return a 4K resolution on my Galaxy S5, but it did not. That is the only 4K camera I have. I would be curious to see if other people are able to get 4K on mobile.