Expose available video track quality levels (bitrate/resolution)

[dmlap]

A number of video formats commonly in use on the web are capable of dynamically adjusting their bitrate to account for bandwidth between the client and the content. A significant percentage of users have preferences for the quality level of their playback and as such, a quality selector button or menu is a common feature of custom video player implementations:

If an application developer is implementing their own adaptive quality switching logic using Media Source Extensions, it's possible to support user quality selection. There doesn't seem to be any way of expressing that on top of the HTMLMediaElement for adaptive formats that are natively supported, however.

Is this sort of capability in scope for the HTML standard?

[foolip]

Big Buck Bunny, I see you!

Do you know what the adaptive formats supported by browser are, is it just HLS or are there more? It's an interesting idea to give the user control over the bitrate in these scenarios. In principle there is nothing preventing the built-in <video controls> from exposing such a menu today, but do you mean added APIs on HTMLMediaElement so that script-created controls can also show such a menu?

[dmlap]

Big Buck Bunny, I see you!

Good eye :)

is it just HLS or are there more?

DASH can also be configured to support client-driven adaptive streaming. I haven't experimented much with it personally but IE11 on Win8.1+ claims native DASH support.

do you mean added APIs on HTMLMediaElement so that script-created controls can also show such a menu?

Yes, that's what I was thinking. Looking over the current state of the media element API, it seems really close to being able to provide this capability through Video/AudioTrackLists. If it were possible for application developers to change selectedIndex, for instance, I think that could be used to build a custom quality selector control.

[foolip]

It is possible to change the selected video track, by setting VideoTrack.selected. This will implicitly deselect all other video tracks. However, there are a few problems:

1. As you have no doubt noticed, the audio/video tracks APIs are not widely implemented yet.
2. With adaptive streaming, there is conceptually still only one video track. Presumably it's possible to have multi-angle video also with HLS or DASH, so there would have to be some way of grouping together video tracks that are conceptually the same.

Implementation is just a matter of time/priorities, but I'm not sure what to do about the second issue.

Is this something you want for video.js, and is it for mobile browsers that support HLS? At least in the case of Android, HLS is implemented by the OS outside of the browser's direct control, so I'm not certain that the APIs even exist to achieve the level of control needed for this:
http://developer.android.com/reference/android/media/MediaPlayer.html

I assume that you have investigated using Media Source Extensions to implement HLS or DASH in the browser, but that the same-origin policy ruins it for you? If not, that would really allow all the flexibility you want.

[dmlap]

All really good points. My thoughts:

1. As you have no doubt noticed, the audio/video tracks APIs are not widely implemented yet.
2. With adaptive streaming, there is conceptually still only one video track. Presumably it's possible to have multi-angle video also with HLS or DASH, so there would have to be some way of grouping together video tracks that are conceptually the same.

If I were designing this in a vacuum by myself, I would be inclined to represent each quality level for the same video as a separate VideoTrack with kind main and leave it up to individual video formats to find some way to distinguish the variants through the label property. As a side benefit, this would allow application developers to introspect on the quality level currently being displayed, which is a common desire for analytics. Maybe the extra functionality would help shift the value proposition for Video/AudioTrack implementors. Alternate angles could continue using the alternative kind.

Is this something you want for video.js, and is it for mobile browsers that support HLS?

Yes, I'm primarily asking for video.js but the scope is DASH and HLS on desktop and mobile. We'd like to support alternate video and audio tracks all over the place so we're looking to polyfill the TrackList APIs where they aren't present. Bitrate selection/introspection is another cross-format player feature and we'd rather expose that through a standardized API (that may or may not get implemented natively in browsers) than come up with a video.js-specific way of doing things.

I assume that you have investigated using Media Source Extensions to implement HLS or DASH in the browser, but that the same-origin policy ruins it for you? If not, that would really allow all the flexibility you want.

Actually, it seems to me most developers would be willing to fight epic battles with CORS and CDNs if they can get rid of Flash. We definitely have the flexibility, we'd just like to continue our design philosophy of polyfilling advanced HTMLMediaElement features rather than inventing our own. Filing this issue was a bit of an experiment-- thanks for taking the time to consider it.

[foolip]

Since all bitrates of the same logical video track would have the same metadata, I'm thinking that if this were to be exposed, some extension to VideoTrack would work. Something like

partial interface VideoTrack {
  sequence<double> getAvailableBitrates();
  // promise resolves when change has taken effect
  Promise<void> setPreferredBitrate(double bitrate);
};

The VideoTracks interface itself isn't shipping in Chromium yet, but I see that Safari has it. @jernoble, is this something you've had requests for? What would make sense?

@dmlap, you mention Flash. Can this be done today using Flash, and if so do you know what Flash's API for doing this looks like?

[dmlap]

Aggregating the different variants onto a single track sounds reasonable to me but I think there will be more factors than bitrate to consider. Selecting variants based on resolution is common as well, for instance. I was thinking that info could be packed into VideoTracks.label in format-specific ways. You could adapt that strategy onto a single VideoTrack, though:

partial interface VideoTrack {
  sequence<string> getAvailableRepresentations();
  // promise resolves when change has taken effect
  Promise<void> setPreferredRendition(string representation);
};

The term "Representation" comes from the DASH world. HLS would probably refer to alternative streams of the same track as "variants".

I'm not an expert on Flash-based adaptive switching but I believe you'd use NetStream.play2(...) to switch quality levels during playback. Each quality level has a name. The player switches quality levels by calling play2() and passing in the name of the stream they wish to use and how soon they'd like the switch to occur. It is the application's job to provide the list of available quality levels-- Flash just provides a seamless way to switch between them. In HTML terms, I think it would be like allowing javascript developers to specify a list of URLs for the VideoTrack and then having a method to choose between those URLs during playback. That doesn't seem very compatible with the current implementations of manifest-based formats like DASH and HLS.

[foolip]

Flash's play2() sounds like it's the Flash API that MSE was intended to replace, in that it's a more low-level "stitch together these streams" API than a high-level "use this quality level from the manifest."

Your getAvailableRepresentations() proposal would most likely work, in that it's always possible to use some string. Do DASH and HLS always have unique string for each representation, or could it be necessary for the implementation to generate a string? If it is, then this would be a bit more work, in that people could then depend on how the string is generated and how to parse the bitrate/resolution from that string.

@jernoble, given that Safari supports HLS, is this something that sounds interesting? I kind of feel like such an API could only be necessary of MSE fails to solve all of the use cases it's supposed to handle, but I don't suppose that native support for HLS is going away, so...

[danrossi]

I'm assuming this isn't available yet to detect available bitrates on native HLS and make it selectable ? Also an event for bitrate change.

There is no way it seems to detect what the video dimension is internally. "videoWidth" is never updated there is a resize event in webkit though.

[foolip]

Ping @jernoble?

[jernoble]

There's no mechanism to manually switch variants in an HLS stream in iOS or macOS. You can detect resolution changes, and we do pipe that through to HTMLMediaElement's videoWidth and videoHeight properties, and we will now fire resize events when those values change. This should be testable in Safari Tech Preview, WebKit Nightlies, or Safari in the developer previews of iOS 10 or macOS Sierra.

See: https://trac.webkit.org/browser/trunk/LayoutTests/http/tests/media/hls/hls-video-resize.html.

If you're getting resize events but videoWidth isn't changing, please file a bug and we'll take a look.

[danrossi]

I tested the resize event. It only happens on startup. No event happens after. videoWidth and videoHeight do not update either from my tests. I tested in Webkit on OSX 10.10.5