Twitch Studio Beta updates
Twitch Studio Beta, our free streaming software designed to make it easy for new streamers to get started on Twitch, takes the guesswork out of setting up and managing your stream.
We’ve recently updated Twitch Studio with four exciting new features: one-click green screen, Chrome browser window capture, Vulkan game capture, and multiple audio source support. What went on behind the scenes to bring these features to life? Read on to find out!
One-Click Green Screen (Chroma Key)
Chroma key, more commonly known as green screen, is a filter used to remove the background from a webcam. You’ve probably already seen it in action on a few of your favorite streamers’ channels. Green screens are a fun way for streamers to bring their personality to their stream, allowing them to dynamically alter their stream background and make their content appear even more engaging and polished. However, the process to set up a green screen can often be manual and time consuming, requiring a lot of adjustments of different settings to get everything set up just right. That’s why we built our green screen auto-detection feature that allows you to automatically configure chroma key settings and detect your green screen with a single click.
So how does a green screen work? By using a solid-colored background, the filter is able to find all pixels of that specific color and make them transparent. This is often done with a green sheet or screen specifically made for this purpose, but can work with any solid color that doesn’t appear in the foreground. While the solid background may appear to be the same color to a human, the computer sees it very differently. Shadows and uneven lighting mean the computer sees that background as a whole range of colors. In order to account for this the filter ignores the luminance or brightness of each pixel and instead looks only at the chrominance or color.
Simply ignoring luminance isn’t always enough. Lighting differences also change the underlying chrominance of the pixels as well. In order to account for this the filter also has a “similarity” setting that controls how close a pixel’s chrominance has to be to the target background color to actually be removed.
This all works well enough if you have reasonably even lighting on the background and can get the settings just right, but tweaking those settings can be a very frustrating exercise. If the similarity value is too large then colors from the foreground will be mistakenly matched and if the value is too small patches of background will continue to be shown. Finding the right settings is complicated by the fact that there are actually two parameters to adjust to find the correct settings—the target color, and the similarity value—and that’s where Twitch Studio’s one-click green screen comes in.
Our one-click green screen system optimizes these settings for you with the press of a button. It starts by sampling the current image in locations that are mostly background in a typical green screen setup:
We then take the chrominance values of these pixels and plot them in UV-colorspace. If we detect a cluster of point in this plot we can then locate the center of the cluster and use that as the target color and measure the radius of the cluster to use as the similarity value:
And just like that, we can select the optimal settings without any fiddling required!
Chrome Capture
Window capture in streaming software has historically used a Windows API function called BitBlt to capture an image of the target window. This is efficient and works well in most cases, but some applications use more modern rendering methods which cause this method to fail. This is most often noticed when trying to capture Chrome or other chromium-based apps (like Discord, Slack, Spotify, VS Code and others), and broadcasting apps can be unreliable and error-prone when capturing Chrome windows. This often results in a black screen without any communication or error messaging to let streamers know what’s going on.
Some workarounds to this problem included using cropped display capture or switching the app to software rendering mode (when possible). Unfortunately, neither of those workarounds are particularly convenient. Cropped display capture introduces a risk to share something on stream accidentally if the window is moved and switching to software rendering mode can have performance implications for the app even when not streaming.
We wanted to improve on the status quo. PrintWindow is another API that can be used to capture the content of other windows, but by default it mostly gives blank or unreliable output for these windows as well. It turns out, however, that the function accepts an undocumented PW_RENDERFULLCONTENT flag which forces a full repaint of the target window and solves this problem.
This new capture method isn’t a panacea since it forces the target window to do additional work to render this output, but that’s certainly better than the alternative of having no output at all. In order to make sure we’re only incurring this performance hit when it’s actually necessary we only enable this capture mode by default for apps that are known to have capture issues with the traditional BitBlt method and allow users to opt-in to this “Compatibility capture” mode for anything else we might have missed. This makes window capture just work in almost all cases while still optimizing performance.
Vulkan Game Capture
We know the gaming landscape is continuously evolving, and a priority for the Twitch Studio team is to keep up with the latest technology and help ensure that creators can stream all the latest games. To that end, Twitch Studio now supports game capture for games running on the Vulkan graphics API. This was an interesting project since, unlike other rendering APIs, Vulkan actually provides a way to capture games and render overlays out of the box. Game capture is normally accomplished by injecting code into the running game which then hooks into the rendering APIs in order to intercept each frame of gameplay to make a copy. Vulkan provides a layer system which allows 3rd party software (like streaming apps) to register a special DLL that will be loaded into all Vulkan games automatically. This streamlines the game capture process, allowing us to bypass much of the complexity of current code injection techniques.
Multiple Audio Sources
Twitch Studio launched into beta with an initial feature set targeted at streamlining the experience for first-time streamers and therefore used a simplified audio model with a single mic and single desktop audio capture. As we expand the feature set based on feedback from streamers, one of the most requested features was support for additional audio channels/devices and recently we launched a major refactor to the underlying audio system to do just that.
With this audio overhaul we wanted to ensure that the default audio experience remained as straightforward as possible while adding more advanced configuration options in a way that was still as simple as possible to use. Under the hood this meant completely changing the way we manage audio devices. Instead of simply selecting the default output and adding mics on demand, we now have the ability to monitor all input and output devices when selecting a new device to add. In addition to simplifying the audio system, this also lays the groundwork for more exciting audio features we have planned for the future, so stay tuned!
At Twitch, we’re committed to making it easier than ever for new streamers to get started on their streaming journey. We’re building a lot more tools this year to help streamers continue to uplevel their streams while keeping the experience as simple and streamlined as possible. If you haven’t checked out Twitch Studio yet, try it out here.