WindowCapture2D

This time, I released a window capture plugin usable in Unreal Engine on GitHub.

Currently, it’s only available on GitHub, but I plan to release it on the Marketplace once the features are a bit more consolidated.

Thoughts on WindowCapture

At one point, I wanted to display a Windows application within a VR space in UE4 for VR content and researched various methods, but couldn’t find a feature that directly matched.

At the time, I didn’t have much time, so I gave up on achieving it in UE4.

Meanwhile, browsing the internet, I found someone who had created the exact feature I wanted in Unity. I tried it and was impressed that I could do exactly what I wanted.

Tried creating a plugin to display the Windows desktop screen as a texture in Unity - Hecomi Tips

While thinking it was amazing, I also thought, if it can be done in Unity, couldn’t it be done in Unreal too? That’s how I started trying.

Capturing a Window

When I changed careers at 25, in what’s now called “engineer from no experience,” the first task I did was window capture on Windows.

I was battling with C++, MFC, and I remember thinking back then, HDC? HWND? What’s that? What’s a memory DC?

Therefore, the technology of capturing windows holds a special place for me.

Capture Methods

The two window capture methods I know are:

  • Get the Window DC and use BitBlt
  • Get the Window DC and use PrintWindow

In terms of speed, BitBlt is faster and better than PrintWindow, but applications like Chrome or those made with UE4 cannot be captured with BitBlt.

Instead of capturing the window directly, capturing from the desktop DC and trimming the necessary area allows capturing any window. However, there were many issues, such as having to wait for the display monitor’s refresh rate to capture, and not being able to capture the back side of overlapping windows.

For the other method, PrintWindow, I initially thought Chrome couldn’t be captured either, but I discovered that giving it an argument not listed in Microsoft’s help (PW_RENDERFULLCONTENT=2) enabled capture.

So, this time I decided to try making it with PrintWindow.

(There’s also said to be a method using DirectX for window capture, but I haven’t tried this yet.)

Implementation in UE4

For now, I tried capturing the target window within Tick and rewriting the texture content.

Rewriting the texture (UTexture2D) can be achieved with the following code:

auto Region = new FUpdateTextureRegion2D(0, 0, 0, 0, TextureTarget->GetSizeX(), TextureTarget->GetSizeY());
TextureTarget->UpdateTextureRegions(0, 1, Region, 4 * TextureTarget->GetSizeX(), 4, (uint8*)m_BitmapBuffer);

TextureTarget is the target Texture2D, and m_BitmapBuffer is the pixel buffer in BGRA format.

These two lines allow dynamic texture rewriting.

When executed, it seemed to work, but I ran into trouble as the stuttering became severe when the target window size increased.

So next, I tried processing only the capture part in the background using Task.

void ACaptureMachine::Tick()
{
    AsyncTask(ENamedThreads::BackgroundThreadPriority, [this]() {
        // Perform capture process
        });
}

This improved the frame rate and was effective, so I then tried capturing multiple windows.

Capturing three windows simultaneously caused the frame rate to drop drastically again, which was problematic.

Thinking about it, I hypothesized that since this window capture uses one Actor to capture one window and processes it within Tick, the processing is chained serially, causing it to become progressively slower.

Therefore, I stopped processing within Tick and modified it to generate a thread within the Actor and perform the capture within the thread.

Then…

Despite a small mistake, I was able to maintain 60fps even when capturing three windows.

I think the load will increase if I increase the number of windows or make the windows larger, like full-size 4K windows, but for now, I feel I’ve implemented the minimum necessary functionality.

How to Use

Usage is as follows: (Usage may change due to future specification changes)

  1. Place the "WindowCapturePlane" actor in the level. image
  2. Set the properties of the placed actor.
image
PropertyDescription
Capture Target TitleEnter the window title string of the capture target
Title Matching Window SearchSelect the matching method for the string entered in Capture Target Title and the window title
Frame RateCapture rate (fps)
Check Window SizeEnabling follows window size changes, but increases processing load
Cut ShadowCaptures excluding the window shadow

Title Matching Window Search

ValueDescription
PerfectMatchExact match with the input string
ForwardMatchCompare with forward match
PartialMatchCompare with partial match
BackwardMatchCompare with backward match
RegularExpressionCompare using regular expressions

With the above, if the settings are correct, the window will be captured when you Play.

However, please be aware of the following points:

*1 Currently, window searching is done only once immediately after startup, so you need to Play the UE4 side with the target window already displayed.

*2 Capturing becomes impossible if the target window is minimized.

Future Development

Currently, window capture is possible, but only display is available, not interaction.

Considering use in VR, for example, being able to operate the window within the space would broaden its applications, so I definitely want to implement window operation (forwarding mouse and keyboard input).

I’d also like to try network sharing, but this seems difficult, so I might give up on it.