ARKit Object Detection

Object detection is a feature available since ARKit 2. By registering data of pre-scanned objects, you can detect registered objects from the AR application you created.

While there are several articles about using this feature with Unity, I couldn’t find any examples using UE4.

I really wanted to use this feature in UE4 (4.22), and after various trials, I succeeded in getting it to work, so I’ll note down the method here.

Object Scanning

First, scan the object to be detected.

Scanning using UE4

To scan using UE4, first set the Session Type of ARSessionConfig to “Object Scanning”, and then apparently use “Get AR Candidate Object”.

There’s a reason I used the word “apparently” above; currently, scanning has not succeeded with the above method.

The application crashed when calling “Get AR Candidate Object”, and scanning was not possible.

If anyone has succeeded, please let me know.

Scanning using Apple-provided tool

Pressing the “Download” button at the top of the site below allows you to download a complete project for an iOS app for object scanning.

Scanning and Detecting 3D Objects | Apple Developer Documentation

Use XCode to install the downloaded project onto your iOS device. The usage is quite intuitive; first, define the area with a yellow box, then scan each face.

I feel that having many small yellow dots (probably object feature points) within the yellow box leads to a higher detection rate later.

When scanning is complete, a file with the extension *.arobject is generated. Transfer this file to your development machine using iCloud or similar methods.

For AR app development using XCode or Unity, the above method is complete, and the *.arobject file can be used as is.

In the case of UE4, it cannot be used directly, and you need to partially rewrite the downloaded scanning app.

createAndShareReferenceObject method in ViewController.swift

Before change:

try object.export(to: documentURL, previewImage: testRun.previewImage)

After change:

try NSKeyedArchiver.archivedData(withRootObject: object, requiringSecureCoding: false).write(to: documentURL)

The rewritten part is where the file is generated, but it uses a different function.

The original uses the export method of the ARReferenceObject class, but I changed it to first binarize it using NSKeyedArchiver and then write it directly to the file.

Also, to distinguish it from the original, I used the file extension (*.arobject2) this time.

This is because NSKeyedUnarchiver is used when restoring the binary in the UE4 ARKit plugin (FAppleARKitConversion::ToARReferenceObjectSet).

Initially, when I tried loading the *.arobject file without rewriting, the restoration of ARReferenceObject failed. After investigation, I found that the above rewrite succeeded.

When loading the *.arobject file in Unity, a static thumbnail image was displayed, so I think *.arobject contains such additional information, and the content saved in the binary differs from the latter file, which might be the cause.

Importing Scan Data into UE4

Creating ARCandidateObject Asset

I want to import the *.arobject2 file generated by scanning into UE4. This time, I wrote a dedicated asset generation class (a class inheriting from UFactory) in C++.

The file content is as follows:

A point to note here is that the above C++ class uses Editor functionality, so a reference to the UnrealEd module needs to be added.

Also, if you put it in your project’s Runtime module, you will get an error when exporting the app, so please put it in the Editor module.

This Historia site explains modules clearly:

[UE4] About Modules | Historia Inc.

After adding the above file to the project and building, dragging the *.arobject2 file will import the file and create an ARCandidateObject data asset.

Opening the asset, you can see large numbers in the “Candidate Object Data” array.

The content of the *.arobject2 file is stored here in binary format. (The array is very large, so opening it is very slow. It’s better not to open it.)

Also, set a name in “Friendly Name” on this screen. This name can be used during detection.

Regarding the “Bounding Box” at the bottom of the screen, looking at the ARKit plugin source, it doesn’t seem to be used, so I think you don’t need to input it for now.

Registering to ARSessionConfig

Register the created ARCandidateObject asset in ARSessionConfig.

This part is similar to image detection.

Add the created asset to the “Candidate Objects” of the ARSessionConfig asset.

Object Detection

Finally, object detection.

Calling “Get All AR Geometries” retrieves all detected recognitions (planes, images, etc.), from which I select those that can be cast to “ARTrackedObject”.

The pre-registered Name can be obtained with “Get Friendly Name”, so you can use this to branch the subsequent processing.

Execution Image

Executing this looks like this.

It recognizes a PreCure character doll and generates a blue box.

Summary

This time, I really wanted to do object detection using UE4, so I managed to succeed through trial and error.

It’s good that it worked in the end, but I wish it could be done a bit more easily. Ideally, it would be great if *.arobject could be imported directly, just like in Unity, so anyone could use it.

It seems possible by rewriting the ARKit plugin, but I don’t want to do that every time the engine updates, so I’ll stick with this method for the time being.

If anyone knows an easier way, please let me know!