In this post, I want to discuss some preliminary tests for recreating similar functionality to an AR spatial sound app called Fields.
Fields was an AR app created by a group called Planeta, they also created the VR game Drops: Rhythm Garden. Fields was an iOS and iPad app that allowed users to experience and create spatial sound fields. These were spatialized sounds, either provided by the app or created by the user via upload or microphone. Check it out below.
I say, was, because it is no longer possible to get Fields on the app store, and the group that made it does not appear to operate anymore.
I want to recreate this app or create a new one with similar functionality because of my interest in spatial AR sound applications. There was/is a lot of potential for spatial computing applications and spatial audio/ambisonics. A few companies showed interest in AR/MR/XR audio as stand-alone applications, like delivering localized directions or audio content based on a user's geolocation. Take for instance the short-lived Bose AR initiative. And even more, companies recently have been leaning into spatial sound and ambisonics for music and podcast production in a more consumer-forward, mainstream way, much like Apple and Dolby.
Since I have been operating in Unreal Engine 5.1 for the moment, I started there. But I know it is not likely the best solution for this form of application.
One feature of the Fields app that I found the most compelling was the ability for the user to record audio from the microphone of the device and place that recording in physical space. The recording would stay at the location it was recorded at and be spatialized with some attenuation settings. This is the feature I wanted to start with this small UE test.
In the following video you can see that I can record multiple takes from my microphone by holding the space bar, as the spacebar is released, the take is recorded and spawned in the game world. The recording then plays on a repeated loop until the application is stopped. The plan for the future will be to have sound spatialized with attenuation settings and be able to be dragged to different points in the world. Check out the recording test below.
Blueprints Recording of Microphone in UE 5.1
This was achieved using Unreal's Audio Capture Component to get the default device microphone and record the input. The capture component contains "start" and "stop" recording methods built in. The spacebar input (isHeld) was used to check for user input and arm the component for recording the mic. Once the space bar is released the recording is stored in a temporary SoundWave file in the editor. The sound wave file is then sent to another Bp actor that is spawned in the world. This actor has the functionality for the recording to repeat once it has finished playing. Attenuation settings can also be applied to spatialize the sound (that will be done at a later time). See the Bp nodes setup below.
BP nodes for Microphone Recording in UE 5.1
This test was more of a proof of concept or just testing an idea out. In the future, I will identify the best platform for developing an application like this (likely something with better web functionality) and try to build out the full application in earnest. Although, I do see how something like this could function as a fun VR or videogame mechanic and still has some merit in exploring in UE. Especially how this workflow could be used with MetaSounds.
Comments