Launch by NTT Data is a global digital business that helps their clients to strategize, ship and scale digital experiences. They have developed innovative solutions for brands such as Ford, Epic Games, and Adidas.

As virtual and augmented reality technology matures and becomes more usable, virtual experiences continue to have an ever-increasing role in how we go about our daily lives. As a leader in creating innovative digital experiences, Launch explores how new devices like the Apple Vision Pro can be used to enable users to interact with their environment, specifically to control smart devices in their homes.

To this end, our Spatial IoT Control using Apple Vision Pro is a visionOS and iOS application that controls home appliances like plugs, lights and other appliances through a sleek and seamless interface.

Our software recognizes and tracks devices in the user’s view. When a device has been identified, a pop-up appears over it that the user can interact with to change the state of the device. For example, when a smart lamp enters the view of the Vision Pro, a toggle switch view is displayed above the lamp, which turns the lamp on and off when selected.

Our software enables actions involving other smart home appliances using various hand gestures recognizable by the Vision Pro, including sliding, pinching, and twisting.

In addition, IoT devices can be controlled via iPhone using our iOS application, which has the same features and functionality as the visionOS app. This provides more flexibility in how users can control their home appliances.

Both applications are written in Swift. Interaction with tracked objects is enabled through the RealityKit and ARKit frameworks, and devices are controlled via HTTP and MQTT protocols.