Case Study: Shared Holographic Experience

Selligent, a global provider of relationship marketing solutions, wanted to demonstrate the potential of shared holographic collaboration at the Digital Marketing Expo (Dmexco) this September. To do so, they built and demoed HoloCampaign, a mixed reality customer relationship management (CRM) application using Microsoft HoloLens. Kazendi, a lean innovation studio based in London, took on the project. Dangling Concepts was part of the technical team for this endeavor to deliver the holographic application.

Here’s what it took to build and implement this application with HoloLens. You can also check out Selligent at DMEXCO on YouTube!

HoloCampaign

The purpose of HoloCampaign is to show a campaign workflow building process for demonstration and training purposes. The user (instructor or demonstrator) can place campaign activities on a grid board and connect them in a sequence like a holographic flow diagram to discuss the different strategies for the campaign at hand. The exciting part is that this happens in a shared holographic environment. This enables other HoloLens devices (trainees or clients) to join planning and see the workflow process while having a live discussion with the trainer.

The following sections describe the tools, technologies, and strategies used to develop this project.

HoloToolkit

HoloToolkit is an amazing open source library maintained on GitHub by the HoloLens team. It contains key components, prefabs, and helper functions in the areas of HoloLens Input (Gaze, Gesture, Voice Command, and Cursor Management), Spatial Mapping, Spatial Audio, Text To Speech and Hologram Sharing.

We utilized HoloToolkit as the baseline framework for our case study, which saved us a ton of time. HoloToolkit provided us with well-built components and sample code that enabled us to write the application quickly.  We used the following components:

  • GestureManager
  • GazeManager
  • CursorManager
  • KeywordManager
  • SpatialMappingManager
  • SpatialMappingObserver
  • SpatialMappingSource
  • BillBoard
  • TagAlong
  • TapToPlace

The HoloToolkit Sharing is built more like client server architecture, where a service component needs to run on a mediator or server. HoloLens devices are essentially clients that use the HoloToolkit Sharing client component to send and receive communication routed through the service/mediator.

Note: Holographic apps built with HoloToolkit Sharing cannot be published on the Windows Store due to the use of prohibited API.

To achieve our shared holographic experience, we turned to the .

Unity Networking (UNET)

Unity Networking is composed of a High Level API (HLAPI) and a Low Level API (LLAPI). The HLAPI abstracts a lot of networking details and provides an easy-to-use interface for developers. The LLAPI can be used for advanced scenarios where low-level control over the network communication is required. For our case study, the HLAPI provided all the required functionality and ease of use for quick prototyping.

We used the following HLAPI components:

NetworkManager

The NetworkManager component is the central piece for managing the network state of a multiplayer app/game implemented entirely using the HLAPI. NetworkManager wraps up a lot of useful functionality into a single place, and makes creating, running, and debugging multiplayer games as simple as possible. The Network Manager can be used entirely without scripting. It has Inspector controls in the Editor that allow configuration of all of its features, including:

  • App/Game State management
  • Scene management
  • Object Spawning/UnSpawning management
  • Object Authority management
  • Network Latency Simulation

 

 

The following functions can be utilized to start a server or host session.

NetworkManager.StartServer();

NetworkManager.StartHost();

Server session wasn’t suitable for our project because it is only for a dedicated server scenario . Host session contains both a server and a local client, which works far better for our project becauseas one HoloLens starts a session and becomes the Host, other devices can connect to it without the need for an intermediary server machine.

NetworkManager also allows the user to specify a list of prefabs that will be spawned and shared across the network for a shared holographic experience.

NetworkDiscovery

The NetworkDiscovery component allows Unity apps to find each other on a local network. It can act as a host to broadcast presence (session) and as a client to listen for broadcasts.

 

The following code snippet starts a NetworkDiscovery Broadcast Server:

NetworkDiscovery.Initialize();

NetworkDiscovery.StartAsServer();

The following code snippet starts a NetworkDiscovery Broadcast Client:

NetworkDiscovery.Initialize();

NetworkDiscovery.StartAsClient();

It also enables joining broadcasted sessions using the NetworkManager. The NetworkDiscovery component is for use on local networks only. This component uses the UDP broadcast feature of the underlying LLAPI and receives broadcast information using the following virtual function:

public virtual void OnReceivedBroadcast(string fromAddress, string data) { }

This component is optional as we can directly utilize the NetworkManager to connect to a host session provided that we know the IP/port of the host session like so:

NetworkManager.networkAddress = ipAddress;

NetworkManager.networkPort = port;

NetworkManager.StartClient();

This is a very helpful component as it allows one HoloLens device to broadcast and another to discover and automatically connect to the session.

NetworkIdentity

NetworkIdentity is a core component required for spawning and sharing objects across network. This component controls an object’s network identity and makes the networking system aware of it.

This component allows the user to specify if the object will be Server Only, Server Authority, or Local Player Authority.

State Synchronization

UNET provides several ways to communicate over the network and synchronize state. The [SyncVar] attribute enables propagation of variable/values from server to all clients. The [Command] attribute enables clients to call functions on the server whereas the [ClientRpc] attribute enables the server to call functions on the clients.

Object/State Sharing Strategy

There are two strategies for Network Object/State sharing: Object sharing and Metadata sharing.

Object sharing is the default way of doing networked experiences with UNET. In this strategy, every object that needs to be shared across network is created as a prefab with the NetworkIdentity Component. When the object is created on the server, the UNET system propagates it across all clients – so every client “creates” an instance of that prefab/object.

var networkedObject = (GameObject) Instantiate(NetworkedObjectPrefab, Vector3.zero, Quaternion.identity);

NetworkServer.Spawn(networkedObject);

When the networked object is deleted on the server, the UNET system propagates deletion to all clients.

All network object prefabs have to be registered with the Network Manager component either using the inspector or via code.

Metadata sharing is a lightweight way to create networked experiences. In this strategy, the metadata or map of the networked app is shared and all clients create the networked experience locally.

For our project we chose the Metadata sharing approach. We have over 50 workflow block types and connectors to share across the network. Creating over 50 prefabs and registering them with the network manager would take more development time and would also result in more generated traffic at runtime. To streamline runtime, we shared the metadata for the workflow grid so that all clients create the objects locally to match the metadata.

To do so, we first needed a shared network object that contains the metadata. The shared object contains a representation of our Grid board (e.g. our grid board has 6 x 10 blocks, so we need an array of 60). Workflow block types are defined in an enumerator like so:

public enum BlockType { None = 0, EmailBlock = 1, TwitterBlock = 2, ReportBlock = 3 …………  SimlinkBlock = 50}

The shared network object contains the grid array that specifies the block types on each block as they are placed on the grid in real time. All clients receive notifications for changes in the grid meta data object and create the blocks locally to place on the local grid board.

Block placement/deletion and connector placement/deletion are handled in a similar fashion.

Spatial Anchor Sharing

A shared holographic experience is much more than a networked app/game experience. Shared holographic experiences have the crucial spatial aspect that makes the experience more grounded in reality. A regular networked game using UNET or any networking technology is played on two separate devices (mobile phones or PCs). A shared holographic app provides more collaboration as it takes place in the same shared physical space. The holograms are truly shared—they appear in the exact same spot, physically, across all Hololens devices.

The Microsoft HoloLens SDK provides the components to achieve this by creating and sharing spatial world anchors.

The first step is to initialize a world anchor store:

WorldAnchorStore.GetAsync( (store) => { worldAnchorStore = store;});

The second step is to place your GameObject in the physical space, attach a WorldAnchor to it, and save it off.

var worldAnchor = gameObject.AddComponent<WorldAnchor>();

 if (worldAnchor.isLocated) {    worldAnchorStore.Save(“gameObjectId”, worldAnchor)  }

The third step is to generate the world anchor transfer data using the WorldAnchorTransferBatch

var worldAnchorTransferBatch = new WorldAnchorTransferBatch();

        worldAnchorTransferBatch.AddWorldAnchor(“gameObjectId”, worldAnchor))

        WorldAnchorTransferBatch.ExportAsync(worldAnchorTransferBatch, WorldAnchorSerializationDataAvailable, WorldAnchorSerializationComplete);

Once the data is generated, it can be transmitted to all clients.

The client devices have to receive the data and import it to create a shared anchor.

WorldAnchorTransferBatch.ImportAsync(importingAnchorBytes, WorldAnchorImportComplete);

 

Conclusion/Summary

The combination of HoloToolkit framework and Unity Networking allowed us to develop HoloCampaign in a short period of time and enabled us to provide a shared Holographic experience across HoloLens devices with direct connectivity. The resultant app can also be published to the Windows Store. We are very excited for the opportunities provided by the HoloLens technology.

Author

Imran Shafiq is a technology enthusiast and a Windows UWP/HoloLens Developer. He works in the Health IT area as a day job and moonlights as an indie game developer. As Dangling Concepts, he has published several games across mobile platforms including #HoloTankz on HoloLens. You can find him on twitter @danglingneuron or on his blog at http://danglingneuron.wordpress.com

See Kazendi, Selligent, and HoloCampaign in action here!

References

Selligent: www.selligent.com

Kazendi: https://www.kazendi.com/selligent-holocampaign

HoloToolkit: https://github.com/Microsoft/HoloToolkit-Unity

Unity Networking Overview: https://docs.unity3d.com/Manual/UNetOverview.html

Selligent @ DEMXCO: https://www.youtube.com/watch?v=DIEso08s3XQ

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *