Info for Developers of Microsoft HoloLens

HoloLens 関連イベントと情報共有に努めるブログ

HoloLens 技術情報の作業過程記録 - HoloLens Documentation: Holograms 101

この記事は、Holograms 101 の作業過程記録です。公開情報をもとにしていますが、細かくスクリーンショットを載せており、説明部分を主に拙訳しています。また作業過程でトラブルが発生した場合も詳細を記載しています。

Holograms 101

このチュートリアルでは、Unity でビルドしプロジェクトを完了させるところまで行います。コア Windows ホログラフィック プラットフォームと、HoloLens の機能である gaze, gestures, voice input, spatial sound と spatial mapping を含みます。

このチュートリアルを完了させるまでおよそ 1 時間掛かります。

 

事前準備

プロジェクト ファイル

  • このプロジェクトに必要な files (380MB) をダウンロードします。
  • デスクトップなど任意の場所にファイルを展開し、フォルダー名を Origami にします.

Chapter 1 - "Holo" world

このチャプターでは、初めての Unity プロジェクトをセットアップし、ビルドとデプロイ プロセスを実施します。

目的

  • Unity のホログラフィック開発環境のセットアップ
  • ホログラムの作成
  • 作成したホログラムを鑑賞する

詳細説明

  • Unity を起動します
  • Open を選択します
  • 展開した Origami フォルダーにアクセスします
  • Origami を選択し、Select Folder をクリックします
    以下の警告ダイアログが表示された。このプロジェクトは Unity 5.4.0b14-HTP バージョンで作成されたようだが、今起動している Unity は 5.4.0f3-HTP. 起動している方が新しいのでこのまま Continue をクリックする。
    f:id:hololensdev:20160917134616p:plain
    問題なくプロジェクトが開けた
  • Origami プロジェクトには scene が含まれていないため、空の既定の scene を新しいファイルとして File / Save Scene As Origami 入力し Save ボタンを押下します

Setup the main virtual camera

  • In the Hierarchy Panel, select Main Camera.
  • In the Inspector set its transform position to 0,0,0.
  • Find the Clear Flags property, and change the dropdown from Skybox to Solid color.
  • Click on the Background field to open a color picker.
  • Set R, G, B, and A to 0.

    f:id:hololensdev:20160917141122p:plain

Setup the scene

  • In the Hierarchy Panel, click on Create and Create Empty.

    f:id:hololensdev:20160917141311p:plain

  • Right-click the new GameObject and select Rename. Rename the GameObject to OrigamiCollection.

    f:id:hololensdev:20160917141655p:plain

  • From the Holograms folder in the Project Panel (expand Assets and select Holograms or double click the Holograms folder in the Project Panel):
    • Drag Stage into the Hierarchy to be a child of OrigamiCollection.
    • Drag Sphere1 into the Hierarchy to be a child of OrigamiCollection.
    • Drag Sphere2 into the Hierarchy to be a child of OrigamiCollection.

      f:id:hololensdev:20160917141847p:plain

  • Right-click the Directional Light object in the Hierarchy Panel and select Delete.
  • From the Holograms folder, drag Lights into the root of the Hierarchy Panel.

    f:id:hololensdev:20160917142037p:plain

  • In the Hierarchy, select the OrigamiCollection.
  • In the Inspector, set the transform position to 0, -0.5, 2.0.

    f:id:hololensdev:20160917142156p:plain

  • Press the Play button in Unity to preview your holograms.
  • You should see the Origami objects in the preview window.

    f:id:hololensdev:20160917142149p:plain

  • Press Play a second time to stop preview mode.

Export the project from Unity to Visual Studio

  • In Unity select File > Build Settings.
  • Select Windows Store in the Platform list and click Switch Platform.
  • Set SDK to Universal 10 and Build Type to D3D.
  • Check Unity C# Projects.
  • Click Add Open Scenes to add the scene.
  • Click Build.

    f:id:hololensdev:20160917142502p:plain

  • In the file explorer window that appears, create a New Folder named "App".
  • Single click the App Folder.
  • Press Select Folder.
  • When Unity is done, a File Explorer window will appear.
  • Open the App folder.
  • Open (double click) Origami.sln.
  • Using the top toolbar in Visual Studio, change the target from Debug to Release and from ARM to X86.
  • Click on the arrow next to the Device button, and select Remote Device.
    • Set the Address to the name or IP address of your HoloLens. If you do not know your device IP address, look in Settings > Network & Internet > Advanced Options or ask Cortana "Hey Cortana, What's my IP address?"
    • Leave the Authentication Mode set to Universal.
    • Click Select
      Wifi 経由は不安定かと思い、USB 経由で接続を試みる

      f:id:hololensdev:20160917150056p:plain
      エラー発生: 重大度レベル コード 説明 プロジェクト ファイル 行 抑制状態
      エラー DEP6200 : ブートストラップに失敗しました。デバイスが見つかりません。0x89731810: Windows Phone が検出されなかったため、配置に失敗しました。電話が接続されていて、電源が入っていることを確認してください。 

      f:id:hololensdev:20160917150437p:plain
      抜き差ししても、PC再起動しても変わらない。とりあえず、HoloLens のバーぞyンを 10.0.14393.0 にアップデートしてみる。
      f:id:hololensdev:20160919105619p:plain

      PCのバージョンは Windows 10 Enterprise 1511 10586.545
      f:id:hololensdev:20160919105730p:plain
      うまく認識しないので、Wifi 経由で検証する

      f:id:hololensdev:20160919221533p:plain
      IP アドレスが変わっていたので、[デバッグ] - [Origami のプロパティ] から変更

      f:id:hololensdev:20160919221759p:plain

  • Click Debug > Start Without debugging or press Ctrl + F5. If this is the first time deploying to your device, you will need to pair it with Visual Studio.
  • The Origami project will now build, deploy to your HoloLens, and then run.
  • Put on your HoloLens and look around to see your new holograms.

    f:id:hololensdev:20160919222235p:plain

Chapter 2 - Gaze

このチャプターでは、ホロレンズと対話する 3 つの方法のうち最初の 1 つ目を紹介します。それは、gaze です。

目的

  • world-locked cursor を使用して視線を視覚化しましょう

手段

  • Go back to your Unity project, and close the Build Settings window if it's still open.
  • Select the Holograms folder in the Project panel.
  • Drag the Cursor object into the Hierarchy panel at the root level.
  • Double-click on the Cursor object to take a closer look at it.
  • Right-click on the Scripts folder in the Project panel.
  • Click the Create sub-menu.
  • Select C# Script.

    f:id:hololensdev:20160919222851p:plain

  • Name the script WorldCursor. Note: The name is case-sensitive. You do not need to add the .cs extension.
    以下のエラーが出た。ファイル作成時に一度ファイル名をNewBehaviourScript で確定させたたため、クラス名が NewBehaviourScript になってるせい。ファイルを再作成する。

    f:id:hololensdev:20160919223135p:plain

  • Select the Cursor object in the Hierarchy panel.
  • Drag and drop the WorldCursor script into the Inspector panel.
  • Double-click the WorldCursor script to open it in Visual Studio.
  • Copy and paste this code into WorldCursor.cs and Save All.

 

WorldCursor.cs[show]
using UnityEngine;

public class WorldCursor : MonoBehaviour
{
    private MeshRenderer meshRenderer;

    // Use this for initialization
    void Start()
    {
        // Grab the mesh renderer that's on the same object as this script.
        meshRenderer = this.gameObject.GetComponentInChildren<MeshRenderer>();
    }

    // Update is called once per frame
    void Update()
    {
        // Do a raycast into the world based on the user's
        // head position and orientation.
        var headPosition = Camera.main.transform.position;
        var gazeDirection = Camera.main.transform.forward;

        RaycastHit hitInfo;

        if (Physics.Raycast(headPosition, gazeDirection, out hitInfo))
        {
            // If the raycast hit a hologram...
            // Display the cursor mesh.
            meshRenderer.enabled = true;

            // Move the cursor to the point where the raycast hit.
            this.transform.position = hitInfo.point;

            // Rotate the cursor to hug the surface of the hologram.
            this.transform.rotation = Quaternion.FromToRotation(Vector3.up, hitInfo.normal);
        }
        else
        {
            // If the raycast did not hit a hologram, hide the cursor mesh.
            meshRenderer.enabled = false;
        }
    }
}

 

  • Rebuild the app from File > Build Settings.
  • Return to the Visual Studio solution previously used to deploy to your HoloLens.
  • Select 'Reload All' when prompted.
  • Click Debug -> Start Without debugging or press Ctrl + F5.
  • Now look around the scene and notice how the cursor interacts with the shape of objects.

    youtu.be

Chapter 3 - Gestures

このチャプターでは、gestures の手助けを追加します。ユーザーが紙の球を選択したときに、Unity の物理エンジンを利用した重力を有効にして、球が落とします。

目的

手順

We'll start by creating a script then can detect the Select gesture.

  • In the Scripts folder, create a script named GazeGestureManager.
  • Drag the GazeGestureManager script onto the OrigamiCollection object in the Hierarchy.
  • Open the GazeGestureManager script in Visual Studio and add the following code:

 

GazeGestureManager.cs[show]
using UnityEngine;
using UnityEngine.VR.WSA.Input;

public class GazeGestureManager : MonoBehaviour
{
    public static GazeGestureManager Instance { get; private set; }

    // Represents the hologram that is currently being gazed at.
    public GameObject FocusedObject { get; private set; }

    GestureRecognizer recognizer;

    // Use this for initialization
    void Awake()
    {
        Instance = this;

        // Set up a GestureRecognizer to detect Select gestures.
        recognizer = new GestureRecognizer();
        recognizer.TappedEvent += (source, tapCount, ray) =>
        {
            // Send an OnSelect message to the focused object and its ancestors.
            if (FocusedObject != null)
            {
                FocusedObject.SendMessageUpwards("OnSelect");
            }
        };
        recognizer.StartCapturingGestures();
    }

    // Update is called once per frame
    void Update()
    {
        // Figure out which hologram is focused this frame.
        GameObject oldFocusObject = FocusedObject;

        // Do a raycast into the world based on the user's
        // head position and orientation.
        var headPosition = Camera.main.transform.position;
        var gazeDirection = Camera.main.transform.forward;

        RaycastHit hitInfo;
        if (Physics.Raycast(headPosition, gazeDirection, out hitInfo))
        {
            // If the raycast hit a hologram, use that as the focused object.
            FocusedObject = hitInfo.collider.gameObject;
        }
        else
        {
            // If the raycast did not hit a hologram, clear the focused object.
            FocusedObject = null;
        }

        // If the focused object changed this frame,
        // start detecting fresh gestures again.
        if (FocusedObject != oldFocusObject)
        {
            recognizer.CancelGestures();
            recognizer.StartCapturingGestures();
        }
    }
}

 

  • Create another script in the Scripts folder, this time named SphereCommands.
  • Expand the OrigamiCollection object in the Hierarchy view.
  • Drag the SphereCommands script onto the Sphere1 object in the Hierarchy panel.
  • Drag the SphereCommands script onto the Sphere2 object in the Hierarchy panel.
  • Open the script in Visual Studio for editing, and replace the default code with this:

 

SphereCommands.cs[show]
using UnityEngine;

public class SphereCommands : MonoBehaviour
{
    // Called by GazeGestureManager when the user performs a Select gesture
    void OnSelect()
    {
        // If the sphere has no Rigidbody component, add one to enable physics.
        if (!this.GetComponent<Rigidbody>())
        {
            var rigidbody = this.gameObject.AddComponent<Rigidbody>();
            rigidbody.collisionDetectionMode = CollisionDetectionMode.Continuous;
        }
    }
}

 

  • Export, build and deploy the app to your HoloLens.
  • Look at one of the spheres.
  • Perform the select gesture and watch the sphere drop onto the surface below.

    youtu.be

Chapter 4 - Voice

このチャプターでは、2つの voice commands の手助けを追加します。落ちた球を元の位置に戻す"Reset world" と、球を落とす "Drop sphere" です。

目的

  • バックグラウンドで常にリッスンしているボイス コマンドを追加する
  • ボイス コマンドに対応するホログラムを作成する

手順

  • In the Scripts folder, create a script named SpeechManager.
  • Drag the SpeechManager script onto the OrigamiCollection object in the Hierarchy
  • Open the SpeechManager script in Visual Studio.
  • Copy and paste this code into SpeechManager.cs and Save All:

 

SpeechManager.cs[show]
using System.Collections.Generic;
using System.Linq;
using UnityEngine;
using UnityEngine.Windows.Speech;

public class SpeechManager : MonoBehaviour
{
    KeywordRecognizer keywordRecognizer = null;
    Dictionary<string, System.Action> keywords = new Dictionary<string, System.Action>();

    // Use this for initialization
    void Start()
    {
        keywords.Add("Reset world", () =>
        {
            // Call the OnReset method on every descendant object.
            this.BroadcastMessage("OnReset");
        });

        keywords.Add("Drop Sphere", () =>
        {
            var focusObject = GazeGestureManager.Instance.FocusedObject;
            if (focusObject != null)
            {
                // Call the OnDrop method on just the focused object.
                focusObject.SendMessage("OnDrop");
            }
        });

        // Tell the KeywordRecognizer about our keywords.
        keywordRecognizer = new KeywordRecognizer(keywords.Keys.ToArray());

        // Register a callback for the KeywordRecognizer and start recognizing!
        keywordRecognizer.OnPhraseRecognized += KeywordRecognizer_OnPhraseRecognized;
        keywordRecognizer.Start();
    }

    private void KeywordRecognizer_OnPhraseRecognized(PhraseRecognizedEventArgs args)
    {
        System.Action keywordAction;
        if (keywords.TryGetValue(args.text, out keywordAction))
        {
            keywordAction.Invoke();
        }
    }
}

 

  • Open the SphereCommands script in Visual Studio.
  • Update the script to read as follows:

 

SphereCommands.cs[show]
 
using UnityEngine;

public class SphereCommands : MonoBehaviour
{
    Vector3 originalPosition;

    // Use this for initialization
    void Start()
    {
        // Grab the original local position of the sphere when the app starts.
        originalPosition = this.transform.localPosition;
    }

    // Called by GazeGestureManager when the user performs a Select gesture
    void OnSelect()
    {
        // If the sphere has no Rigidbody component, add one to enable physics.
        if (!this.GetComponent<Rigidbody>())
        {
            var rigidbody = this.gameObject.AddComponent<Rigidbody>();
            rigidbody.collisionDetectionMode = CollisionDetectionMode.Continuous;
        }
    }

    // Called by SpeechManager when the user says the "Reset world" command
    void OnReset()
    {
        // If the sphere has a Rigidbody component, remove it to disable physics.
        var rigidbody = this.GetComponent<Rigidbody>();
        if (rigidbody != null)
        {
            DestroyImmediate(rigidbody);
        }

        // Put the sphere back into its original local position.
        this.transform.localPosition = originalPosition;
    }

    // Called by SpeechManager when the user says the "Drop sphere" command
    void OnDrop()
    {
        // Just do the same logic as a Select gesture.
        OnSelect();
    }
}

 

  • Export, build and deploy the app to your HoloLens.
  • Look at one of the spheres, and say "Drop Sphere".
  • Say "Reset World" to bring them back to their initial positions.
    上記手順を実施したデモ。英語の発音が悪かったので Google 翻訳の声で代用したら素晴らしい認識力。

    youtu.be

Chapter 5 - Spatial sound

このチャプターでは、アプリに音楽を追加し、各アクションをトリガーにしてサウンド エフェクトを設定します。We'll be using spatial sound を使用して、3D 空間の特定の場所にサウンドを設定します。

目的

  • 現実世界でホログラムが聞こえるようにします。

手順

  • In Unity select from the top menu Edit > Project Settings > Audio
  • In the Inspector Panel on the right side, find the Spatializer Plugin setting and select MS HRTF Spatializer.
    f:id:hololensdev:20160922035922p:plain
  • From the Holograms folder in the Project panel, drag the Ambience object onto the OrigamiCollection object in the Hierarchy Panel.
  • Select OrigamiCollection and find the Audio Source component in the Inspector panel. Change these properties:
    • Check the Spatialize property.
    • Check the Play On Awake.
    • Change Spatial Blend to 3D by dragging the slider all the way to the right. The value should change from 0 to 1 when you move the slider.
    • Check the Loop property.
    • Expand 3D Sound Settings, and enter 0.1 for Doppler Level.
    • Set Volume Rolloff to Custom Rolloff.

      f:id:hololensdev:20160922040138p:plain

  • In the Scripts folder, create a script named SphereSounds.
  • Drag and drop SphereSounds to the Sphere1 and Sphere2 objects in the Hierarchy.
  • Open SphereSounds in Visual Studio, update the following code and Save All.

 

SphereSounds.cs[show]
using UnityEngine;

public class SphereSounds : MonoBehaviour
{
    AudioSource audioSource = null;
    AudioClip impactClip = null;
    AudioClip rollingClip = null;

    bool rolling = false;

    void Start()
    {
        // Add an AudioSource component and set up some defaults
        audioSource = gameObject.AddComponent<AudioSource>();
        audioSource.playOnAwake = false;
        audioSource.spatialize = true;
        audioSource.spatialBlend = 1.0f;
        audioSource.dopplerLevel = 0.0f;
        audioSource.rolloffMode = AudioRolloffMode.Custom;

        // Load the Sphere sounds from the Resources folder
        impactClip = Resources.Load<AudioClip>("Impact");
        rollingClip = Resources.Load<AudioClip>("Rolling");
    }

    // Occurs when this object starts colliding with another object
    void OnCollisionEnter(Collision collision)
    {
        // Play an impact sound if the sphere impacts strongly enough.
        if (collision.relativeVelocity.magnitude >= 0.1f)
        {
            audioSource.clip = impactClip;
            audioSource.Play();
        }
    }

    // Occurs each frame that this object continues to collide with another object
    void OnCollisionStay(Collision collision)
    {
        Rigidbody rigid = this.gameObject.GetComponent<Rigidbody>();

        // Play a rolling sound if the sphere is rolling fast enough.
        if (!rolling && rigid.velocity.magnitude >= 0.01f)
        {
            rolling = true;
            audioSource.clip = rollingClip;
            audioSource.Play();
        }
        // Stop the rolling sound if rolling slows down.
        else if (rolling && rigid.velocity.magnitude < 0.01f)
        {
            rolling = false;
            audioSource.Stop();
        }
    }

    // Occurs when this object stops colliding with another object
    void OnCollisionExit(Collision collision)
    {
        // Stop the rolling sound if the object falls off and stops colliding.
        if (rolling)
        {
            rolling = false;
            audioSource.Stop();
        }
    }
}

 

  • Save the script, and return to Unity.
  • Export, build and deploy the app to your HoloLens.
  • Move closer and further from the Stage and turn side-to-side to hear the sounds change.

Chapter 6 - Spatial mapping

最後に、 spatial mapping を利用して現実世界の物体に Origami ゲーム盤を配置します。

目的

  • 仮想世界に現実世界を持ち込みましょう
  • あなたのお気に入りの場所にホログラムを配置しましょう

手順

  • In Unity, click on the Holograms folder in the Project panel.
  • Drag the Spatial Mapping asset into the root of the Hierarchy.
  • Click on the Spatial Mapping object in the Hierarchy.
  • In the Inspector panel, change the following properties:
    • Check the Draw Visual Meshes box.
    • Locate Draw Material and click the circle on the right. Type "wireframe" into the search field at the top. Click on the result and then close the window. When you do this, the value for Draw Material should get set to Wireframe.
  • Export, build and deploy the app to your HoloLens.
  • When the app runs, a wireframe mesh will overlay your real world.
  • Watch how a rolling sphere will fall off the stage, and onto the floor!

    youtu.be

Now we'll show you how to move the OrigamiCollection to a new location:

  • In the Scripts folder, create a script named TapToPlaceParent.
  • In the Hierarchy, expand the OrigamiCollection and select the Stage object.
  • Drag the TapToPlaceParent script onto the Stage object.
  • Open the TapToPlaceParent script in Visual Studio, and update it to be the following:

 

TapToPlaceParent.cs[show]
using UnityEngine;

public class TapToPlaceParent : MonoBehaviour
{
    bool placing = false;

    // Called by GazeGestureManager when the user performs a Select gesture
    void OnSelect()
    {
        // On each Select gesture, toggle whether the user is in placing mode.
        placing = !placing;

        // If the user is in placing mode, display the spatial mapping mesh.
        if (placing)
        {
            SpatialMapping.Instance.DrawVisualMeshes = true;
        }
        // If the user is not in placing mode, hide the spatial mapping mesh.
        else
        {
            SpatialMapping.Instance.DrawVisualMeshes = false;
        }
    }

    // Update is called once per frame
    void Update()
    {
        // If the user is in placing mode,
        // update the placement to match the user's gaze.

        if (placing)
        {
            // Do a raycast into the world that will only hit the Spatial Mapping mesh.
            var headPosition = Camera.main.transform.position;
            var gazeDirection = Camera.main.transform.forward;

            RaycastHit hitInfo;
            if (Physics.Raycast(headPosition, gazeDirection, out hitInfo,
                30.0f, SpatialMapping.PhysicsRaycastMask))
            {
                // Move this object's parent object to
                // where the raycast hit the Spatial Mapping mesh.
                this.transform.parent.position = hitInfo.point;

                // Rotate this object's parent object to face the user.
                Quaternion toQuat = Camera.main.transform.localRotation;
                toQuat.x = 0;
                toQuat.z = 0;
                this.transform.parent.rotation = toQuat;
            }
        }
    }
}

 

  • Export, build and deploy the app.
  • Now you should now be able to place the game in a specific location by gazing at it, using the Select gesture and then moving to a new location, and using the Select gesture again.
    なぜか、スクリプトとオブジェクトのひも付けが解除されていたので再度関連付けをして Build

    youtu.be

Chapter 7 - Holographic fun

目的

手順

Now we'll show you how to uncover the holographic underworld:

  • From the Holograms folder in the Project Panel:
    • Drag Underworld into the Hierarchy to be a child of OrigamiCollection.

      f:id:hololensdev:20160922234913p:plain

  • In the Scripts folder, create a script named HitTarget.
  • In the Hierarchy, expand the OrigamiCollection.
  • Expand the Stage object and select the Target object (blue fan).
  • Drag the HitTarget script onto the Target object.
  • Open the HitTarget script in Visual Studio, and update it to be the following:

 

HitTarget.cs[show]
using UnityEngine;

public class HitTarget : MonoBehaviour
{
    // These public fields become settable properties in the Unity editor.
    public GameObject underworld;
    public GameObject objectToHide;

    // Occurs when this object starts colliding with another object
    void OnCollisionEnter(Collision collision)
    {
        // Hide the stage and show the underworld.
        objectToHide.SetActive(false);
        underworld.SetActive(true);

        // Disable Spatial Mapping to let the spheres enter the underworld.
        SpatialMapping.Instance.MappingEnabled = false;
    }
}

 

  • In Unity, select the Target object.
  • Two public properties are now visible on the Hit Target component and need to reference objects in our scene:
    • Drag Underworld from the Hierarchy panel to the Underworld property on the Hit Target component.
    • Drag Stage from the Hierarchy panel to the Object to Hide property on the Hit Target component.

      f:id:hololensdev:20160922234854p:plain

  • Export, build and deploy the app.
  • Place the Origami Collection on the floor, and then use the Select gesture to make a sphere drop.
  • 球がターゲット (青いファン) に当たったとき、爆発が起きます。Origami コレクションは消され、アンダーワールドへの入り口が現れます。

    youtu.be

The end

このチュートリアルの最後です。

習得したこと:

  • Unity でホログラフィック App をどうやって作成するか
  • gaze, gesture, voice, sound, and spatial mapping の使用方法
  • Visual Studio を使用したアプリのビルドとデプロイ方法

お好きなホログラフィックを作成する準備が整いました!