Mixed Reality Lab 2020, Course Project

• Mixed Reality Lab 2020, Course Project
  • Team
  • Project Description
  • Features
    • Manipulation
    • Plane Slicers
    • Hand Slicer
    • Hand or Near Menu
    • Right-Left Hand
    • Document Viewer
  • CT-Scans
  • Importing Document in Slate

Team

• Dominik Alberto (@doalberto)
  doalbert@student.ethz.ch
• Daniele Chiappalupi (@daniCh8)
  dchiappal@student.ethz.ch
• Jorel Elmiger (@elmigerj)
  elmigerj@student.ethz.ch
• Elena Iannucci (@eleiannu)
  eiannucci@student.ethz.ch
• Hamza Javed (@hamzajaved05)
  javedh@student.ethz.ch

Project Description

The goal of this project is to create a HoloLens 2 application able to assist surgeons during complex fracture surgeries. In some critical cases, bones can be fractured into many separate pieces that need to be realigned and fixed with an artificial supportive structure. The position and orientation of the bone structures is usually not well visible during the surgery, and surgeons have to remember their position from the scan inspection prior to the surgery. The major task is hence to allow surgeons to reinspect the scan data during the surgery and better understand individual bone positions as needed.

Using the HoloLens 2 to do so would be of great benefit, since the surgeon can’t physically touch any object during the surgery, to not compromise the sterilization of the surgery room.

Features

The project we developed provides many features that can be used by the surgeon depending on the way he wants to use the application.

Below is a list of the features that are available right now in the app.

Manipulation

We allow different ways of manipulating the object in the scene. The user can choose whether to move all the objects together locking the relative position within them, or to move just the bone group or the scans group.

You can click on the GIF above to see the full demo-video.

Plane Slicers

The plane slicers can be used to visualize a section of the bone on the windows above it. The two sliders allow to create either a horizontal or a vertical slice of the bone. The colour of the borders of the windows matches the colour of the sliders’ squares.

You can click on the GIF above to see the full demo-video.

Hand Slicer

One of the main feature of the application is to use the hand as a bone slicer. Putting the hand on top of the bone allows to create a section of it with more freedom than using the plane slicers. Moreover, we developed a plane-locking mechanism that allows the user to lock the plane at a given position. The mechanism works in the following way: keeping the thumb aligned with the other fingers will keep slicing the bone, and raising the thumb perpendicular to the other fingers will lock the plane position.

The plane locking mechanism can also be activated using voice-commands. The keyphrase “Stop Tracking” will lock the plane, and the keyphrase “Track My Hand” will unlock it back. See this video with volume for a demonstration.

You can click on the GIF above to see the full demo-video.

Hand or Near Menu

For controlling the scene, we provide the user with a menu that allows him to have much freedom inside the application. This menu exists in two form, the near menu and the hand menu. Both the menu are synchronized in terms of button-states. The default menu is the near one, and the user can then decide which one to use pressing the Switch to Hand Menu button. Likewise, the user can toggle the near menu back pressing the Switch to Near Menu button inside the hand menu.

You can click on the GIF above to see the full demo-video.

Right-Left Hand

The hand slicer can be used either with the right hand or with the left hand. To allow a better precision when tracking the hand slicer, we separate the two cases. By default, the hand slicer uses the right hand. If the user wants to use the left hand, he should first press the relative button on the menu.

You can click on the GIF above to see the full demo-video.

Document Viewer

We provide the user of a document viewer to visualize pre-uploaded documents. This may be useful in a real-world scenario, giving to the surgeon the possibility of visualizing memos and notes about the operation. See the Importing Document in Slate section to know how to learn documents into the project.

You can click on the GIF above to see the full demo-video.

CT-Scans

The medical scans we are provided with are in DICOM format. The pipeline we followed to use those scans is the following:

• We used 3D Slicer to turn them from binary format to nrrd format. Inside Slicer, we imported the DICOM folder containing the files, and saved the data in nrrd extension.
• nrrd format is a very broad one. To ease our computations, we allow only a specific nrrd setting in our project. To convert any nrrd file to the required settings, we used this python script from an older version of the project. Note that this script will override the existing nrrd file.
• Finally, we can use the processed file inside our project. To do so, since in Unity we only work with Text Assets, we need to rename the nrrd file to have a bytes extension (i.e. from ct.nrrd to ct.bytes). This file can be then provided to a script inside Unity to read it. Scripts able to do so can be found in Assets/CT/.

Importing Document in Slate

Displaying PDFs using Unity and C# is not a trivial task. Our first solution to this problem has been to convert each page of the pdf into a picture, and then using each picture to create a material that will be displayed on the Slate. An example of this can be found in Assets/PDF-Viewer/.

After creating a material per slide, these materials should be placed ordered inside the Pages component of the ContentQuad inside the Slate hierarchy. The size parameter should be set accordingly. This will allow the view of any document inside the Slate.

混合现实实验室2020，课程项目记录在Slate中

基本cookies

始终处于活动状态

分析cookies

• Mixed Reality Lab 2020，课程项目团队项目描述功能操作平面切片器手部切片器手部或近菜单左右手文档查看器CT扫描导入板岩文档

基本cookies

• Dominik Alberto (@doalberto)
  doalbert@student.ethz.ch
• Daniele Chiappalupi (@daniCh8)
  dchiappal@student.ethz.ch
• Jorel Elmiger (@elmigerj)
  elmigerj@student.ethz.ch
• Elena Iannucci (@eleiannu)
  eiannucci@student.ethz.ch
• Hamza Javed (@hamzajaved05)
  javedh@student.ethz.ch

始终处于活动状态

这个项目的目标是创建一个hololens2应用程序，能够在复杂的骨折手术中帮助外科医生。在一些严重的情况下，骨骼可能会断裂成许多单独的碎片，需要重新排列并用人工支撑结构固定。在手术过程中，骨结构的位置和方向通常不太明显，外科医生必须在手术前通过扫描检查记住它们的位置。因此，主要任务是让外科医生在手术过程中重新检查扫描数据，并根据需要更好地了解单个骨的位置。

分析cookies

使用HoloLens 2这样做会有很大的好处，因为外科医生在手术过程中不能接触任何物体，不会影响手术室的消毒。

我们开发的项目提供了许多功能，外科医生可以根据其使用应用程序的方式使用这些功能。

Manipulation

下面是应用程序中现在可用的功能列表。

我们允许以不同的方式操纵场景中的对象。用户可以选择是将所有对象移动到一起，锁定其中的相对位置，还是只移动骨骼组或扫描组。

You can click on the GIF above to see the full demo-video.

Plane Slicers

您可以单击上面的GIF查看完整的演示视频。

可以使用平面切片器在其上方的窗口上显示骨骼的一部分。两个滑块允许创建水平或垂直的骨骼切片。窗户边框的颜色与滑块方块的颜色相匹配。

Hand Slicer

One of the main feature of the application is to use the hand as a bone slicer. Putting the hand on top of the bone allows to create a section of it with more freedom than using the plane slicers. Moreover, we developed a plane-locking mechanism that allows the user to lock the plane at a given position. The mechanism works in the following way: keeping the thumb aligned with the other fingers will keep slicing the bone, and raising the thumb perpendicular to the other fingers will lock the plane position.

<>

您可以单击上面的GIF查看完整的演示视频。

该应用程序的主要功能之一是将手用作骨骼切片器。将手放在骨头上可以创建一个比使用平面切片器更自由的部分。此外，我们开发了一个平面锁定机制，允许用户将平面锁定在给定位置。该机制的工作原理如下：保持拇指与其他手指对齐将继续切片骨骼，将拇指垂直于其他手指抬起将锁定平面位置。

Hand or Near Menu

也可以使用语音命令激活飞机锁定机制。按键“停止跟踪”将锁定飞机，按键“跟踪我的手”将解锁飞机。请看这段带音量的视频进行演示。

Right-Left Hand

您可以单击上面的GIF查看完整的演示视频。

为了控制场景，我们为用户提供了一个菜单，允许用户在应用程序内部有很大的自由度。这个菜单有两种形式，近景菜单和手动菜单。两个菜单在按钮状态方面是同步的。默认菜单是附近的菜单，然后用户可以按“切换到手动菜单”按钮来决定使用哪个菜单。同样地，用户可以在手动菜单内按“切换到近菜单”按钮来切换近菜单。

You can click on the GIF above to see the full demo-video.

Document Viewer

您可以单击上面的GIF查看完整的演示视频。

手动切片机可用于右手或左手。为了在跟踪手动切片机时获得更好的精度，我们将这两种情况分开。默认情况下，手部切片器使用右手。如果用户想用左手，应该先按菜单上的相关按钮。

CT-Scans

The medical scans we are provided with are in DICOM format. The pipeline we followed to use those scans is the following:

• We used 3D Slicer to turn them from binary format to nrrd format. Inside Slicer, we imported the DICOM folder containing the files, and saved the data in nrrd extension.
• nrrd format is a very broad one. To ease our computations, we allow only a specific nrrd setting in our project. To convert any nrrd file to the required settings, we used this python script from an older version of the project. Note that this script will override the existing nrrd file.
• Finally, we can use the processed file inside our project. To do so, since in Unity we only work with Text Assets, we need to rename the nrrd file to have a bytes extension (i.e. from ct.nrrd to ct.bytes). This file can be then provided to a script inside Unity to read it. Scripts able to do so can be found in Assets/CT/.

Importing Document in Slate

您可以单击上面的GIF查看完整的演示视频。