Skip to content

ExoMem: Augmented Reality based human memory enhancement system using AI

License

Notifications You must be signed in to change notification settings

IS2AI/ExoMem-AR-Memory

Repository files navigation

ExoMem_github_april

This repository contains the source code developed for the AR-based human memory augmentation system, ExoMem. ExoMem is the first Augmented Reality (AR) and Artificial Intelligence (AI) enhanced head-worn system that constructs a holographic visuospatial memory in an indoor environment. Microsoft HoloLens AR Goggles sense the environment, exchange data over a wireless network with a computing module (PC with Ubuntu 16.04), and construct a holographic visuospatial memory. The computing module performs computer vision-based localization and object detection on first-person view (FPV) data received from the HoloLens.

Dependencies

AI environment perception

CV-based user localization using ArUco fiducial markers

  1. Ubuntu 16.04 https://releases.ubuntu.com/16.04/

  2. ROS Kinetic http://wiki.ros.org/kinetic/Installation/Ubuntu

  3. Create and build a catkin workspace:

  • $ mkdir -p ~/catkin_ws/src
  • $ cd ~/catkin_ws/
  • $ catkin_make
  1. Copy and paste folder ros_aruco to the directory ~/catkin_ws/src:
  • $ sudo apt-get install ros-kinetic-fiducials
  • $ catkin_make
  1. Update the system. If not updated remove the old key and import the new key:
  • $ sudo apt-get update
  • $ sudo apt-key del 421C365BD9FF1F717815A3895523BAEEB01FA116
  • $ sudo -E apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
  • $ sudo apt-get update
  1. Check the system:
  • roscore
  • cd catkin_ws/
  • source devel/setup.bash
  • rosrun ros_aruco aruco_node

Object recognition using YOLO V3 object detector

  1. Install Python 3.7 from Source Code (Python 2.7 is already installed during the ROS Kinetic installatipon in Step 2) https://www.osetc.com/en/how-to-install-the-latest-python-3-7-on-ubuntu-16-04-or-18-04.html
  • $ sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.7 2
  • $ sudo update-alternatives --install /usr/local/bin/python python /usr/local/bin/python3.7 3
  1. Istall OpenCV to Python 3.7.x, but first upgrade pip3 and setuptools to the latest versions
  • $ pip3 install --upgrade pip setuptools
  • $ sudo pip3 install opencv-contrib-python
  1. Copy and paste folder AI_Environment_Perception to the directory ~/home

  2. Download yolov3-tiny.weights.zip https://drive.google.com/drive/folders/1d0y73MaLYDGnXHFyNfsM0MxcvIwlrBDm

  • Unzip and paste yolov3-tiny.weights to the directory ~/home/AI_Environment_Perception
  1. Check the system, 172.20.10.4 is the IP address of HoloLens:
  • $ cd AI_environment_perception/
  • $ sudo update-alternatives --config python
  • $ unset PYTHONPATH
  • $ python AI_environment_perception.py -a 172.20.10.4

AR environment

  1. Visual Studio 2019 https://visualstudio.microsoft.com/ru/vs/older-downloads/
  • In the Visual Studio Installer select the following components
  • .Net desktop development
  • Universal Windows Platform development
  • Desktop development with C++
  • .Net Core cross-platform development
  1. Unity 2018.4.28f1 https://unity3d.com/get-unity/download/archive

Unity application for creating holographic visuospatial memory in AR

  • Download ExoMem_Record_Memory_HoloLens.zip from https://drive.google.com/drive/folders/1CuzTKzWoNJYEFS3oJBJ-ZdGjIIBZWmXQ
  • Go to the directory ~/ExoMem_Record_Memory_HoloLens/App/ExoMem_Record_Memory_App
  • Rigth click ExoMem_Record_Memory_HoloLens.sln and open with Microsoft Visual Studio 2019
  • Visual Studio 2019, select (Release, ARM, Remote Machine)
  • In Debug Properties, Machine Name: type the HoloLens' IP Address
  • In Debug, select Start Without Debugging

Unity application for loading holographic visuospatial memory in AR

  • Download ExoMem_Load_Memory_HoloLens.zip from https://drive.google.com/drive/folders/1CuzTKzWoNJYEFS3oJBJ-ZdGjIIBZWmXQ
  • Go to the directory ~/ExoMem_Load_Memory_HoloLens/App/ExoMem_Load_Memory
  • Rigth click ExoMem_Load_Memory_HoloLens.sln and open with Microsoft Visual Studio 2019
  • Visual Studio 2019, select (Release, ARM, Remote Machine)
  • In Debug Properties, Machine Name: type the HoloLens' IP Address
  • In Debug, select Start Without Debugging

Unity application for running the computer-based test during map-pointing activity

  • Download Retrieve_Memory_Computer_Based_Test.zip from https://drive.google.com/drive/folders/1CuzTKzWoNJYEFS3oJBJ-ZdGjIIBZWmXQ
  • Open Unity 2018.4.28f1, at the start menu select open the project and go to the directory Retrieve_Memory_Computer_Based_Test
  • Go to scenes, open input_user_data_scene and run the program
  • Select AR:0, Day:1, ID: 1 (for example)
  • Go to scenes, open load_memory_desktop and run the program
  • Select AR:0, Day:1, ID: 1 (for example)

ExoMem

Create the holographic visuospatial memory in AR

  1. Start the ros master
  • Open a new terminal, roscore will use python 2.7
  • $ sudo update-alternatives --config python
  • Choose the version of python 2.7
  • $ unset PYTHONPATH
  • $ source /opt/ros/kinetic/setup.bash
  • $ roscore
  1. Run the aruco_node
  • Go to the catkin_ws directory in ros environment
  • $ cd catkin_ws/
  • $ source devel/setup.bash
  • $ rosrun ros_aruco aruco_node
  1. Run the python code
  • Go to the directory where the python code is stored
  • $ cd AI_environment_perception/
  • For the code the python 3.7 will be used, therefore run the following command
  • $ sudo update-alternatives --config python
  • Choose the version of python 3.7
  • $ unset PYTHONPATH
  • $ python AI_environment_perception.py -a 172.20.10.4
  1. Run the Unity application in HoloLens 2
  • ExoMem_Record_Memory_HoloLens
ExoMem_AR_Memory_memorization_activity.mp4
ExoMem_AR_Memory_computing_module.mp4

Load the holographic visuospatial memory in AR

  1. Run the Unity application in HoloLens 2
  • ExoMem_Load_Memory_HoloLens

Computer-based test for map-pointing activity

  1. Run the Unity application in Desktop computer
  • Retrieve_Memory_Computer_Based_Test
ExoMem_AR_Memory_map_pointing_activity.mp4