Data Overview

1. Structure of the data

The data in the storage is always structured in the same way. The data is stored in the following format:

- Project
    - Participant_1
        - Task_1
            - sensor_data.txt
            - video_data.mp4
        - Task_2
        - Task_n
    - Participant_2 
    - Participant_n

2. Sensor Data

The format of the sensor data file is .txt. The file contains:

A header, with details of the sensors and their sampling rates (50Hz) and divisors, together with general details such as timestamp, connect device, etc:

  • File/DateTime: the date and time this file was created
  • File/FileFormat: the internal format ID of this file
  • File/User: the apps username assigned to this data
  • File/Tags: the tags assigned to the user
  • File/Category,Video: the task category
  • File/Exercise: the details of the task i.e. for video task this will be the video name
  • ConnectDevice/Name: the name of the iPad used for data collection
  • Software/Build.buildTag
  • Firmware/Build.buildTag
  • Device/Version.serialId
  • Device/Version.hardware: version numbers of software, firmware and hardware
  • Accelerometer/Gyroscope/Magnetometer/etc: the sensors that are used by the OCOsense glasses, with their sampling rates (50 Hz), divisors, etc.
  • Calibration: which expressions have been calibrated

The data: short description

  • Timestamp: Unix format timestamp of each dataframe
  • Accelerometer/Raw: raw data from the 3 accelerometer channels
  • Gyroscope/Raw: raw data from the 3 gyroscope channels
  • Magnetometer/Raw: raw data from the 3 magnetometer channels
  • Eular/Raw: raw data from the 3 Euler angles
  • Nav/Raw: raw data from the 6 navigation sensors
  • Proximity/Raw: raw data from the 6 proximity sensors
  • Pressure/Raw: raw data from the pressure sensor
  • Battery: the percentage of remaining power on the OCOsense glasses
  • RSSI: the relative quality level of the Bluetooth signal received on the tablet
  • DebugBits/Flags: the quality of the navigation & proximity sensor data
  • MicBits: whether the front or rear facing microphones are detecting audio

Algorithm output:

  • OCO/State: the state of the glasses (0=not worn, 1=calibrating neutral, 2=ready)
  • Expression intensity is the current size of expression recognized by the analysis
  • Expression certainty is how confident the analysis is of its recognition results
  • The higher statistics from the analysis are:
    • Attention (time spent attending to the screen)
    • Expressivity (time spent in recognized expressions)
    • Interaction (amount of screen interaction)
    • Valence (positive vs negative expressions over time)
    • Engagement (a combination of expressivity & attention)
  • Annotations: the user defined label strings from the app

3. Video Data

The format of the video data is always the same. The video is stored in the .mp4 format. The resolution of the video is 640x480, codec H265. For opening the video, we recommend using the VLC player.

4. Processing Outputs

The format of the output.csv file has the following columns:

  • Timestamp: Unix format timestamp of each dataframe
  • Expression/Type: The type of expression recognized by the analysis

The format of the output.json file is a JSON file with the following structure:

  • start_timestamp: Unix format timestamp of the start of the task
  • end_timestamp: Unix format timestamp of the end of the task
  • labels: the label of the expression/activity recognized by the analysis

The format of the output_proba.json file is a JSON file with the following structure:

  • pred_proba_0.0: the probability of the expression/activity being 0.0
  • pred_proba_1.0: the probability of the expression/activity being 1.0
  • pred_proba_2.0: the probability of the expression/activity being 2.0
  • pred_proba_3.0: the probability of the expression/activity being 3.0
  • pred_proba_4.0: the probability of the expression/activity being 4.0
  • pred_proba_5.0: the probability of the expression/activity being 5.0

The format of the output.mp4 file is a video file with the recognized expressions overlaid on the video. case