Ego Exo4D

Ego-Exo4D: A Multimodal, Multi-view Video Dataset for Skill-based Human Activities

Ego-Exo4D is an innovative multimodal, multi-view video dataset and benchmark challenge designed to capture both first-person and external perspectives of skill-based human activities. This comprehensive dataset serves as a valuable resource for advancing research in multi-modal machine perception, particularly in understanding and analyzing daily human activities.

Dataset Collection

Ego-Exo4D was meticulously collected through the participation of 839 volunteers across 13 cities worldwide. These contributors wore specialized cameras while performing various skill-based tasks, resulting in an extensive collection of 1422 hours of video data. The dataset adheres strictly to ethical and privacy standards, with all participants providing informed consent.

Unique Features

Ego-Exo4D offers a rich combination of multi-modal data types:

• Synchronized First-person and External Perspectives: The dataset captures both the personal viewpoint (first-person) and external viewpoints, providing a comprehensive understanding of human activities from multiple angles.
• Natural Language Descriptions: It includes three distinct types of text data paired with video:
  • Expert Annotations: Detailed descriptions provided by domain experts.
  • Participant-Provided Narratives: Tutorial-style explanations from the volunteers themselves.
  • Atomic Action Descriptions: Concise, one-sentence descriptions of individual actions.
• Multi-Sensory Modalities: Beyond video data, Ego-Exo4D incorporates various sensory inputs:
  • Multiple Microphone Arrays
  • Two IMU (Inertial Measurement Unit) sensors
  • A Barometer
  • A Magnetometer

Research Applications

Ego-Exo4D is specifically designed to support research in:

• Multi-modal Machine Perception: Combining visual, auditory, and sensory data for advanced activity recognition and understanding.
• Skill-based Human Activity Analysis: Investigating complex human movements and tasks through both first-person and external perspectives.

For more information about Ego-Exo4D and its research applications, please visit the official website.