DEAR: Depth-Estimated Action Recognition

Sadegh Rahmani[1], Filip Rybansky[2], Quoc Vuong[2], Frank Guerin[1], Andrew Gilbert[1],
[1] Unviersity of Surrey [2] Unviersityof Newcastle
The European Conference of Computer Vision 2024 Human-inspired Computer Vision Workshop
Paper Poster Code

We perform supervised action recognition using a fusion of RGB and depth map frames. We used S4V network for processing RGB and the same network plus VideoMamba to extract depth features. Gated cross-attention (GCA) has been used for fusing modalities, while mean operation has been selected for fusing models' scores.

Abstract

Detecting actions in videos, particularly within cluttered scenes, poses significant challenges due to the limitations of 2D frame analysis from a camera perspective. Unlike human vision, which benefits from 3D understanding, recognizing actions in such environments can be difficult. This research introduces a novel approach integrating 3D features and depth maps alongside RGB features to enhance action recognition accuracy. Our method involves processing estimated depth maps through a separate branch from the RGB feature encoder and fusing the features to understand the scene and actions comprehensively. Using the Side4Video framework and VideoMamba, which employ CLIP and VisionMamba for spatial feature extraction, our approach outperformed our implementation of the Side4Video network on the Something-Something V2 dataset.

Poster

BibTeX

@inproceedings{Rahmani:Dear:ECCVWS:2024,
        AUTHOR = Rahmani, Sadegh and Rybansky, Filip and Vuong, Quoc and Guerin, Frank and Gilbert, Andrew ",
        TITLE = "DEAR: Depth-Estimated Action Recognition",
        BOOKTITLE = " The European Conference of Computer Vision 2024, Human-inspired Computer Vision Workshop",
        YEAR = "2024",
        }