To build Video Question Answering (VideoQA) systems capable of assisting humans in daily activities, seeking answers from long-form videos with diverse and complex events is a must. Existing multi-modal VQA models achieve promising performance on images or short video clips, especially with the recent success of large-scale multi-modal pre-training. However, when extending these methods to long-form videos, new challenges arise. On the one hand, using a dense video sampling strategy is computationally prohibitive. On the other hand, methods relying on sparse sampling struggle in scenarios where multi-event and multi-granularity visual reasoning are required. In this work, we introduce a new model named $\mathcal{M}ulti{-}$ · modal Iterative $\mathcal{S}$ .patial-temporal Transformer $(\mathcal{MIST})$ ) to better adapt pre-trained models for long-form VideoQA. Specifically, $\mathcal{MIST}$ decomposes traditional dense spatial-temporal self-attention into cascaded segment and region selection modules that adaptively select frames and image regions that are closely relevant to the question itself. Visual concepts at different granularities are then processed efficiently through an attention module. In addition, $\mathcal{MIST}$ iteratively conducts selection and attention over multiple layers to support reasoning over multiple events. The experimental results on four VideoQA datasets, including AGQA, NExT-QA, STAR, and Env-QA, show that $\mathcal{MIST}$ achieves state-of-the-art performance and is superior at efficiency. The code is available at github.com/showlab/mist.

Differentiable Top-k with Optimal Transport