论文分享 | 多模态大模型相关研究进展

1. CTC-Assisted LLM-Based Contextual ASR
2. An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models
3. NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics
4. Open-World Task and Motion Planning via Vision-Language Model Inferred Constraints
5. Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Models

1.CTC-Assisted LLM-Based Contextual ASR

Authors:Guanrou Yang, Ziyang Ma, Zhifu Gao, Shiliang Zhang, Xie Chen

https://arxiv.org/abs/2411.06437

论文摘要

Contextual ASR or hotword customization holds substan tial practical value. Despite the impressive performance of current end-to-end (E2E) automatic speech recognition (ASR) systems, they often face challenges in accurately recognizing rare words. Typical E2E contextual ASR mod els commonly feature complex architectures and decoding mechanisms, limited in performance and susceptible to in terference from distractor words. With large language model (LLM)-based ASR models emerging as the new mainstream, we propose a CTC-Assisted LLM-Based Contextual ASR model with an efficient filtering algorithm. By using coarse CTC decoding results to filter potential relevant hotwords and incorporating them into LLM prompt input, our model attains WER/B-WER of 1.27%/3.67% and 2.72%/8.02% on the Librispeech test-clean and test-other sets targeting on recognizing rare long-tail words, demonstrating significant improvements compared to the baseline LLM-based ASR model, and substantially surpassing other related work. More remarkably, with the help of the large language model and proposed filtering algorithm, our contextual ASR model still performs well with 2000 biasing words.

论文简评

这篇论文提出了一种通过利用粗粒度CTC解码结果进行过滤的上下文ASR模型，旨在提高对罕见单词的识别准确性。该模型在Librispeech数据集上的表现优于基准模型，特别是在WER和B-WER方面。实验结果显示，该方法显著提升了ASR性能。此外，将大语言模型与CTC编码相结合的研究方向，在当前的上下文ASR领域具有一定的创新和实用性。总的来说，该研究为解决罕见长尾词识别问题提供了新的思路，并有望推动这一领域的进一步发展。

2.An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models

Authors:Fatemeh Shiri, Xiao-Yu Guo, Mona Golestan Far, Xin Yu, Gholamreza Haffari, Yuan-Fang Li

https://arxiv.org/abs/2411.06048

论文摘要

Large Multimodal Models (LMMs) have achieved strong performance across a range of vision and language tasks. However, their spatial reasoning capabilities are under investigated. In this paper, we construct a novel VQA dataset, Spatial-MM, to compre hensively study LMMs’ spatial understanding and reasoning capabilities. Our analyses on object-relationship and multi-hop reasoning re veal several important findings. Firstly, bound ing boxes and scene graphs, even synthetic ones, can significantly enhance LMMs’ spa tial reasoning. Secondly, LMMs struggle more with questions posed from the human perspec tive than the camera perspective about the im age. Thirdly, chain of thought (CoT) prompt ing does not improve model performance on complex multi-hop questions involving spa tial relations. Lastly, our perturbation analysis on GQA-spatial reveals that LMMs are much stronger at basic object detection than complex spatial reasoning. We believe our benchmark dataset and in-depth analyses can spark further research on LMMs spatial reasoning.

论文简评

这篇论文的主要贡献在于引入了Spatial-MM数据集来评估大规模多模态模型（LMM）的空间推理能力。通过分析LMM在空间理解方面的表现，这篇论文揭示了与相机视角相比，LMM在处理人类视角问题时的困难，以及链式思考提示对解决复杂多步推理问题的帮助有限。研究发现，尽管边界框和场景图能够增强空间推理能力，但当前的LMM通常更擅长基本对象检测而非高级空间推理，而在处理复杂空间推理任务时存在不足。

论文的关键观点是：首先，该文为从不同角度探索空间推理的重要性提供了基础；其次，其结论为未来研究方向提供了一定的指导意义，即需要进一步优化LMM的性能以满足更复杂的空间推理需求。总的来说，这篇论文不仅填补了现有关于LMM空间推理能力研究的空白，而且对相关领域的发展具有重要的启示作用。

3.NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics

Authors:David Robinson, Marius Miron, Masato Hagiwara, Olivier Pietquin

https://arxiv.org/abs/2411.07186

论文摘要

Large language models (LLMs) prompted with text and audio represent the state of the art in various auditory tasks, including speech, music, and general audio, showing emergent abilities on unseen tasks. However, these capabilities have yet to be fully demonstrated in bioacoustics tasks, such as detecting animal vocaliza tions in large recordings, classifying rare and endangered species, and labeling context and behavior—tasks that are crucial for conservation, biodiversity mon itoring, and the study of animal behavior. In this work, we present NatureLM audio, the first audio-language foundation model specifically designed for bioa coustics. Our carefully curated training dataset comprises text-audio pairs span ning a diverse range of bioacoustics, speech, and music data, designed to address the challenges posed by limited annotated datasets in the field. We demonstrate successful transfer of learned representations from music and speech to bioacous tics, and our model shows promising generalization to unseen taxa and tasks. Im portantly, we test NatureLM-audio on a novel benchmark (BEANS-Zero) and it sets the new state of the art (SotA) on several bioacoustics tasks, including zero shot classification of unseen species. To advance bioacoustics research, we also open-source the code for generating training and benchmark data, as well as for training the model.

论文简评

NatureLM-audio是阿里巴巴云研发的一项创新成果，基于大型语言模型设计了一种针对生物声学任务的独特音频语言基础模型。该研究旨在解决生物声学领域的一个重要问题：如何从音乐和语音中学习到生物声学知识。NatureLM-audio不仅展示了其在零样本分类方面的强大性能，且在不同物种的表现也令人印象深刻，并成功实现了跨域学习，将音乐和语音的知识应用于生物声学领域。此外，作者还提供了一个开源代码库和一个名为BEANS-Zero的新基准集，用于评估模型在不同生物声学任务上的表现。这一系列工作为生物声学领域的研究提供了新的思路和方法，也为其他领域的研究者们提供了宝贵的参考和资源。总之，NatureLM-audio是一个具有重要意义的研究成果，它的出现为生物声学领域的未来发展带来了无限可能。

4.Open-World Task and Motion Planning via Vision-Language Model Inferred Constraints

Authors:Nishanth Kumar, Fabio Ramos, Dieter Fox, Caelan Reed Garrett

https://arxiv.org/abs/2411.08253

论文摘要

Foundation models trained on internet-scale data, such as Vision Language Models (VLMs), excel at performing tasks involving common sense, such as visual question answering. Despite their impressive capabilities, these models cannot currently be directly applied to challenging robot manipulation problems that require complex and precise continuous reasoning. Task and Motion Planning (TAMP) systems can control high-dimensional continuous systems over long horizons through combining traditional primitive robot operations. How ever, these systems require detailed model of how the robot can impact its envi ronment, preventing them from directly interpreting and addressing novel human objectives, for example, an arbitrary natural language goal. We propose deploy ing VLMs within TAMP systems by having them generate discrete and continu ous language-parameterized constraints that enable TAMP to reason about open world concepts. Specifically, we propose algorithms for VLM partial planning that constrain a TAMP system’s discrete temporal search and VLM continuous constraints interpretation to augment the traditional manipulation constraints that TAMP systems seek to satisfy. We demonstrate our approach on two robot em bodiments, including a real world robot, across several manipulation tasks, where the desired objectives are conveyed solely through language.

论文简评

这篇论文提出了一个名为OWL-TAMP的框架，旨在将视觉语言模型（VLM）集成到任务和运动规划（TAMP）系统中，以增强机器人抓取能力。它介绍了生成连续和离散约束的方法，这些约束允许TAMP系统解释开放世界概念，并执行涉及对象操作的任务。实验结果显示，与基准方法相比，提出的解决方案在各种抓取任务中的成功率更高。此外，这项工作具有重要性和前瞻性，随着人工智能技术的进步及其在机器人领域的应用，这一研究领域正变得越来越重要。

5.Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Models

Authors:Arshia Hemmat, Adam Davies, Tom A. Lamb, Jianhao Yuan, Philip Torr, Ashkan Khakzar, Francesco Pinto

https://arxiv.org/abs/2411.06287

论文摘要

Despite the importance of shape perception in human vision, early neural im age classifiers relied less on shape information for object recognition than other (often spurious) features. While recent research suggests that current large Vision Language Models (VLMs) exhibit more reliance on shape, we find them to still be seriously limited in this regard. To quantify such limitations, we introduce IllusionBench, a dataset that challenges current cutting-edge VLMs to deci pher shape information when the shape is represented by an arrangement of visual elements in a scene. Our extensive evaluations reveal that, while these shapes are easily detectable by human annotators, current VLMs struggle to rec ognize them, indicating important avenues for future work in developing more robust visual perception systems. The full dataset and codebase are available at: https://arshiahemmat.github.io/illusionbench/

论文简评

该篇论文以IllusionBench为新数据集，旨在评估视觉语言模型（VLM）在形状识别能力方面的局限性。研究表明，尽管人类注释者可以轻松识别形状，但许多最先进的VLM却表现不佳，这一事实凸显了它们在视觉感知方面的重大局限性。论文通过零样本、少样本以及领域通用化等场景的深入评估，揭示了VLM更倾向于依赖场景元素而非形状识别的趋势。

论文的核心在于它直接面对当前VLM们在形状识别上的局限性，并通过一系列实验展示了其在不同任务中的性能差异。这些实验不仅强调了人类视觉认知的重要性，还进一步证明了形状在视觉识别中的核心地位。这一发现对未来VLM的研究具有重要意义，因为它提醒我们在构建和优化VLM时应更加关注形状感知能力的提升。总的来说，这篇论文是一个非常有价值的成果，为我们理解VLM如何处理复杂视觉信息提供了新的视角，并对未来的视觉计算机科学研究具有重要的指导意义。

我们欢迎您在评论区中留下宝贵的建议！包括但不限于：

• 可以提出推文中论文简评的不足！
• 可以分享最近更值得推荐的论文并给出理由！

推荐阅读