跨手语迁移学习:域自适应与多尺度时间对齐

Cross-Sign Language Transfer Learning Using Domain Adaptation with Multi-scale Temporal Alignment

精选理由

手语识别资源少,这篇论文用域自适应做跨手语迁移,比传统迁移学习效果好,还比较了RGB和光流,值得做手语识别的朋友看看。

AI 摘要

该研究针对超100种手语识别资源匮乏的问题,提出使用域自适应方法TA3N进行跨手语迁移学习。TA3N利用时间关系网络(TRN)模块对齐多尺度时间关系,实验显示域自适应优于神经网络迁移学习,尤其提升美国手语(ASL)识别。研究还发现对齐源域和目标域的短期时间特征更有效。在RGB与光流模式对比中,RGB在多数情况下表现更优。

原文 · arXiv cs.AI

Cross-Sign Language Transfer Learning Using Domain Adaptation with Multi-scale Temporal Alignment

Sign language serves as a vital means of communication for individuals with hearing impairments, yet recognition resources for the over 100 distinct sign languages are severely lacking. In response, we present our work on sign language recognition using transfer learning and the domain adaptation method TA3N, which utilizes the Temporal Relational Network (TRN) module for aligning multi-scale temporal relations. Our findings highlight the superior performance of Domain Adaptation to neural network-based transfer learning, particularly in improving recognition of American Sign Language (ASL). Our research also identifies the effectiveness of aligning shorter-term temporal features between source and target domains. In addition to using RGB, we conducted experiments using Optical Flow mode for the sign language samples, ultimately determining that RGB outperforms Optical Flow in the majority of cases. Our work aims to improve accessibility and communication for individuals who rely on sign language as their primary mode of communication.