Back to overview

Deep Sign: Hybrid CNN-HMM for Continuous Sign Language Recognition

Type of publication Peer-reviewed
Publikationsform Proceedings (peer-reviewed)
Author Koller O, Zargaran S, Ney H, Bowden R,
Project SMILE: Scalable Multimodal sign language Technology for sIgn language Learning and assessmEnt
Show all

Proceedings (peer-reviewed)

Title of proceedings Proceedings of British Machine Vision Conference
Place York, UK
DOI 10.5244/c.30.136

Open Access


This paper introduces the end-to-end embedding of a CNN into a HMM, while inter- preting the outputs of the CNN in a Bayesian fashion. The hybrid CNN-HMM combines the strong discriminative abilities of CNNs with the sequence modelling capabilities of HMMs. Most current approaches in the field of gesture and sign language recognition disregard the necessity of dealing with sequence data both for training and evaluation. With our presented end-to-end embedding we are able to improve over the state-of-the-art on three challenging benchmark continuous sign language recognition tasks by between 15% and 38% relative and up to 13.3% absolute.