Sign Language Processing

Here’s a collection of Sign Language related resources. It focuses mainly on literature on automatic Sign Language processing. Other links were collected as a result of this main search and included whenever found to be potentially interesting. This is of course an ongoing effort. If you have contributions not listed here, please send me a link (your contribution will of course be acknowledged).

Additionally, you may want to look at my Multimodality page.

Sign Languages are visual languages consisting of formalized gestures, facial expressions including lip movement and eye-gaze. Gestures usually involve whole body movements, in particular movements of the hands and arms. Facial expressions are commonly used as part of the language. In American Sign Language (ASL), for instance, determining if an utterance is a statement of fact or a question, depends on eyebrow and facial configurations (raised eyebrows on a tilted forward head are used to mark questions); in some sentences, determining if it is affirmative or negative depends on interpreting head movements. Being full languages, Sign Languages are complex and pose at least similar obstacles to machine processing as spoken languages do.

Sign language processing constitutes a specialization of the broader human movement analysis research and is anchored on similar techniques as the ones employed for, e.g. gesture and facial analysis in general. There are a variety of algorithms that target gesture (both through vision and through instrumented gloves), facial expression and lipreading, as well as other sources of data (modalities) such as eye-gaze detection. Most of the work targets one of the possible modalities independently from the others, particularly in Sign Language processing, where the bulk of the work is done on interpretation of hand-gestures, or even more narrowly on fingerspelling. Existing techniques for analyzing independent human motion modalities are mainly based on traditional processing tools and methods, e.g., Hidden Markov Models (HMMs) (e.g. Starner and Pentland, 1996; Assan and Grobel, 1998; Hienz et al, 1999), neural nets (e.g. Wilson and Anspach, 1993; Erenshteyn et al., 1994; Vamplew, 1996), rules extracted by a temporal concept learner (Kadous, 1998), bayesian nets (Garg et al., 2000), finite state machines (Hong et al., 2000), colored Petri nets coupled to HMMs (Huang and Jeng, 2001).

Research Centers / Projects

Listed are centers that did research in the listed areas at some point in time. Some continue their research efforts, others not. Researchers may also have moved to other centers. We list the year the last related publication appeared in press.

Center/Project	Last pub	People
University of New South Wales	2002	Mohammed Waleed Kadous
Visicast Project	2002
	2001	Zon Wei; Yuan Kui; Liu Jindong; Luo Bencheng
Aachen University of Technology	2001	Hermann Hienz, Britta Bauer, Kirsti Grobel, Karl-Friedrich Kraiss, Marcell Assan
Nagoya Institute of Technology	2001	Marcus Vinicius Lamar (U Federal do Paraná, Brazil), Md.S.Bhuiyan, A.Iwata
National Tsing-Hua University, Taiwan	2001	Chung-Lin Huang, Ming-Shan Wu, Sheng-Hung Jeng
Paris XI Orsay University	2001	Annelies Braffort
University of Macedonia in Thessaloniki, Greece	2001	Vassilia N. Pashaloudi
University of Pennsylvania	2001	Christian Vogler, Dimitris Metaxas
University of Western Australia	2001	Eun-Jung Holden, Robyn Owens, Geoffrey G. Roy
	2000	K. Imagawa, Shan Lu, Seiji Igi, Hideaki Matsuo, Yuji Nagashima, Yuji Takata, and Terutaka Teshima
Hitachi Research Laboratory	2000	Hirohiko Sagawa, Masaru Takeuchi, Masaru Ohki, Tomoko Sakiyama, Eiji Oohira, Hisashi Ikeda, Hiromichi Fujisawa
Institute of Computing Tech, Chinese Academy of Sciences	2000	Wen Gao, Jiyong Ma, Jiangqin Wu Chunli Wang
Michigan State University	2000	Yuntao Cui, John J. Weng
MIT · Mechanical Engineering Department	2000	Vivek A. Sujan, Marco A. Meggiolaro (PUC, Brazil)
	1999	Sugiyama, H.; Tanahashi, S.; Aoki, Y. - Seong-Hyo Shin; Sang-Woon Kim
Kyoto Institute of Technology	1999	Takao Kurokawa, Sumihiro Kawano, K. Senba
University of Illinois at Urbana-Champaign Beckman Institute	1999	Ming-Hsuan Yang, Narendra Ahuja
University of Ljubljana	1999	Franc Solina, Slavko Krapez, Ales Jaklic
Communications Research Laboratory, Japan	1998	Seiji Igi, Shan Lu, K. Imagawa, H. Matsuo, H. Sakato, Y. Nagashima
DalTech (Technical University of Nova Scotia)	1998	Moussa Habib Abdallah, A.E. Marble, Charoensak Charayaphan
MIT Media Lab	1998	Alex Pentland, Thad Starner, Joshua Weaver
National Taiwan University	1998	Rung-Huei Liang, Ming Ouhyoung (Shih-Chien University, Taiwan)
Ohio State University	1998	Chung-Lin Huang
	1997	Yamamoto, Y., Uchida, Masafumi, Ide, Hideto
	1997	Henrik Birk, Thomas B. Moeslund, Claus B. Madsen
KAIST, Korea, Electrical Engineering	1997	Zeungnam Bien; Gyu-Tae Park; Won Jang (Agency for Devence Development, Korea); Jong-Sung Kim
University of Delaware	1997	Roman Erenshteyn, Pavel Laskov, Richard A. Foulds, Garland Stern, Lynn Messing
University of Essex, UK	1997	G.J. Sweeney, Andy C. Dowton
University of Tasmania	1996	Peter Vamplew, Anthony Adams
	1995	Yamaguchi T., Yoshihara M., Akiba M., Kuga M., Kanazawa N. and Kamata K
	1995	Chung-Lin Huang; Wen-Yi Huang; Cheng-Chang Lien
Ohio State University, Biomedical Engineering	1995	M.B. Waldron, S. Kim
	1994	Hamilton J. and Micheli-Tzanakou E.
	1994	Geoff D. Roberts
Simon Fraser University, Canada	1994	Brigitte Dorner, Eli Hagen
	1993	A. Sutherland
Raytheon Co	1993	Elizabeth J. Wilson, Gretel Anspach
University of Tokyo	1993	Tosiyasu L. Kunii, Jintae Lee
	1988	S. Tamura, S. Kawasaki
	1982	R. Harrison

Web resources

Sign Language
American Sign Language Research Project at Boston U	http://www.bu.edu/asllrp/
National Center for Sign Language and Gesture Resources	http://www.bu.edu/asllrp/cslgr/
Communications Research Laboratory, Japan - Research and development on sign language recognition	http://www2.crl.go.jp/jt/a131/research/univ-e.html
Leiden U’s Sign Phonology Group Sign language sites on the WWW	http://www.leidenuniv.nl/hil/sign-lang/sl-sites.html
Sign Stream	http://www.bu.edu/asllrp/SignStream/
U of Hamburg’s International Bibliography of Sign Language	http://www.sign-lang.uni-hamburg.de/bibweb/
Waleed’s page on Machine Gesture and Sign Language Recognition	http://www.cse.unsw.edu.au/~waleed/gsl-rec/
Beckman Institute’s Gesture Interpretation using Spatio-Temporal analysis – GIST	http://vision.ai.uiuc.edu/mhyang/gist.html

Animation
DePaul University	http://asl.cs.depaul.edu/relwork.html
Signing avatar	http://www.signingavatar.com/
Visicast	http://www.visicast.sys.uea.ac.uk

Transcription / Notation
HamNoSys	http://www.sign-lang.uni-hamburg.de/Projects/HamNoSys.html
SignWriting	http://www.SignWriting.org/
U Hamburg Bibliography “Transcription / notation”	http://www.sign-lang.uni-hamburg.de/bibweb/Lidat.acgi?KEYWORDALTID=459

Translation
Visicast	http://www.visicast.sys.uea.ac.uk
Penn’s TEAM	http://www.cis.upenn.edu/~lwzhao/research/TEAM.html
The SASL Project	http://www.cs.sun.ac.za/~lynette/signlang.html

Linguistics
Kearsy Cormier	http://ccwf.cc.utexas.edu/~kearsy/

Education
ICICLE Project	http://www.asel.udel.edu/nli/nlp/icicle.html

Software
Intel OpenCV	http://www.intel.com/research/mrl/research/opencv/
Entropic’s HTK	http://htk.eng.cam.ac.uk/index.shtml

Journals

Computer Vision and Image Understanding: CVIU

IJCV - International Journal of Computer Vision

IEEE Trans on Pattern Analysis and Machine Intelligence

International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI)

Conferences

IEEE International Conference on Automatic Face and Gesture Recognition

FG 2002 Washington, D.C. May 2002
FG'00, FG'98, FG'96

IEEE Conference on Computer Vision and Pattern Recognition

CVPR 2001 Hawaii December 2001
CVPR'00, CVPR'99, CVPR'9 8, CVPR'97, CVPR'96

IEEE International Conference on Computer Vision

ICCV 2001 Vancouver July 2001
ICCV'99

International Workshop on Gesture and Sign Language based Human-Computer Interaction

Workshop on Human Motion

HUMO 2000 http://www.computer.org/proceedings/humo/0939/0939toc.htm

International Conference on Spoken Language Processing

ICSLP 2002 - Denver

News lists

So-qiwoa, a computational sign linguistics discussion list:

Send an email to <listserv@list.unm.edu> saying

"subscribe so-qiwoa-l firstname lastname", example:

"subscribe so-qiwoa-l Laurent Clerc"

Bibliography

Surveys / Overviews

[Aggarwal and Cai(1999)]

J. K. Aggarwal and Q. Cai.

Human motion analysis: A review.

Computer and Vision Image Understanding, 73 (3): 428-440, March 1999.

http://www.idealibrary.com/servlet/doi/10.1006/cviu.1998.0744

Human motion analysis is receiving increasing attention from computer vision researchers. This interest is motivated by a wide spectrum of applications, such as athletic performance analysis, surveillance, man-machine interfaces, content-based image storage and retrieval, and video conferencing. This paper gives an overview of the various tasks involved in motion analysis of the human body. We focus on three major areas related to interpreting human motion: (1) motion analysis involving human body parts, (2) tracking a moving human from a single view or multiple camera perspectives, and (3) recognizing human activities from image sequences. Motion analysis of human body parts involves the low-level segmentation of the human body into segments connected by joints and recovers the 3D structure of the human body using its 2D projections over a sequence of images. Tracking human motion from a single view or multiple perspectives focuses on higher-level processing, in which moving humans are tracked without identifying their body parts. After successfully matching the moving human image from one frame to another in an image sequence, understanding the human movements or activities comes naturally, which leads to our discussion of recognizing human activities.

[Cedras and Shah(1995)]

C. Cedras and M. Shah.

Motion based recognition: A survey.

Image and Vision Computing, 13 (2): 129-155, 1995.

[Gavrila(1999)]

D. Gavrila.

The visual analysis of human movement: A survey.

Computer and Vision Image Understanding, 73 (1): 82-98, January 1999.

http://citeseer.nj.nec.com/400214.html

The ability to recognize humans and their activities by vision is key for a machine to interact intelligently and effortlessly with a human inhabited environment. Because of many potentially important applications, "Looking at People" is currently one of the most active application domains in computer vision. This survey identifies a number of promising applications and provides an overview of recent developments in this domain. The scope of this survey is limited to work on whole-body or hand motion; it does not include work on human faces. The emphasis is on discussing the various methodologies; they are grouped in 2-D approaches with or without explicit shape models and 3-D...

[Kohler and Schroeter(1998)]

M. Kohler and S. Schroeter.

A survey of video-based gesture recognition - stereo and mono systems.

Technical Report 693, Informatik VII, University of Dortmund, Germany, August 1998.

ftp://ls7-ftp.cs.uni-dortmund.de/pub/reports/ls7/1998/rr-693.ps.gz

[LaViola(1999)]

Joseph J. LaViola, Jr.

A survey of hand posture and gesture recognition techniques and technology.

Technical Report CS-99-11, Department of Computer Science, Brown University, June 1999.

ftp://ftp.cs.brown.edu/pub/techreports/99/cs99-11.ps.Z

[Moeslund and Granum(2001)]

T. B. Moeslund and E. Granum.

A survey of computer vision-based human motion capture.

International Journal of Computer Vision, 81 (3), 2001.

http://citeseer.nj.nec.com/moeslund01survey.html

[Pavlovic et al.(1997)]

V. I. Pavlovic, R. Sharma, and T. S. Huang.

Visual interpretation of hand gestures for human-computer interaction: A review.

IEEE Transactions on Pattern Analysis and Machine Intelligence, 19 (7): 677-695, July 1997.

http://computer.org/tpami/tp1997/i0677abs.htm

The use of hand gestures provides an attractive alternative to cumbersome interface devices for human-computer interaction (HCI). In particular, visual interpretation of hand gestures can help in achieving the ease and naturalness desired for HCI. This has motivated a very active research area concerned with computer vision-based analysis and interpretation of hand gestures. We survey the literature on visual interpretation of hand gestures in the context of its role in HCI. This discussion is organized on the basis of the method used for modeling, analyzing, and recognizing gestures. Important differences in the gesture interpretation approaches arise depending on whether a 3D model of the human hand or an image appearance model of the human hand is used. 3D hand models offer a way of more elaborate modeling of hand gestures but lead to computational hurdles that have not been overcome given the real-time requirements of HCI. Appearance-based models lead to computationally efficient "purposive" approaches that work well under constrained situations but seem to lack the generality desirable for HCI. We also discuss implemented gestural systems as well as other potential applications of vision-based gesture recognition. Although the current progress is encouraging, further theoretical as well as computational advances are needed before gestures can be widely used for HCI. We discuss directions of future research in gesture recognition, including its integration with other natural modes of human-computer interaction.

[Wu and Huang(1999)]

Y. Wu and T. S. Huang.

Vision-based gesture recognition: A review.

In A. Braffort, R. Gherbi, S. Gibet, J. Richardson, and D. Teil, editors, Proceedings of the Gesture Workshop, volume 1739 of Lecture Notes in Computer Science. Springer, 1999.

http://link.springer-ny.com/link/service/series/0558/papers/1739/17390103.pdf

Recognition / Translation

[Abdallah(1998)]

M. Abdallah.

A neuro-hierarchical multilayer network in the translation of the american sign language.

In Proceedings of the IEEE Southeastcon:Region 3, pages 224-227, 1998.

http://ieeexplore.ieee.org/iel4/5503/14789/00673334.pdf

A neuro hierarchial approach based on an adaptive back-propagation algorithm is proposed. In a separate preprocessing step, the input and the output vector are generated using the encoded-sequence representation. The algorithm is applied to translate the American sign language from spoken words. Experimental results indicate that this approach results in fast convergence, stable learning and a relatively small network size when compared to traditional methods.

[Abe et al.(1994)]

M. Abe, H. Sakou, and H. Sagawa.

Sign language translation based on syntactic and semantic analysis.

Systems and Computers in Japan, 25, 1994.

[Akyol et al.(2000)]

S. Akyol, B. Bauer, and U. Canzler.

Gesture and mimic interpretation for sign language recognition.

In Proceedings of the 4th International Student Conference on Electrical Engineering, May 2000.

http://www.techinfo.rwth-aachen.de/Veroeffentlichungen/v001_2000.pdf

[Aoki et al.(1998)]

Y. Aoki, R. Mitsumori, J. Li, and A. Burger.

Sign language communication between japanese-korean and japanese-portuguese using cg animation.

In IEEE International Conference on acoustics speech and signal processing, pages 3765-3768, 1998.

http://ieeexplore.ieee.org/iel4/5518/14898/00679703.pdf

We propose a sign language communication between different languages such as Japanese-Korean and Japanese-Portuguese using computer graphics (CG) animation of sign language based on the intelligent image communication method. For this purpose sign language animation is produced using gesture or text data expressing sign language. In the production process of CG animation of sign language, MATLAB and LIFO language are used, where MATLAB is useful for three-dimensional signal processing of gestures and for displaying animation of sign language. On the other hand the LIFO language, which is a descendant of the LISP and FORTH language families, is developed and used to produce live CG animations, resulting in a high-speed interactive system of designing and displaying sign language animations. A simple experiment was conducted to translate Japanese sign language into Korean and Portuguese sign languages using the developed CG animation system.

[Assan and Grobel(1997)]

M. Assan and K. Grobel.

Video-based sign language recognition using hidden markov models.

In Proceedings of the Gesture Workshop, volume 1371 of Lecture Notes in Artificial Intelligence, pages 97-110. Springer, 1997.

http://link.springer-ny.com/link/service/series/0558/bibs/1371/13710097.htm

This paper is concerned with the video-based recognition of signs. Concentrating on the manual parameters of sign language, the system aims for the signer dependent recognition of 262 different signs taken from Sign Language of the Netherlands . For Hidden Markov Modelling a sign is considered a doubly stochastic process, represented by an unobservable state sequence. The observations emitted by the states are regarded as feature vectors, that are extracted from video frames. This work deals with three topics: Firstly the recognition of isolated signs, secondly the influence of variations of the feature vector on the recognition rate and thirdly an approach for the recognition of connected signs. The system achieves recognition rates up to 94% for isolated signs and 73% for a reduced vocabulary of connected signs.

[Bauer(2001)]

B. Bauer.

Towards an automatic sign language recognition system using subunits.

In Proceedings of the Gesture Workshop, London, 2001.

http://www.techfak.uni-bielefeld.de/ags/wbski/gw2001book/draftpapers/bauer_gw01_02.pdf

Draft.

[Bauer and Hienz(2000)]

B. Bauer and H. Hienz.

Relevant features for video-based continuous sign language recognition.

In IEEE International Conference on Automatic Face and Gesture Recognition, Grenoble, France, 2000.

http://computer.org/proceedings/fg/0580/05800440abs.htm

[Bauer et al.(2000)]

B. Bauer, H. Hienz, and K.-F. Kraiss.

Video-based continuous sign language recognition using statistical methods.

In Proceedings of the International Conference on Pattern Recognition, volume II, pages 463-466, 2000.

http://www.computer.org/proceedings/icpr/0750/Volume%202/07502463abs.htm

[Bauer et al.(1999)]

B. Bauer, S. Niessen, and H. Hienz.

Towards an automatic sign language translation system.

In Proc. of the International Workshop on Physicality and Tangibility in Interaction: Towards New Paradigms for Interaction Beyond the Desktop, Siena, Italy, Oct. 1999.

http://www-i6.informatik.rwth-aachen.de/PostScript/InterneArbeiten/Niessen_SignTranslation_Physicality_Workshop_99.ps

[Birk et al.(1997)]

H. Birk, T. B. Moeslund, and C. B. Madsen.

Real-time recognition of hand alphabet gestures using principal component analysis.

In Scandinavian Conference on Image Analysis, 1997.

http://www.lut.fi/scia97/papers/paper131.htm

[Braffort(1996a)]

A. Braffort.

Argo: An architecture for sign language recognition and interpretation.

In P. Harling and A. Edwards, editors, Proceedings of the Gesture Workshop, pages 17-30. Springer, 1996.

[Braffort(1996b)]

A. Braffort.

A gesture recognition architecture for sign language.

In Second International ACM/SIGCAPH Conference on Assistive Technologies (ASSETS'96), pages 102-109, British Columbia, Canada, 1996.

http://www.acm.org/pubs/citations/proceedings/assets/228347/p102-braffort/

[Braffort(1997)]

A. Braffort.

A computer system dedicated to sign language.

In IEA'97, Tampere, Finland, 1997.

[Braffort et al.(1994)]

A. Braffort, C. Collet, and D. Teil.

Anthropomorphic model for hand gesture interface.

In M. Galer, S. Harker, and J. Ziegler, editors, Proceedings of ACM CHI'94 Conference on Human Factors in Computing Systems, 1994.

[Chapman(1998)]

R. N. Chapman.

Lexicon for computer translation of american sign language.

In Applications in Robotics, User Interfaces, and Natural Language Processing, volume 1458 of Lecture Notes in Computer Science. Springer, 1998.

http://link.springer-ny.com/link/service/series/0558/bibs/1458/14580033.htm

This work presents a method for translation of American Sign Language (ASL) to English using a feature-based lexicon, designed to exploit ASL's phonology by searching the lexicon for the sign's manual and non-manual information. Manual sign information consists of phonemes sig (movement), tab (location), dez (handshape), and ori (hand orientation), which we use as the ASL unit of analysis. Non-manual sign information consists of specific facial and body configurations. A camera acquires the sign and individual frames are analyzed and values assigned to the sig, tab, dez, and ori sign parameters as well as other sign features, for referencing during lexical search. ASL formational constraints are exploited to target specific image segments for analysis and linguistic constraints serve to further reduce the lexical search space. Primary keys for lexical search are sig and tab, the most discriminating sign features, followed by the remaining features, as necessary, until a single lexical entry is extracted from the lexicon. If a single lexical candidate cannot be determined, an exception is raised, signaling search failure. This method of using ASL phonological constraints to aid image analysis and lexical search process simplifies the task of sign identification.

[Charayaphan and Marble(1992)]

C. Charayaphan and A. Marble.

An image processing system for interpreting motion in american sign language.

Journal of Biomedical Engineering, 14 (15): 419-425, 1992.

[Cui et al.(1995)]

Y. Cui, D. L. Swets, and J. J. Weng.

Learning-based hand sign recognition using shoslif-m.

In Proceedings of the IEEE International Conference on Computer Vision, pages 631-636, Boston, MA, June 1995.

http://citeseer.nj.nec.com/cui95learningbased.html

[Cui and Weng(1999)]

Y. Cui and J. Weng.

A learning-based prediction-and-verification segmentation scheme for hand sign image sequence.

IEEE Transactions on Pattern Analysis and Machine Intelligence, 21 (8): 798-804, Aug. 1999.

http://dlib.computer.org/tp/books/tp1999/pdf/i0798.pdf

We present a prediction-and-verification segmentation scheme using attention images from multiple fixations. A major advantage of this scheme is that it can handle a large number of different deformable objects presented in complex backgrounds. The scheme is also relatively efficient. The system was tested to segment hands in sequences of intensity images, where each sequence represents a hand sign in American Sign Language. The experimental result showed a 95 percent correct segmentation rate with a 3 percent false rejection rate.

[Cui and Weng(2000)]

Y. Cui and J. Weng.

Appearance-based hand sign recognition from intensity image sequences.

Computer and Vision Image Understanding, 78: 157-176, 2000.

http://www.cse.msu.edu/~weng/research/SHOSLIF-M-rec.ps

[Cui and Weng(1995)]

Y. Cui and J. J. Weng.

Learning-based hand sign recognition.

In IEEE International Conference on Automatic Face and Gesture Recognition, pages 201-206, Zurich, Switzerland, June 1995.

[Cui and Weng(1996a)]

Y. Cui and J. J. Weng.

Hand segmentation using learning-based prediction and verification for hand sign recognition.

In IEEE Conference on Computer Vision and Pattern Recognition, pages 88-93, San Francisco, CA, June 1996.

http://www.cps.msu.edu/~weng/research/HandCVPR96.ps

[Cui and Weng(1996b)]

Y. Cui and J. J. Weng.

View-based hand segmentation and hand-sequence recognition with complex backgrounds.

In Proceedings of the International Conference on Pattern Recognition, volume III, pages 617-621, Vienna, Austria, Aug. 1996.

http://www.cps.msu.edu/~weng/research/SegICPR96.ps

[Dorner(1993)]

B. Dorner.

Hand shape identification and tracking for sign language interpretation.

In International Joint Conference on Artificial Intelligence, Chambery, 1993.

[Dorner(1994)]

B. Dorner.

Chasing the colour glove: Visual hand tracking.

Master's thesis, Simon Fraser University, 1994.

ftp://fas.sfu.ca/pub/thesis/1994/BrigitteDornerMSc.ps

[Dorner and Hagen(1994)]

B. Dorner and E. Hagen.

Towards an american sign language interface.

Artificial Intelligence Review, 8 (2/3), 1994.

http://www.wkap.nl/oasis.htm/61598

[Erenshteyn et al.(1995)]

R. Erenshteyn, R. Foulds, L. Messing, G. Stern, and S. Galuska.

Handshape recognition-a step toward computer translation of american sign language into english.

In WCNN '95. World Congress on Neural Networks, volume 1, pages 216-19, Mahwah, NJ, USA, 1995. Lawrence Erlbaum Associates.

[Erenshteyn and Laskov(1996)]

R. Erenshteyn and P. Laskov.

A multi-stage approach to fingerspelling and gesture recognition.

In Proceedings of the Workshop on the Integration of Gesture in Language and Speech, pages 185-194, Wilmington, DE, 1996.

http://citeseer.nj.nec.com/73836.html

Human gesture analysis and recognition as many others recognition applications require the development and implementation of multiclass recognition system. We examine the application of two approaches to multiclass recognition: a hierarchical (multi-stage) neural network classifier and error-correcting codes together with collective decision making (mixture of formal "experts"). Applications that have been solved include the recognition of fingerspelled letters from American Sign Language and recognition of static handshapes. These applications are examples of multiclass problem - 26 and 77 classes respectively. We address the problem of finding a hierarchy of classifiers, optimal with respect to recognition accuracy, given a labeled corpus of signing samples. This problem stems from the necessity of recognizing a large number of classes for which a single neural network classifier requires prohibitively large training sets and infeasible training time. We outline the general framework of the problem of finding an optimal hierarchy and explain its relationship to the problems of decision tree inference and object clustering. Another approach involves Golay(23,12) code generation and its use for system outputs encoding. These codes have to satisfy some initial condi tions, such as minimum inter-code Hamming distance, number of classes and inter-digit correlation. Code redundancy digits are used to improve overall recognition accuracy. Coding has been combined with two-output neural networks that produced output codes. Collective decision making procedures have been applied. Higher accuracy and less training time are the main advantage of the developed approach. Both approaches have been compared on two problems of human gesture recognition.

[Erenshteyn et al.(1997)]

R. Erenshteyn, P. Laskov, and R. Foulds.

Human gesture recognition using neural networks and multi-class encoding.

In Proceedings of ANNIE'97 (Artificial Neural Networks in Engineering), volume 7, pages 549-554, St.Louis, MO, 1997.

http://www.cis.udel.edu/~laskov/publications/ANNIE-97.ps.gz

[Erenshteyn et al.(1996a)]

R. Erenshteyn, P. Laskov, R. Foulds, L. Messing, and G. Stern.

Recognition approach to gesture language understanding.

In Proceedings of the International Conference on Pattern Recognition, Vienna, 1996.

http://www.cis.udel.edu/~laskov/publications/ICPR-96.ps.gz

We explore recognition implications of understanding gestural communication, having chosen American Sign Language as an example of a gestural language. An instrumented glove and specially developed software have been used for data collection and labeling. We address the problem of recognizing dynamic signing, i.e. signing performed at natural speed. Two neural network architectures have been used for recognition of different types of fingerspelled sentences. Experimental results are presented suggesting that two features of signing affect recognition accuracy: signing frequency which to a large extent can be accounted for by training a network on the samples of the respective frequency, and coarticulation effect which a network fails to identify. As a possible solution to coarticulation problem two post-processing algorithms for temporal segmentation are proposed and experimentally evaluated.

[Erenshteyn et al.(1996b)]

R. Erenshteyn, P. Laskov, R. Foulds, L. Messing, G. Stern, and S. Galuska.

Static and dynamic recognition of fingerspelled sentences.

In World Congress on Neural Networks, International Neural Network Society 1996 Annual Meeting, pages 368-71, Mahwah, NJ, USA, 1996. Lawrence Erlbaum Assoc.

http://www.cis.udel.edu/~laskov/publications/ANNIE-97.ps.gz

[Erenshteyn et al.(1994)]

R. Erenshteyn, L. Messing, R. A. Foulds, G. Stern, and S. Galuska.

Back propagation neural network for american sign language recognition.

In P. Werbos, H. Szu, and B. Widrow, editors, World Congress on Neural Networks, volume I, pages 405-409, San Diego, CA, USA, 1994. Erlbaum.

[Fang et al.(2001a)]

G. Fang, W. Gao, X. Chen, C. Wang, and J. Ma.

Signer-independent continuous sign language recognition based on srn/hmm.

In Proceedings of the Gesture Workshop, London, Apr. 2001.

http://www.techfak.uni-bielefeld.de/ags/wbski/gw2001book/draftpapers/SRNpaper_10.pdf

Draft.

[Fang et al.(2001b)]

G. Fang, W. Gao, and J. Ma.

Signer-independent sign language recognition based on sofm/hmm.

In IEEE ICCV Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems, pages 90-95, 2001.

http://ieeexplore.ieee.org/iel5/7480/20325/00938915.pdf

The aim of sign language recognition is to provide an efficient and accurate mechanism to transcribe sign language into text or speech. State-of-the-art sign language recognition should be able to solve the signer-independent problem for practical application. In this paper, a hybrid SOFM/HMM system, which combines self-organizing feature maps (SOFMs) with hidden Markov models (HMMs), is presented for signer-independent Chinese sign language recognition. We implement the SOFM/HMM sign recognition system. Meanwhile, results from the HMM-based system are provided as comparison. Experimental results show the SOFM/HMM system increases the recognition accuracy by 5% than the HMM-based one. Furthermore, a self-adjusting recognition algorithm is also proposed for improving the SOFM/HMM discrimination. When it is applied to the SOFM/HMM system it can improve the recognition accuracy by 1.9%. All experiments were performed in real-time with the dictionary size 208.

[Fujishige and Kurokawa(1997)]

E. Fujishige and T. Kurokawa.

Japanese processing for japanese-to-sign language translation via semantic network.

Human interface news and report, 12: 45-50, 1997.

[Gao et al.(2000a)]

W. Gao, J. Ma, S. Shan, X. Chen, W. Zheng, H. Zhang, J. Yan, and J. Wu.

Handtalker: A multimodal dialog system using sign language and 3-d virtual human.

In T. Tan, Y. Shi, and W. Gao, editors, Proceedings of the Third International Conference on Advances in Multimodal Interfaces, volume 1948 of Lecture Notes in Computer Science, pages 582-589, Oct. 2000.

http://link.springer-ny.com/link/service/series/0558/bibs/1948/19480564.htm

In this paper, we describe HandTalker: a system we designed for making friendly communication reality between deaf people and normal hearing society. The system consists of GTS (Gesture/Sign language To Spoken language) part and STG (Spoken language To Gesture/Sign language) part. GTS is based on the technology of sign language recognition, and STG is based on 3D virtual human synthesis. Integration of the sign language recognition and 3D virtual human techniques greatly improves the system performance. The computer interface for deaf people is data-glove, camera and computer display, and the interface for hearing-abled is microphone, keyboard, and display. HandTalker now can support no domain limited and continuously communication between deaf and hearing-abled Chinese people.

[Gao et al.(2000b)]

W. Gao, J. Ma, J. Wu, and C. Wang.

Sign language recognition based on hmm/ann/dp.

International Journal of Pattern Recognition and Artificial Intelligence, 14 (5), Aug. 2000.

http://www.wspc.com.sg/profiles/anncat/anncat/jnlarticle/ijpraiv14n5/S0218001400000386.pdf

In this paper, a system designed for helping the deaf to communicate with others is presented. Some useful new ideas are proposed in design and implementation. An algorithm based on geometrical analysis for the purpose of extracting invariant feature to signer position is presented. An ANN-DP combined approach is employed for segmenting subwords automatically from the data stream of sign signals. To tackle the epenthesis movement problem, a DP-based method has been used to obtain the context-dependent models. Some techniques for system implementation are also given, including fast matching, frame prediction and search algorithms. The implemented system is able to recognize continuous large vocabulary Chinese Sign Language. Experiments show that proposed techniques in this paper are efficient on either recognition speed or recognition performance.

[Gibet et al.(1996)]

S. Gibet, A. Braffort, C. Collet, F. Forest, R. Gherbi, and T. Lebourque.

Gesture in human-machine communication: Capture, analysis-synthesis, recognition, semantics.

In P. Harling and A. Edwards, editors, Proceedings of the Gesture Workshop, pages 89-95. Springer, 1996.

[Gibet et al.(1997)]

S. Gibet, J. Richardson, T. Lebourque, and A. Braffort.

Corpus of 3d natural movements and sign language primitives of movement.

In I. Wachsmuth and M. Fröhlich, editors, Proceedings of the Gesture Workshop, volume 1371 of Lecture Notes in Computer Science, pages 111-121, Bielefeld, Germany, 1997. Springer.

This paper describes the development of a corpus or database of hand-arm pointing gestures, considered as a basic element for gestural communication. The structure of the corpus is defined for natural pointing movements carried out in different directions, heights and amplitudes. It is then extended to movement primitives habitually used in sign language communication. The corpus is based on movements recorded using an optoelectronic recording system that allows the 3D description of movement trajectories in space. The main technical characteristics of the capture and pretreatment system are presented, and perspectives are highlighted for recognition and generation purposes.

[Grobel(1994)]

K. Grobel.

Recognition of fingerspelling from video.

In Proceedings of the Sixth Biennial Conference of the International Society for Augmentative and Alternative Communication, pages 282-286, 1994.

[Grobel and Assan(1997)]

K. Grobel and M. Assan.

Isolated sign language recognition using hidden markov models.

In IEEE International Conference on Systems, Man & Cybernetics, pages 162-167, Piscataway, NJ, 1997.

http://ieeexplore.ieee.org/iel3/4942/13619/00625742.pdf

This paper is concerned with the video-based recognition of isolated signs. Concentrating on the manual parameters of sign language, the system aims for the signer dependent recognition of 262 different signs. For hidden Markov modelling a sign is considered a doubly stochastic process, represented by an unobservable state sequence. The observations emitted by the states are regarded as feature vectors, that are extracted from video frames. The system achieves recognition rates up to 94%.

[Grobel and Hienz(1996a)]

K. Grobel and H. Hienz.

Fuzzy video-based handshape recognition.

In K. e. a. George, editor, Proceedings of the 1996 ACM Symposium on Applied Computing, pages 614-618, Philadelphia, Feb. 1996.

[Grobel and Hienz(1996b)]

K. Grobel and H. Hienz.

Video-based handshape recognition using a handshape structure model in real-time.

In Proceedings of the International Conference on Pattern Recognition, pages 446-450, Wien, 1996.

[Grobel and Hienz(1996c)]

K. Grobel and H. Hienz.

Video-based recognition of fingerspelling in real-time.

In T. Lehmann, I. Scholl, and K. Spitzer, editors, Proceedings des Aachener Workshops am Bildverarbeitung für die Medizin. Algorithmen. Systeme. Anwendungen, pages 197-202, 1996.

http://citeseer.nj.nec.com/118789.html

[Hagen(1994)]

E. Hagen.

A flexible american sign language interface to deductive databases.

Master's thesis, Computer Science, Simon Fraser University, 1994.

ftp://fas.sfu.cs/pub/theses/1994/EliHagenMSc.ps

[Hamilton and Micheli-Tzanakou(1994)]

J. Hamilton and E. Micheli-Tzanakou.

Alopex neural networks for manual alphabet recognition.

In Proceedings of the 16th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 1994.

http://ieeexplore.ieee.org/iel4/3230/9234/00415347.pdf

Alopex and backpropagation were used to train neural networks to recognize signs from the American Manual Alphabet. In many cases, the resulting networks gave comparable performance. The Alopex optimization technique did not converge to low error percentages during training as well as backpropagation did; backpropagation gave poorer performance on networks with a small number of hidden nodes.

[Harrison(1982)]

R. Harrison.

Computer representation of deaf sign language.

In D. S. Greenaway and E. Warman, editors, Eurographics '82 : proceedings of the international conference and exhibition, pages 303-308, Manchester, UK, 1982. North-Holland Publ. Co.

[Hienz and Bauer(1999)]

H. Hienz and B. Bauer.

Sign language recognition based on statistical methods.

In Bi-Annual Report - Department of Technical Computer Science 1997/1998, pages 34-37. Shaker, 1999.

[Hienz and Grobel(1998)]

H. Hienz and K. Grobel.

An automatic video-based sign language recognition system as part of a sign printing system.

In P. Kopacek, editor, IEEE International Conference on Intelligent Engineering Systems, pages 163-168, 1998.

[Hienz et al.(1996)]

H. Hienz, K. Grobel, and G. Beckers.

Video-based handshape recognition using artificial neural networks.

In H.-J. Zimmermann, editor, Fourth European Congress on Intelligent Techniques and Soft Computing, pages 1659-1663, Aachen, Germany, Sept. 1996. Wissenschaftsverlag.

[Hienz and Kraiss(1999)]

H. Hienz and K.-F. Kraiss.

Hmm-based continuous sign language recognition using stochastic grammars.

In Proceedings of the Gesture Workshop, volume 1739 of Lecture Notes in Artificial Intelligence, Gif-sur-Yvette, France, Mar. 1999. Springer.

http://www.techinfo.rwth-aachen.de/Veroeffentlichungen/V001_1999.pdf

[Hienz et al.(1999)]

H. Hienz, K.-F. Kraiss, and B. Bauer.

Continuous sign language recognition using hidden markov models.

In Y. Tang, editor, The Second International Conference on Multimodal Interface, pages IV10--IV15, Hong Kong, China, 1999.

http://www.techinfo.rwth-aachen.de/Veroeffentlichungen/V002_1999.pdf

[Holden and Roy(1992)]

E. Holden and G. Roy.

The graphical translation of english text into signed english in the hand sign translator system.

Computer Graphics Forum (Eurographics'92), 11 (3): C357--C366, 1992.

http://www.cs.uwa.edu.au/~eunjung/mypubs/GForumpaper.ps.gz

Signed English is a manual interpretation of English using fingerspelling and signs. A prototype of the Hand Sign Translator (HST) system was developed to graphically translate English into Signed English, using two-handed animation. The HST consists of a practical interface that aims to help users learn Signed English, and the translation process where English text is transformed into a series of images that represent corresponding signs. This paper describes the translation process which involves two stages; the input environment and the animation process. The input environment consists of text analysis in order to extract corresponding kinematic data from the database, named English-Sign ...

[Holden(1991)]

E. J. Holden.

Graphical representation of hand movement as in deaf sign language: The hand sign translator system.

Master's thesis, University of Western Australia, 1991.

[Holden(1997)]

E.-J. Holden.

Visual Recognition of Hand Motion.

PhD thesis, University of Western Australia, 1997.

http://www.cs.uwa.edu.au/~eunjung/mypubs/PhDThesis.ps.gz

[Holden and Owens(2001)]

E. J. Holden and R. Owens.

Visual sign language recognition.

In R. Klette, T. Huang, and G. Gimen'garb, editors, 10th International Workshop on Theoretical Foundations of Computer Vision, volume 2032 of Lecture Notes in Computer Science. Springer, Mar. 2001.

http://link.springer.de/link/service/series/0558/bibs/2032/20320270.htm

Also in Proceedings of DICTA'99 (Digital Image Computing: Techniques & Applications), pp. 275-279.

Automatic gesture recognition systems generally require two separate processes: a motion sensing process where some motion features are extracted from the visual input; and a classification process where the features are recognised as gestures. We have developed the Hand Motion Understanding (HMU) system that uses the combination of a 3D model-based hand tracker for motion sensing and an adaptive fuzzy expert system for motion classification. The HMU system understands static and dynamic hand signs of the Australian Sign Language (Auslan). This paper presents the hand tracker that extracts 3D hand configuration data with 21 degrees-of-freedom (DOFs) from a 2D image sequence that is captured from a single viewpoint, with the aid of a colour-coded glove. Then the temporal sequence of 3D hand configurations detected by the tracker is recognised as a sign by an adaptive fuzzy expert system. The HMU system was evaluated with 22 static and dynamic signs. Before training the HMU system achieved 91% recognition, and after training it achieved over 95% recognition.

[Holden et al.(1999a)]

E.-J. Holden, R. Owens, and G. G. Roy.

3d hand tracker for visual sign recognition.

Technical Report 99/2, Department of Computer Science, The University of Western Australia, 1999.

http://citeseer.nj.nec.com/263562.html

[Holden et al.(1999b)]

E.-J. Holden, R. Owens, and G. G. Roy.

Adaptive fuzzy expert system for sign recognition.

Technical Report 99/3, Department of Computer Science, The University of Western Australia, 1999.

http://citeseer.nj.nec.com/holden99adaptive.html

[Holden et al.(1996)]

E. J. Holden, G. G. Roy, and R. Owens.

Hand movement classification using an adaptive fuzzy expert system.

International Journal of Expert Systems, 9 (4): 465-480, 1996.

http://www.cs.uwa.edu.au/~eunjung/mypubs/IJESpaper.ps.gz

[Huang and Huang(1998)]

C. Huang and W. Huang.

Sign language recognition using model-based tracking and a 3d hopfield neural-network.

Machine Vision and Applications, 10 (5/6): 292-307, 1998.

http://link.springer.de/link/service/journals/00138/bibs/8010005/80100292.htm

[Huang et al.(1992)]

C.-L. Huang, C. L. Cheng, and L. L. Pau.

Chinese sign language interpretation through motion and shape analysis.

In V. Srinivasa, O. S. Heng, and A. Y. Hock, editors, Proceedings of the 2nd Singapore International Conference on Image Processing, pages 576-580, 1992.

[Huang et al.(1995)]

C.-L. Huang, W.-Y. Huang, and C.-C. Lien.

Sign language recognition using 3-d hopfield neural network.

In Proceedings International Conference on Image Processing, pages 611-614, 1995.

http://ieeexplore.ieee.org/iel3/4052/11607/00537553.pdf

This paper presents a sign language recognition system which consists of three modules: model-based hand tracking, feature extraction, and gesture recognition using a 3-D Hopfield neural network. In the experiments, we illustrate that this system can recognize 15 different gestures accurately.

[Imagawa et al.(1997)]

K. Imagawa, S. Lu, and S. Igi.

Tracking hands by skin color from sign language image sequences.

In Proc. of 54th IPSJ Annual Conference, pages 249-250, 1997.

[Imagawa et al.(1998a)]

K. Imagawa, S. Lu, and S. Igi.

Color-based hands tracking system for sign language recognition.

In IEEE International Conference on Automatic Face and Gesture Recognition, pages 462-467, 1998.

http://dlib.computer.org/conferen/fg/8344/pdf/83440462.pdf

[Imagawa et al.(1998b)]

K. Imagawa, S. Lu, and S. Igi.

Real-time tracking of human hands from a sign-language image sequence.

In The Third Asian Conference on Computer Vision '98, 1998., 1998.

[Imagawa et al.(2000)]

K. Imagawa, H. Matsuo, R. Taniguchi, D. Arita, S. Lu, and S. Igi.

Recognition of local features for camera-based sign language recognition system.

In Proceedings of the International Conference on Pattern Recognition, pages 849-853, 2000.

http://www.computer.org/proceedings/icpr/0750/Volume 4/07504849abs.htm

[Kadous(1995)]

M. W. Kadous.

Grasp - recognition of australian sign language using instrumented gloves.

Bachelor's thesis, University of New South Wales, Oct. 1995.

http://www.cse.unsw.edu.au/~waleed/thesis/

[Kadous(1996)]

M. W. Kadous.

Machine recognition of auslan signs using powergloves: Towards large-lexicon recognition of sign language.

In L. Messing, editor, Proceedings of WIGLS. The Workshop on the Integration of Gesture in Language and Speech, pages 165-174, Oct. 1996.

http://www.cse.unsw.edu.au/~waleed/paper-wigls/paper-wigls.html

[Kadous(1998a)]

M. W. Kadous.

Auslan sign recognition using computers and gloves.

In Deaf Studies Research Symposium, 1998.

http://www.cse.unsw.edu.au/~waleed/phd/auslrec2.ps

[Kadous(1998b)]

M. W. Kadous.

A general architecture for supervised classification of multivariate time series.

Technical Report UNSW-CSE-TR-9806, Department of Artificial Intelligence, School of Computer Science & Engineering, University of New South Wales, Sept. 1998.

ftp://ftp.cse.unsw.edu.au/pub/users/waleed/tr9806.ps.gz

[Kadous(1999)]

M. W. Kadous.

Learning comprehensible descriptions of multivariate time series.

In Proceedings of the International Conference on Machine Learning, pages 454-463. Morgan Kaufmann, San Francisco, CA, 1999.

http://citeseer.nj.nec.com/kadous99learning.html

[Kadous(2002a)]

M. W. Kadous.

Expanding the scope of concept learning using metafeatures. Submitted paper.

In Proceedings of the International Conference on Machine Learning, 2002.

http://www.cse.unsw.edu.au/~waleed/phd/paper.pdf

We present a general automated preprocessing technique called metafeatures. Using metafeatures, the scope of traditional propositional attribute-value learning is expanded to domains that do not normally fit in the propositional model. These are domains that contain instances that have some kind of recurring substructure, such as strokes in handwriting recognition, or local maxima in time series data. Metafeatures are applied to three domains: sign language recognition, ECG classification and Chinese handwriting recognition. Using metafeatures we are able to generate classifiers that are both comprehensible and accurate, producing results that are comparable to hand-crafted feature extraction and in one case comparable to human experts.

[Kadous(2002b)]

M. W. Kadous.

Temporal Classification: Extending the Classifiation Paradigm to Multivariate Time Series.

PhD thesis, The University of New South Wales, School of Computer Science and Engineering, Jan. 2002.

http://www.cse.unsw.edu.au/~waleed/phd/phd.pdf

[Kamata et al.(1989)]

K. Kamata, T. Yoshida, M. Watanabe, and Y. Usui.

An approach to japanese-sign language translation system.

In IEEE International Conference on Systems, Man and Cybernetics, volume 3, pages 1089-1090, 1989.

http://ieeexplore.ieee.org/iel2/861/2458/00071466.pdf

A method was developed for translating written (spoken) language (text in Japanese) into a sequence of sign words of Japanese sign language called Simultaneous Japanese Sign Language (SJSL). The Japanese-to-Sign translation system is described that uses a Japanese-to-Sign translation dictionary with about 250 signs.

[Kang and Park()]

S. H. Kang and S. H. Park.

Toward korean text-to-sign language translation system (test).

http://citeseer.nj.nec.com/34981.html

[Kim et al.(1999a)]

J. Kim, A. Hasimoto, Y. Aoki, and A. Burger.

Design of a sign-language translation system between the japanese-korean by java-lifo language.

In Proceedings of the IEEE Region 10 Conference TENCON 99, volume 1, pages 423-426, 1999.

[Kim et al.(1996)]

J.-S. Kim, W. Jang, and Z. Bien.

A dynamic gesture recognition system for the korean sign language (ksl).

IEEE Transactions on Systems, Man and Cybernetics, Part B, 26 (2): 354-359, Apr. 1996.

http://ieeexplore.ieee.org/iel1/3477/10374/00485888.pdf

The sign language is a method of communication for the deaf-mute. Articulated gestures and postures of hands and fingers are commonly used for the sign language. This paper presents a system which recognizes the Korean sign language (KSL) and translates into a normal Korean text. A pair of data-gloves are used as the sensing device for detecting motions of hands and fingers. For efficient recognition of gestures and postures, a technique of efficient classification of motions is proposed and a fuzzy min-max neural network is adopted for on-line pattern recognition.

[Kim et al.(1999b)]

S.-W. Kim, J.-W. Lee, J.-Y. Oh, and Y. Aoki.

Development of a sign-language communication system between korea and japan through a 3d character model on internet.

In International Conference on Image Processing, 1999.

[Klima and Bellugi(1979)]

E. Klima and U. Bellugi.

The Signs of Language.

Harvard University Press, 1979.

[Kurokawa et al.(1993)]

T. Kurokawa, T. Morichi, and S. Watanabe.

Bidirectional translation between sign language and japanese for communication with deaf-mute people.

In G. Salvendy and M. J. Smith, editors, Human-Computer Interaction: A: Applications & Case Studies; B: Software & Hardware Interfaces., pages 1109-1114. Elsevier, 1993.

[Lamar(2001)]

M. Lamar.

Hand Gesture Recognition using T-CombNET - A Neural Network dedicated to Temporal Information Processing.

PhD thesis, Nagoya Institute of Technology, Japan, Mar. 2001.

http://www.eletrica.ufpr.br/lamar/public/mvlamar.pdf

[Lamar et al.(1998)]