Pen and Speech Recognition in the User Interface for Mobile Multimedia Terminals

Shankar Narayanaswamy,  Ph.D. 1996 (advisor: Bob Brodersen).

Portable computers have soared in popularity over the last few years. Vendors are introducing new models with smaller form factors, longer battery life, communications capabilities and unique user interfaces using pen or audio input. The design and implementation of a networked user interface architecture using hand-writing, recognition and speech recognition is explored. Although the user interface, was designed for mobile multimedia terminals such as the InfoPad system, it is more, generally applicable in any application domain where pen and/or spoken input are preferable to keyboard input. We examine the kinds of handwriting and speech recognizers needed to provide an effective user interface. There are several aspects to this problem. Firstly, there is the user interaction model which determines where and how the user uses each input modality. Secondly, there is the applications programming model which determines the level of abstraction and the extent of encapsulation of the recognizer’s functionality. Thirdly, there is the service provision model which determines whether the recognizer is part of the application or whether it runs as a separate thread or process, or somewhere in between. The latter allows off-loading the recognition computation onto a remote, possibly specialized server and minimizes the impact of compute-intensive recognizers on other applications. The entire infrastructure for a pen and speech based user interface is described, including a software hidden Markov model based writer-independent hand-print recognizer, a VLSI hidden Markov model based large-vocabulary speaker independent continuous speech recognizer, type servers to handle the new data types and applications that exercise the entire user interface architecture.