Master thesis

"Detection of accents and their use in recognition of dialog acts"

My master thesis is written in German. Download the german, gzipped postscript version here (1.6 MB)

It covers the following topics:

The following lines give you a first impression of our (Dr. Elmar Nöth, Dr. Anton Batliner et al.) work. You can find more detailed informations in this english, gzipped postscript Verbmobil report (0.2 MB).

Speech production and prosody 

This chapter introduces the anatomy and mechanisms of speech production. It further defines the term "prosody". Prosody is concerned with suprasegmental events in spoken language. These events overlay spoken units, which cover more than only a single isolated sound (phoneme). Prosody means by that e.g. the speech melody or articulation of a syllable, a word, a phrase or a sentence.

We restrict ourselves on detection of prosodic accents. A prosodic accent in our definition is nothing else than word stress.  A word can be stressed in different ways. Word stress depends on changing the pitch, the loudness and the duration or their combination.


Extraction of prosodic features 

We are interested in the detection of prosodic accents in spontaneously spoken german language. Therefore we have to define and to compute prosodic features. I my thesis I use the 276 prosodic features developed by Dr. Andreas Kießling and described in his Ph.D. They rely on an automatic time alignment and cover the duration, the pitch, the energy (or loudness) of  words and/or syllables.

Verbmobil project 

My thesis is embedded into the Verbmobil project. It aims to construct an automatic german-english translator. For more information please refer to the Verbmobil homepage.

Automatically labeling of prosodic accents 

In the Verbmobil project exists a large recorded collection of spontaneous spoken speech data. After recording, the dialogs have been transliterated and are now available as ASCII text together with the original recording. A small subset of the database has been marked with prosodic information (e.g. pause or word stress information) by listener judgment.

Dr. Anton Batliner and I developed a rule-based system to automatically label word stress. What is special about our system is that it needs only the ASCII word chain (the transliteration), information about pauses and additionally a lexicon with annotated parts of speech.

Information about pauses can automatically be robust computed as Dr. Andreas Kießling and Dr. Ralf Kompe in their Ph.Ds. show. Another way is the one which has been gone by Dr. Anton Batliner, who annotated pause information, by only inspecting the transliteration.

Upon this base our rule based labeling system for spontaneous german speech in the context of the Verbmobil project was constructed. The experiments show that there is a close relationship between the labeling of word stress by listener judgment and by  our rule based approach.

Our approach has the advantage to label a large database in a very short time. Its drawback is, that the label are bit less exact than the listener judgment labels. Compared to the sometimes wrong listener judgment labels we can label 76%  correctly.


 Recognition of prosodic accents using neural networks 

After (re-)labelling the databases we extract the above mentioned prosodic features and train artificial neural networks (ANN). We restrict ourselves to a special kind of ANN, the multi layer perceptron (MLP). As software tool we use the freely available ANN simulator SNNS. Our experiments show, that we can achieve a recognition rate of 80% for listener judgment labels and 78% for labels marked by our rule based approach. A combination of the two MLP slightly improves the recognition rates.

Recognition of prosodic accent using MLP and n-grams 

Due to the automatic labeling of accent information (word stress) we are able to label the up to now unlabeled huge database. This enables us to train 3-grams for accent recognition. We achieve recognition rate about 87%. Further we can slightly improve these rate to 88% by a combination of MLP and 3-gram, the MLP/3-gram-hybrid.

The use of accents to distinguish dialogacts 

In natural language processing sentences can not be viewed as basis units, because often speakers interrupt or correct themselves, without starting a new sentence. The main criterion for understanding is not the grammar, but the intention of a phrase. The Verbmobil project  introduces the dialogact to determine the meaning of an utterance. E.g. "Good morning" , "Hello"  and "Hi, Stephanie" belong to the dialogact "Greet".

Our job was to determine the influence of prosodic accent information (word stress) in classification of  dialogacts. Our experiments on a subset of the Verbmobil dialogacts show, that considering prosodic information improves the recognition rate of dialogacts. These results show the importance of prosodic information in the field of automatic speech recognition.

  Matthias Nutt, August 1998