Some general reasons why the speech industry has not yet widely considered the use of prosodic features in commercial applications might be the following:
- the research community does not yet provide a consistent prosodic framework neither for a single language, nor for the multi-lingual description of prosody
- prosody is very much speaker-dependent, even ideosyncratic (F0 range, speech rate, etc.), so the issue of normalization is crucial with problematic implications for speaker-independent recognition, for instance
- we know relatively little about the way in which different types of information (linguistic, para-linguistic, non-linguistic) are coded into bundles of prosodic features - not just a single feature such as F0
On Text-To-Speech (TTS):
Listening to some of nowadays’ best commercial TTS systems we might be led to think that synthesis is a solved problem. Current state-of-the-art systems sound very natural, but still not as if the computer knew what it was talking about. We have to admit though that for current applications of TTS there exist technologically feasible solutions. It seems embarrassing for the prosody community that most parametric approaches based on loads of prosodic knowledge sound very poor compared with corpus based synthesis.
Based on the observation that synthesis from text as a highly impoverished representation will exhibit a stereotypical prosody, there have been proposals for mark-up languages for enhancing the symbolic prosodic input to a synthesizer. This kind of mark-up can especially be justified when the synthesizer is running in a limited domain, concept-to-speech mode, where we find highly recurrent prosodic patterns (weather reports, stock reports, etc.).
On speech recognition:
On the other hand we have to keep in mind that prosody is not everything in ASR. In the presence of other information which facilitates top-down processing (syntactic structure, morphemic markers) we might not even need it. Yes/no question in Japanese or Finnish, for instance, are marked by question particles [7]. Therefore there is no need for a rising F0 at the end of utterances of yes/no questions. Besides, prosody is not an issue in nowadays applications of ASR because
- we hope to deal with a cooperative user
- currently dialog models are mostly machine-guided
- most recognizer are expected to work speaker-independently. Although we have robust acoustic models, we lack this kind of model on the prosody side and we do not want recognizers to be confused by erroneous prosody recognition.
What can we gain by exploiting prosodic cues ? A few benefits may be the following:
- detection of minor (non-pause) boundaries
- dialog act classification (declarative, interrogative, unfinished, ...) where morphemic markers are absent
- detection of focus structure (important and less important words)
- emotion detection.
The advantages are obvious: A prosody-aware recognizer can facilitate a more flexible dialog with topic shifts induced by the user. It can also detect which items in the discourse are most important to the user.
In the decade that has passed since the introduction of the ToBI system for the transcription of prosody, speech technology has moved out of the laboratory and into commercial applications on several fronts. However, virtually none of the commercial products have made large-scale use of prosody. Nevertheless, researchers in both recognition and synthesis continue to agree that better utilization of prosody is essential to improving the performance and acceptability of commercial systems.
<speak>I already told you I <emphasis level="strong">really like</emphasis> that person.</speak> Check out Ha Eun's Interspeech paper on TTS!