From an Information Theory/ Informatics PoV is a Modality the use of a specific channel within our actions in Order to provide information to a system

Machine learning is a field of study in artificial intelligence that focuses on developing algorithms and models that enable computers to learn and make predictions or decisions based on data without being explicitly programmed.

Approaches for machine learning

Supervised Learning: learns from (high amount of) labeled data

Semi-supervised Learning: learns from labeled and unlabeled data

Unsupervised Learning: learns patterns and structures from unlabeled data

Self-supervised Learning: learns from unlabeled data by predicting hidden parts

Reinforcement Learning: learns from interacting with an environment by maximizing a reward

Overview of multimodal interactive systems. Classification of interaction modalities (Bernsen, 1999)

• Linguistic vs. non-linguistic

• Analogue vs. non-analogue

• Arbitrary vs. non-arbitrary

• Graphical (visually perceivable), acoustic (auditorily perceivable), haptic

• Static vs. dynamic (only output modalities)

• Speech: Language, dialect, speaking style, …

• Speaker:

  • dependent vs. independent vs. adaptive

  • known vs. unknown

  • cooperative vs. uncooperative

• Target units: Type, number, complexity

• Environment: Background noise, transmission channels, etc.

→ Automatic speech recognition is a very difficult task!

Task complexity:

• Variances and invariances in the speech signal have to be differentiated

  „to recognize speech“ vs. „to wreck a nice beach“

  → Contextual knowledge helps to understand

• Speech is a continuous signal, not a sequence of elementary sounds (even across word boundaries)

• Articulation depends on the surrounding sounds

  (co-articulation)

The feature extraction block finds the information in the speech signal that is relevant for the detection of what words or sentences were actually spoken. Most importantly, the resulting features should not include information that holds no value for the detection such as information about the speaker and the environment. The is done for speech blocks of about 𝟐𝟎 ms.

In the next step, the phoneme probabilities must be estimated from these feature vectors (classification). This is done using both the acoustic model and the vocabulary. There exist different approaches for this, such as the Hidden Markov Model (HMM) or the neural network.

From the probabilities, the decoder combines consecutive phonemes to a full word and finally to a full sentence. To do this, the information about the grammar (language model) is used in addition to the vocabulary

(PSOLA = Concatenative)

a) Parametric synthesis

b) HMM-based synthesis

c) Random shift synthesis

d) Unit-selection synthesis

0.782608695652174