speech to text using ml

Automatic Speech Recognition with Transformer

Author: Apoorv Nandan Date created: 2021/01/13 Last modified: 2021/01/13 Description: Training a sequence-to-sequence Transformer for automatic speech recognition.

Introduction

Automatic speech recognition (ASR) consists of transcribing audio speech segments into text. ASR can be treated as a sequence-to-sequence problem, where the audio can be represented as a sequence of feature vectors and the text as a sequence of characters, words, or subword tokens.

For this demonstration, we will use the LJSpeech dataset from the LibriVox project. It consists of short audio clips of a single speaker reading passages from 7 non-fiction books. Our model will be similar to the original Transformer (both encoder and decoder) as proposed in the paper, "Attention is All You Need".

References:

• Attention is All You Need
• Very Deep Self-Attention Networks for End-to-End Speech Recognition
• Speech Transformers
• LJSpeech Dataset

Define the Transformer Input Layer

When processing past target tokens for the decoder, we compute the sum of position embeddings and token embeddings.

When processing audio features, we apply convolutional layers to downsample them (via convolution strides) and process local relationships.

Transformer Encoder Layer

Transformer decoder layer, complete the transformer model.

Our model takes audio spectrograms as inputs and predicts a sequence of characters. During training, we give the decoder the target character sequence shifted to the left as input. During inference, the decoder uses its own past predictions to predict the next token.

Download the dataset

Note: This requires ~3.6 GB of disk space and takes ~5 minutes for the extraction of files.

Preprocess the dataset

Callbacks to display predictions, learning rate schedule, create & train the end-to-end model.

In practice, you should train for around 100 epochs or more.

Some of the predicted text at or around epoch 35 may look as follows:

Transformers documentation

Speech2Text

Transformers.

and get access to the augmented documentation experience

to get started

The Speech2Text model was proposed in fairseq S2T: Fast Speech-to-Text Modeling with fairseq by Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, Juan Pino. It’s a transformer-based seq2seq (encoder-decoder) model designed for end-to-end Automatic Speech Recognition (ASR) and Speech Translation (ST). It uses a convolutional downsampler to reduce the length of speech inputs by 3/4th before they are fed into the encoder. The model is trained with standard autoregressive cross-entropy loss and generates the transcripts/translations autoregressively. Speech2Text has been fine-tuned on several datasets for ASR and ST: LibriSpeech , CoVoST 2 , MuST-C .

This model was contributed by valhalla . The original code can be found here .

Speech2Text is a speech model that accepts a float tensor of log-mel filter-bank features extracted from the speech signal. It’s a transformer-based seq2seq model, so the transcripts/translations are generated autoregressively. The generate() method can be used for inference.

The Speech2TextFeatureExtractor class is responsible for extracting the log-mel filter-bank features. The Speech2TextProcessor wraps Speech2TextFeatureExtractor and Speech2TextTokenizer into a single instance to both extract the input features and decode the predicted token ids.

The feature extractor depends on torchaudio and the tokenizer depends on sentencepiece so be sure to install those packages before running the examples. You could either install those as extra speech dependencies with pip install transformers"[speech, sentencepiece]" or install the packages separately with pip install torchaudio sentencepiece . Also torchaudio requires the development version of the libsndfile package which can be installed via a system package manager. On Ubuntu it can be installed as follows: apt install libsndfile1-dev

• ASR and Speech Translation

Multilingual speech translation

For multilingual speech translation models, eos_token_id is used as the decoder_start_token_id and the target language id is forced as the first generated token. To force the target language id as the first generated token, pass the forced_bos_token_id parameter to the generate() method. The following example shows how to transate English speech to French text using the facebook/s2t-medium-mustc-multilingual-st checkpoint.

See the model hub to look for Speech2Text checkpoints.

Speech2TextConfig

Class transformers. speech2textconfig.

( vocab_size = 10000 encoder_layers = 12 encoder_ffn_dim = 2048 encoder_attention_heads = 4 decoder_layers = 6 decoder_ffn_dim = 2048 decoder_attention_heads = 4 encoder_layerdrop = 0.0 decoder_layerdrop = 0.0 use_cache = True is_encoder_decoder = True activation_function = 'relu' d_model = 256 dropout = 0.1 attention_dropout = 0.0 activation_dropout = 0.0 init_std = 0.02 decoder_start_token_id = 2 scale_embedding = True pad_token_id = 1 bos_token_id = 0 eos_token_id = 2 max_source_positions = 6000 max_target_positions = 1024 num_conv_layers = 2 conv_kernel_sizes = (5, 5) conv_channels = 1024 input_feat_per_channel = 80 input_channels = 1 **kwargs )

• vocab_size ( int , optional , defaults to 10000) — Vocabulary size of the Speech2Text model. Defines the number of different tokens that can be represented by the inputs_ids passed when calling Speech2TextModel
• encoder_layers ( int , optional , defaults to 12) — Number of encoder layers.
• encoder_ffn_dim ( int , optional , defaults to 2048) — Dimensionality of the “intermediate” (often named feed-forward) layer in encoder.
• encoder_attention_heads ( int , optional , defaults to 4) — Number of attention heads for each attention layer in the Transformer encoder.
• decoder_layers ( int , optional , defaults to 6) — Number of decoder layers.
• decoder_ffn_dim ( int , optional , defaults to 2048) — Dimensionality of the “intermediate” (often named feed-forward) layer in decoder.
• decoder_attention_heads ( int , optional , defaults to 4) — Number of attention heads for each attention layer in the Transformer decoder.
• encoder_layerdrop ( float , optional , defaults to 0.0) — The LayerDrop probability for the encoder. See the LayerDrop paper for more details.
• decoder_layerdrop ( float , optional , defaults to 0.0) — The LayerDrop probability for the decoder. See the LayerDrop paper for more details.
• use_cache ( bool , optional , defaults to True ) — Whether the model should return the last key/values attentions (not used by all models).
• is_encoder_decoder ( bool , optional , defaults to True ) — Whether the model is set up as an encoder-decoder architecture for sequence-to-sequence tasks.
• activation_function ( str or function , optional , defaults to "relu" ) — The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu" , "relu" , "silu" and "gelu_new" are supported.
• d_model ( int , optional , defaults to 256) — Dimensionality of the layers and the pooler layer.
• dropout ( float , optional , defaults to 0.1) — The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
• attention_dropout ( float , optional , defaults to 0.0) — The dropout ratio for the attention probabilities.
• activation_dropout ( float , optional , defaults to 0.0) — The dropout ratio for activations inside the fully connected layer.
• init_std ( float , optional , defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
• decoder_start_token_id ( int , optional , defaults to 2) — The initial token ID of the decoder when decoding sequences.
• scale_embedding ( bool , optional , defaults to True ) — Whether the embeddings are scaled by the square root of d_model .
• pad_token_id ( int , optional , defaults to 1) — Padding token id.
• bos_token_id ( int , optional , defaults to 0) — The id of the beginning-of-sequence token.
• eos_token_id ( int , optional , defaults to 2) — The id of the end-of-sequence token.
• max_source_positions ( int , optional , defaults to 6000) — The maximum sequence length of log-mel filter-bank features that this model might ever be used with.
• max_target_positions ( int , optional , defaults to 1024) — The maximum sequence length that this model might ever be used with. Typically, set this to something large just in case (e.g., 512 or 1024 or 2048).
• num_conv_layers ( int , optional , defaults to 2) — Number of 1D convolutional layers in the conv module.
• conv_kernel_sizes ( Tuple[int] , optional , defaults to (5, 5) ) — A tuple of integers defining the kernel size of each 1D convolutional layer in the conv module. The length of conv_kernel_sizes has to match num_conv_layers .
• conv_channels ( int , optional , defaults to 1024) — An integer defining the number of output channels of each convolution layers except the final one in the conv module.
• input_feat_per_channel ( int , optional , defaults to 80) — An integer specifying the size of feature vector. This is also the dimensions of log-mel filter-bank features.
• input_channels ( int , optional , defaults to 1) — An integer specifying number of input channels of the input feature vector.

This is the configuration class to store the configuration of a Speech2TextModel . It is used to instantiate a Speech2Text model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Speech2Text facebook/s2t-small-librispeech-asr architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Speech2TextTokenizer

Class transformers. speech2texttokenizer.

( vocab_file spm_file bos_token = '<s>' eos_token = '</s>' pad_token = '<pad>' unk_token = '<unk>' do_upper_case = False do_lower_case = False tgt_lang = None lang_codes = None additional_special_tokens = None sp_model_kwargs : Optional = None **kwargs )

• vocab_file ( str ) — File containing the vocabulary.
• spm_file ( str ) — Path to the SentencePiece model file
• bos_token ( str , optional , defaults to "<s>" ) — The beginning of sentence token.
• eos_token ( str , optional , defaults to "</s>" ) — The end of sentence token.
• unk_token ( str , optional , defaults to "<unk>" ) — The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead.
• pad_token ( str , optional , defaults to "<pad>" ) — The token used for padding, for example when batching sequences of different lengths.
• do_upper_case ( bool , optional , defaults to False ) — Whether or not to uppercase the output when decoding.
• do_lower_case ( bool , optional , defaults to False ) — Whether or not to lowercase the input when tokenizing.
• tgt_lang ( str , optional ) — A string representing the target language.

enable_sampling : Enable subword regularization.

nbest_size : Sampling parameters for unigram. Invalid for BPE-Dropout.

• nbest_size = {0,1} : No sampling is performed.
• nbest_size > 1 : samples from the nbest_size results.
• nbest_size < 0 : assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm.

alpha : Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout.

Construct an Speech2Text tokenizer.

This tokenizer inherits from PreTrainedTokenizer which contains some of the main methods. Users should refer to the superclass for more information regarding such methods.

build_inputs_with_special_tokens

( token_ids_0 token_ids_1 = None )

Build model inputs from a sequence by appending eos_token_id.

get_special_tokens_mask

• token_ids_0 ( List[int] ) — List of IDs.
• token_ids_1 ( List[int] , optional ) — Optional second list of IDs for sequence pairs.
• already_has_special_tokens ( bool , optional , defaults to False ) — Whether or not the token list is already formatted with special tokens for the model.

A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer prepare_for_model method.

create_token_type_ids_from_sequences

• token_ids_0 ( List[int] ) — The first tokenized sequence.
• token_ids_1 ( List[int] , optional ) — The second tokenized sequence.

The token type ids.

Create the token type IDs corresponding to the sequences passed. What are token type IDs?

Should be overridden in a subclass if the model has a special way of building those.

save_vocabulary

( save_directory : str filename_prefix : Optional = None )

Speech2TextFeatureExtractor

Class transformers. speech2textfeatureextractor.

( feature_size = 80 sampling_rate = 16000 num_mel_bins = 80 padding_value = 0.0 do_ceptral_normalize = True normalize_means = True normalize_vars = True **kwargs )

• feature_size ( int , optional , defaults to 80) — The feature dimension of the extracted features.
• sampling_rate ( int , optional , defaults to 16000) — The sampling rate at which the audio files should be digitalized expressed in hertz (Hz).
• num_mel_bins ( int , optional , defaults to 80) — Number of Mel-frequency bins.
• padding_value ( float , optional , defaults to 0.0) — The value that is used to fill the padding vectors.
• do_ceptral_normalize ( bool , optional , defaults to True ) — Whether or not to apply utterance-level cepstral mean and variance normalization to extracted features.
• normalize_means ( bool , optional , defaults to True ) — Whether or not to zero-mean normalize the extracted features.
• normalize_vars ( bool , optional , defaults to True ) — Whether or not to unit-variance normalize the extracted features.

Constructs a Speech2Text feature extractor.

This feature extractor inherits from Speech2TextFeatureExtractor which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

This class extracts mel-filter bank features from raw speech using TorchAudio if installed or using numpy otherwise, and applies utterance-level cepstral mean and variance normalization to the extracted features.

( raw_speech : Union padding : Union = False max_length : Optional = None truncation : bool = False pad_to_multiple_of : Optional = None return_tensors : Union = None sampling_rate : Optional = None return_attention_mask : Optional = None **kwargs )

• raw_speech ( np.ndarray , List[float] , List[np.ndarray] , List[List[float]] ) — The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep.
• True or 'longest' : Pad to the longest sequence in the batch (or no padding if only a single sequence if provided).
• 'max_length' : Pad to a maximum length specified with the argument max_length or to the maximum acceptable input length for the model if that argument is not provided.
• False or 'do_not_pad' (default): No padding (i.e., can output a batch with sequences of different lengths).
• max_length ( int , optional ) — Maximum length of the returned list and optionally padding length (see above).
• truncation ( bool ) — Activates truncation to cut input sequences longer than max_length to max_length .

What are attention masks?

For Speech2TextTransformer models, attention_mask should always be passed for batched inference, to avoid subtle bugs.

• 'tf' : Return TensorFlow tf.constant objects.
• 'pt' : Return PyTorch torch.Tensor objects.
• 'np' : Return Numpy np.ndarray objects.
• sampling_rate ( int , optional ) — The sampling rate at which the raw_speech input was sampled. It is strongly recommended to pass sampling_rate at the forward call to prevent silent errors.
• padding_value ( float , defaults to 0.0) — The value that is used to fill the padding values / vectors.

Main method to featurize and prepare for the model one or several sequence(s).

Speech2TextProcessor

Class transformers. speech2textprocessor.

( feature_extractor tokenizer )

• feature_extractor ( Speech2TextFeatureExtractor ) — An instance of Speech2TextFeatureExtractor . The feature extractor is a required input.
• tokenizer ( Speech2TextTokenizer ) — An instance of Speech2TextTokenizer . The tokenizer is a required input.

Constructs a Speech2Text processor which wraps a Speech2Text feature extractor and a Speech2Text tokenizer into a single processor.

Speech2TextProcessor offers all the functionalities of Speech2TextFeatureExtractor and Speech2TextTokenizer . See the call () and decode() for more information.

( *args **kwargs )

When used in normal mode, this method forwards all its arguments to Speech2TextFeatureExtractor’s call () and returns its output. If used in the context as_target_processor() this method forwards all its arguments to Speech2TextTokenizer’s call () . Please refer to the doctsring of the above two methods for more information.

from_pretrained

( pretrained_model_name_or_path : Union cache_dir : Union = None force_download : bool = False local_files_only : bool = False token : Union = None revision : str = 'main' **kwargs )

• a string, the model id of a pretrained feature_extractor hosted inside a model repo on huggingface.co.
• a path to a directory containing a feature extractor file saved using the save_pretrained() method, e.g., ./my_model_directory/ .
• a path or url to a saved feature extractor JSON file , e.g., ./my_model_directory/preprocessor_config.json . **kwargs — Additional keyword arguments passed along to both from_pretrained() and ~tokenization_utils_base.PreTrainedTokenizer.from_pretrained .

Instantiate a processor associated with a pretrained model.

This class method is simply calling the feature extractor from_pretrained() , image processor ImageProcessingMixin and the tokenizer ~tokenization_utils_base.PreTrainedTokenizer.from_pretrained methods. Please refer to the docstrings of the methods above for more information.

save_pretrained

( save_directory push_to_hub : bool = False **kwargs )

• save_directory ( str or os.PathLike ) — Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will be created if it does not exist).
• push_to_hub ( bool , optional , defaults to False ) — Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with repo_id (will default to the name of save_directory in your namespace).
• kwargs ( Dict[str, Any] , optional ) — Additional key word arguments passed along to the push_to_hub() method.

Saves the attributes of this processor (feature extractor, tokenizer…) in the specified directory so that it can be reloaded using the from_pretrained() method.

This class method is simply calling save_pretrained() and save_pretrained() . Please refer to the docstrings of the methods above for more information.

batch_decode

This method forwards all its arguments to Speech2TextTokenizer’s batch_decode() . Please refer to the docstring of this method for more information.

This method forwards all its arguments to Speech2TextTokenizer’s decode() . Please refer to the docstring of this method for more information.

Speech2TextModel

Class transformers. speech2textmodel.

( config : Speech2TextConfig )

• config ( Speech2TextConfig ) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The bare Speech2Text Model outputting raw hidden-states without any specific head on top. This model inherits from PreTrainedModel . Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

• input_features ( torch.FloatTensor of shape (batch_size, sequence_length, feature_size) ) — Float values of fbank features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a .flac or .wav audio file into an array of type List[float] or a numpy.ndarray , e.g. via the soundfile library ( pip install soundfile ). To prepare the array into input_features , the AutoFeatureExtractor should be used for extracting the fbank features, padding and conversion into a tensor of type torch.FloatTensor . See call ()
• 1 for tokens that are not masked ,
• 0 for tokens that are masked .

Indices can be obtained using SpeechToTextTokenizer . See PreTrainedTokenizer.encode() and PreTrainedTokenizer. call () for details.

What are decoder input IDs?

• 1 indicates the head is not masked ,
• 0 indicates the head is masked .
• encoder_outputs ( tuple(tuple(torch.FloatTensor) , optional ) — Tuple consists of ( last_hidden_state , optional : hidden_states , optional : attentions ) last_hidden_state of shape (batch_size, sequence_length, hidden_size) , optional ) is a sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.

Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see past_key_values input) to speed up sequential decoding.

• decoder_inputs_embeds ( torch.FloatTensor of shape (batch_size, target_sequence_length, hidden_size) , optional ) — Optionally, instead of passing decoder_input_ids you can choose to directly pass an embedded representation. If past_key_values is used, optionally only the last decoder_inputs_embeds have to be input (see past_key_values ). This is useful if you want more control over how to convert decoder_input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
• use_cache ( bool , optional ) — If set to True , past_key_values key value states are returned and can be used to speed up decoding (see past_key_values ).
• output_attentions ( bool , optional ) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
• output_hidden_states ( bool , optional ) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
• return_dict ( bool , optional ) — Whether or not to return a ModelOutput instead of a plain tuple.

transformers.modeling_outputs.Seq2SeqLMOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqLMOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False ) comprising various elements depending on the configuration ( Speech2TextConfig ) and inputs.

loss ( torch.FloatTensor of shape (1,) , optional , returned when labels is provided) — Language modeling loss.

logits ( torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size) ) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

past_key_values ( tuple(tuple(torch.FloatTensor)) , optional , returned when use_cache=True is passed or when config.use_cache=True ) — Tuple of tuple(torch.FloatTensor) of length config.n_layers , with each tuple having 2 tensors of shape (batch_size, num_heads, sequence_length, embed_size_per_head) ) and 2 additional tensors of shape (batch_size, num_heads, encoder_sequence_length, embed_size_per_head) .

decoder_hidden_states ( tuple(torch.FloatTensor) , optional , returned when output_hidden_states=True is passed or when config.output_hidden_states=True ) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size) .

Hidden-states of the decoder at the output of each layer plus the initial embedding outputs.

decoder_attentions ( tuple(torch.FloatTensor) , optional , returned when output_attentions=True is passed or when config.output_attentions=True ) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length) .

Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the self-attention heads.

cross_attentions ( tuple(torch.FloatTensor) , optional , returned when output_attentions=True is passed or when config.output_attentions=True ) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length) .

Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads.

encoder_last_hidden_state ( torch.FloatTensor of shape (batch_size, sequence_length, hidden_size) , optional ) — Sequence of hidden-states at the output of the last layer of the encoder of the model.

encoder_hidden_states ( tuple(torch.FloatTensor) , optional , returned when output_hidden_states=True is passed or when config.output_hidden_states=True ) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size) .

Hidden-states of the encoder at the output of each layer plus the initial embedding outputs.

encoder_attentions ( tuple(torch.FloatTensor) , optional , returned when output_attentions=True is passed or when config.output_attentions=True ) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length) .

Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the self-attention heads.

The Speech2TextModel forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Speech2TextForConditionalGeneration

Class transformers. speech2textforconditionalgeneration.

The Speech2Text Model with a language modeling head. Can be used for summarization. This model inherits from PreTrainedModel . Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

• labels ( torch.LongTensor of shape (batch_size, sequence_length) , optional ) — Labels for computing the language modeling loss. Indices should either be in [0, ..., config.vocab_size] or -100 (see input_ids docstring). Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size] .

The Speech2TextForConditionalGeneration forward method, overrides the __call__ special method.

TFSpeech2TextModel

Class transformers. tfspeech2textmodel.

( config : Speech2TextConfig *inputs **kwargs )

The bare Speech2Text Model outputting raw hidden-states without any specific head on top. This model inherits from TFPreTrainedModel . Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a keras.Model subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior.

TensorFlow models and layers in transformers accept two formats as input:

• having all inputs as keyword arguments (like PyTorch models), or
• having all inputs as a list, tuple or dict in the first positional argument.

The reason the second format is supported is that Keras methods prefer this format when passing inputs to models and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second format outside of Keras methods like fit() and predict() , such as when creating your own layers or models with the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first positional argument:

• a single Tensor with input_ids only and nothing else: model(input_ids)
• a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids])
• a dictionary with one or several input Tensors associated to the input names given in the docstring: model({"input_ids": input_ids, "token_type_ids": token_type_ids})

Note that when creating models and layers with subclassing then you don’t need to worry about any of this, as you can just pass inputs like you would to any other Python function!

• input_features ( tf.Tensor of shape (batch_size, sequence_length, feature_size) ) — Float values of fbank features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a .flac or .wav audio file into an array of type List[float] or a numpy.ndarray , e.g. via the soundfile library ( pip install soundfile ). To prepare the array into input_features , the AutoFeatureExtractor should be used for extracting the fbank features, padding and conversion into a tensor of floats. See call ()

Indices can be obtained using Speech2TextTokenizer . See PreTrainedTokenizer.encode() and PreTrainedTokenizer. call () for details.

SpeechToText uses the eos_token_id as the starting token for decoder_input_ids generation. If past_key_values is used, optionally only the last decoder_input_ids have to be input (see past_key_values ).

• decoder_attention_mask ( tf.Tensor of shape (batch_size, target_sequence_length) , optional ) — will be made by default and ignore pad tokens. It is not recommended to set this for most use cases.
• encoder_outputs ( tf.FloatTensor , optional ) — hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. of shape (batch_size, sequence_length, hidden_size) is a sequence of
• past_key_values ( Tuple[Tuple[tf.Tensor]] of length config.n_layers ) — contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all decoder_input_ids of shape (batch_size, sequence_length) .
• decoder_inputs_embeds ( tf.FloatTensor of shape (batch_size, target_sequence_length, hidden_size) , optional ) — Optionally, instead of passing decoder_input_ids you can choose to directly pass an embedded representation. If past_key_values is used, optionally only the last decoder_inputs_embeds have to be input (see past_key_values ). This is useful if you want more control over how to convert decoder_input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
• output_attentions ( bool , optional ) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
• output_hidden_states ( bool , optional ) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
• return_dict ( bool , optional ) — Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in eager mode, in graph mode the value will always be set to True.
• training ( bool , optional , defaults to False ) — Whether or not to use the model in training mode (some modules like dropout modules have different behaviors between training and evaluation).

transformers.modeling_tf_outputs.TFSeq2SeqModelOutput or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFSeq2SeqModelOutput or a tuple of tf.Tensor (if return_dict=False is passed or when config.return_dict=False ) comprising various elements depending on the configuration ( Speech2TextConfig ) and inputs.

last_hidden_state ( tf.Tensor of shape (batch_size, sequence_length, hidden_size) ) — Sequence of hidden-states at the output of the last layer of the decoder of the model.

If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.

past_key_values ( List[tf.Tensor] , optional , returned when use_cache=True is passed or when config.use_cache=True ) — List of tf.Tensor of length config.n_layers , with each tensor of shape (2, batch_size, num_heads, sequence_length, embed_size_per_head) ).

Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see past_key_values input) to speed up sequential decoding.

decoder_hidden_states ( tuple(tf.Tensor) , optional , returned when output_hidden_states=True is passed or when config.output_hidden_states=True ) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size) .

decoder_attentions ( tuple(tf.Tensor) , optional , returned when output_attentions=True is passed or when config.output_attentions=True ) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length) .

cross_attentions ( tuple(tf.Tensor) , optional , returned when output_attentions=True is passed or when config.output_attentions=True ) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length) .

encoder_last_hidden_state ( tf.Tensor of shape (batch_size, sequence_length, hidden_size) , optional ) — Sequence of hidden-states at the output of the last layer of the encoder of the model.

encoder_hidden_states ( tuple(tf.Tensor) , optional , returned when output_hidden_states=True is passed or when config.output_hidden_states=True ) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size) .

encoder_attentions ( tuple(tf.Tensor) , optional , returned when output_attentions=True is passed or when config.output_attentions=True ) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length) .

The TFSpeech2TextModel forward method, overrides the __call__ special method.

TFSpeech2TextForConditionalGeneration

Class transformers. tfspeech2textforconditionalgeneration.

The Speech2Text Model with a language modeling head. Can be used for summarization. This model inherits from TFPreTrainedModel . Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

• labels ( tf.Tensor of shape (batch_size, sequence_length) , optional ) — Labels for computing the masked language modeling loss. Indices should either be in [0, ..., config.vocab_size] or -100 (see input_ids docstring). Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size] .

transformers.modeling_tf_outputs.TFSeq2SeqLMOutput or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFSeq2SeqLMOutput or a tuple of tf.Tensor (if return_dict=False is passed or when config.return_dict=False ) comprising various elements depending on the configuration ( Speech2TextConfig ) and inputs.

loss ( tf.Tensor of shape (n,) , optional , where n is the number of non-masked labels, returned when labels is provided) — Language modeling loss.

logits ( tf.Tensor of shape (batch_size, sequence_length, config.vocab_size) ) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

The TFSpeech2TextForConditionalGeneration forward method, overrides the __call__ special method.

Speech to text

An AI Speech feature that accurately transcribes spoken audio to text.

Make spoken audio actionable

Quickly and accurately transcribe audio to text in more than 100 languages and variants. Customize models to enhance accuracy for domain-specific terminology. Get more value from spoken audio by enabling search or analytics on transcribed text or facilitating action—all in your preferred programming language.

High-quality transcription

Get accurate audio to text transcriptions with state-of-the-art speech recognition.

Customizable models

Add specific words to your base vocabulary or build your own speech-to-text models.

Flexible deployment

Run Speech to Text anywhere—in the cloud or at the edge in containers.

Production-ready

Access the same robust technology that powers speech recognition across Microsoft products.

Accurately transcribe speech from various sources

Convert audio to text from a range of sources, including  microphones ,  audio files , and  blob storage . Use speaker diarisation to determine who said what and when. Get readable transcripts with automatic formatting and punctuation.

Customize speech models to your needs

Tailor your speech models to understand organization- and industry-specific terminology. Overcome speech recognition barriers such as background noise, accents, or unique vocabulary.  Customize your models  by uploading audio data and transcripts. Automatically  generate custom models using Office 365 data  to optimize speech recognition accuracy for your organization.

Deploy anywhere

Run Speech to Text wherever your data resides. Build speech applications that are optimized for robust cloud capabilities and on-premises using  containers .

Fuel App Innovation with Cloud AI Services

Learn 5 key ways your organization can get started with AI to realize value quickly.

Comprehensive privacy and security

AI Speech, part of Azure AI Services, is  certified  by SOC, FedRAMP, PCI DSS, HIPAA, HITECH, and ISO.

View and delete your custom speech data and models at any time. Your data is encrypted while it's in storage.

Your data remains yours. Your audio input and transcription data aren't logged during audio processing.

Backed by Azure infrastructure, AI Speech offers enterprise-grade security, availability, compliance, and manageability.

Comprehensive security and compliance, built in

Microsoft invests more than $1 billion annually on cybersecurity research and development.

We employ more than 3,500 security experts who are dedicated to data security and privacy.

Azure has more certifications than any other cloud provider. View the comprehensive list .

Flexible pricing gives you the control you need

With Speech to Text, pay as you go based on the number of hours of audio you transcribe, with no upfront costs.

Get started with an Azure free account

After your credit, move to  pay as you go  to keep building with the same free services. Pay only if you use more than your free monthly amounts.

Documentation and resources

Get started.

Browse the  documentation

Create an AI Speech service with the  Microsoft Learn course

Explore code samples

Check out our  sample code

See customization resources

Explore and customize your voice-to-text solution with  Speech Studio . No code required.

Frequently asked questions about Speech to Text

What is speech to text.

It is a feature within the Speech service that accurately and quickly transcribes audio to text.

What are Azure AI Services?

AI Services  are a collection of customizable, prebuilt AI models that can be used to add AI to applications. There are a variety of domains, including Speech, Decision, Language, and Vision. Speech to Text is one feature within the Speech service. Other Speech related features include  Text to Speech ,  Speech Translation , and  Speaker Recognition . An example of a Decision service is  Personalizer , which allows you to deliver personalized, relevant experiences. Examples of AI Languages include  Language Understanding ,  Text Analytics  for natural language processing,  QnA Maker  for FAQ experiences, and  Translator  for language translation.

Start building with AI Services

Our approach

• Responsibility
• Infrastructure
• Try Meta AI

Equipping machines with the ability to recognize and produce speech can make information accessible to many more people, including those who rely entirely on voice to access information. However, producing good-quality machine learning models for these tasks requires large amounts of labeled data — in this case, many thousands of hours of audio, along with transcriptions. For most languages, this data simply does not exist. For example, existing speech recognition models only cover approximately 100 languages — a fraction of the 7,000+ known languages spoken on the planet. Even more concerning, nearly half of these languages are in danger of disappearing in our lifetime.

RECOMMENDED READS

• Meta’s new AI-powered translation system for Hokkien pioneers new approach for unwritten languages
• MuAViC: The first audio-video speech translation benchmark
• Advancing direct speech-to-speech modeling with discrete units

In the Massively Multilingual Speech (MMS) project, we overcome some of these challenges by combining wav2vec 2.0 , our pioneering work in self-supervised learning, and a new dataset that provides labeled data for over 1,100 languages and unlabeled data for nearly 4,000 languages. Some of these, such as the Tatuyo language, have only a few hundred speakers, and for most of these languages, no prior speech technology exists. Our results show that the Massively Multilingual Speech models outperform existing models and cover 10 times as many languages. Meta is focused on multilinguality in general: For text, the NLLB project scaled multilingual translation to 200 languages , and the Massively Multilingual Speech project scales speech technology to many more languages.

Today, we are publicly sharing our models and code so that others in the research community can build upon our work. Through this work, we hope to make a small contribution to preserve the incredible language diversity of the world.

Collecting audio data for thousands of languages was our first challenge because the largest existing speech datasets cover at most 100 languages. To overcome it, we turned to religious texts, such as the Bible, that have been translated in many different languages and whose translations have been widely studied for text-based language translation research. These translations have publicly available audio recordings of people reading these texts in different languages. As part of this project, we created a dataset of readings of the New Testament in over 1,100 languages, which provided on average 32 hours of data per language.

By considering unlabeled recordings of various other Christian religious readings, we increased the number of languages available to over 4,000. While this data is from a specific domain and is often read by male speakers, our analysis shows that our models perform equally well for male and female voices. And while the content of the audio recordings is religious, our analysis shows that this does not overly bias the model to produce more religious language. We believe this is because we use a Connectionist Temporal Classification approach, which is far more constrained compared with large language models (LLMs) or sequence to-sequence models for speech recognition.

We preprocessed the data to improve quality and to make it usable by our machine learning algorithms. To do so, we trained an alignment model on existing data in over 100 languages and used this model together with an efficient forced alignment algorithm that can process very long recordings of about 20 minutes or more. We applied multiple rounds of this process and performed a final cross-validation filtering step based on model accuracy to remove potentially misaligned data. To enable other researchers to create new speech datasets, we added the alignment algorithm to PyTorch and released the alignment model.

Thirty-two hours of data per language is not enough to train conventional supervised speech recognition models. This is why we built on wav2vec 2.0 , our prior work on self-supervised speech representation learning, which greatly reduced the amount of labeled data needed to train good systems. Concretely, we trained self-supervised models on about 500,000 hours of speech data in over 1,400 languages — this is nearly five times more languages than any known prior work. The resulting models were then fine-tuned for a specific speech task, such as multilingual speech recognition or language identification.

To get a better understanding of how well models trained on the Massively Multilingual Speech data perform, we evaluated them on existing benchmark datasets, such as FLEURS .

We trained multilingual speech recognition models on over 1,100 languages using a 1B parameter wav2vec 2.0 model. As the number of languages increases, performance does decrease, but only very slightly: Moving from 61 to 1,107 languages increases the character error rate by only about 0.4 percent but increases the language coverage by over 18 times.

In a like-for-like comparison with OpenAI’s Whisper , we found that models trained on the Massively Multilingual Speech data achieve half the word error rate, but Massively Multilingual Speech covers 11 times more languages. This demonstrates that our model can perform very well compared with the best current speech models.

Next, we trained a language identification (LID) model for over 4,000 languages using our datasets as well as existing datasets, such as FLEURS and CommonVoice, and evaluated it on the FLEURS LID task. It turns out that supporting 40 times the number languages still results in very good performance.

We also built text-to-speech systems for over 1,100 languages. Current text-to-speech models are typically trained on speech corpora that contain only a single speaker. A limitation of the Massively Multilingual Speech data is that it contains relatively few different speakers for many languages, and often only a single speaker. However, this is an advantage for building text-to-speech systems, and so we trained such systems for over 1,100 languages. We found that the speech produced by these systems is of good quality, as the examples below show.

We are encouraged by our results, but as with all new AI technologies, our models aren’t perfect. For example, there is some risk that the speech-to-text model may mistranscribe select words or phrases. Depending on the output, this could result in offensive and/or inaccurate language. We continue to believe that collaboration across the AI community is critical to the responsible development of AI technologies.

Toward a single speech model supporting thousands of languages

Many of the world’s languages are in danger of disappearing, and the limitations of current speech recognition and speech generation technology will only accelerate this trend. We envision a world where technology has the opposite effect, encouraging people to keep their languages alive since they can access information and use technology by speaking in their preferred language.

The Massively Multilingual Speech project presents a significant step forward in this direction. In the future, we want to increase the language coverage to support even more languages, and also tackle the challenge of handling dialects, which is often difficult for existing speech technology. Our goal is to make it easier for people to access information and to use devices in their preferred language. There are also many concrete use cases for speech technology — such as VR/AR technology — which can be used in a person’s preferred language - to messaging services that can understand everyone’s voice.

We also envision a future where a single model can solve several speech tasks for all languages. While we trained separate models for speech recognition, speech synthesis, and language identification, we believe that in the future, a single model will be able to accomplish all these tasks and more, leading to better overall performance.

This blog post was made possible by the work of Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Ali Elkahky, Zhaoheng Ni, Sayani Kundu, Maryam Fazel-Zarandi, Apoorv Vyas, Alexei Baevski, Yossef Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli.

Our latest updates delivered to your inbox

Subscribe to our newsletter to keep up with Meta AI news, events, research breakthroughs, and more.

Join us in the pursuit of what’s possible with AI.

Product experiences

Foundational models

Latest news

Meta © 2024

Navigation Menu

Search code, repositories, users, issues, pull requests..., provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications You must be signed in to change notification settings

A robust, efficient, low-latency speech-to-text library with advanced voice activity detection, wake word activation and instant transcription.

KoljaB/RealtimeSTT

Folders and files, repository files navigation, realtimestt.

Easy-to-use, low-latency speech-to-text library for realtime applications

About the Project

RealtimeSTT listens to the microphone and transcribes voice into text.

Hint: Check out Linguflex , the original project from which RealtimeSTT is spun off. It lets you control your environment by speaking and is one of the most capable and sophisticated open-source assistants currently available.

It's ideal for:

• Voice Assistants
• Applications requiring fast and precise speech-to-text conversion

Latest Version: v0.1.15

See release history .

Hint: Since we use the multiprocessing module now, ensure to include the if __name__ == '__main__': protection in your code to prevent unexpected behavior, especially on platforms like Windows. For a detailed explanation on why this is important, visit the official Python documentation on multiprocessing .

• Voice Activity Detection : Automatically detects when you start and stop speaking.
• Realtime Transcription : Transforms speech to text in real-time.
• Wake Word Activation : Can activate upon detecting a designated wake word.

Hint : Check out RealtimeTTS , the output counterpart of this library, for text-to-voice capabilities. Together, they form a powerful realtime audio wrapper around large language models.

This library uses:

• WebRTCVAD for initial voice activity detection.
• SileroVAD for more accurate verification.
• Faster_Whisper for instant (GPU-accelerated) transcription.
• Porcupine for wake word detection.

These components represent the "industry standard" for cutting-edge applications, providing the most modern and effective foundation for building high-end solutions.

Installation

This will install all the necessary dependencies, including a CPU support only version of PyTorch.

Although it is possible to run RealtimeSTT with a CPU installation only (use a small model like "tiny" or "base" in this case) you will get way better experience using:

GPU Support with CUDA (recommended)

Additional steps are needed for a GPU-optimized installation. These steps are recommended for those who require better performance and have a compatible NVIDIA GPU.

Note : To check if your NVIDIA GPU supports CUDA, visit the official CUDA GPUs list .

To use RealtimeSTT with GPU support via CUDA please follow these steps:

Install NVIDIA CUDA Toolkit 11.8 :

• Visit NVIDIA CUDA Toolkit Archive .
• Select operating system and version.
• Download and install the software.

Install NVIDIA cuDNN 8.7.0 for CUDA 11.x :

• Visit NVIDIA cuDNN Archive .
• Click on "Download cuDNN v8.7.0 (November 28th, 2022), for CUDA 11.x".

Install ffmpeg :

Note : Installation of ffmpeg might not actually be needed to operate RealtimeSTT *thanks to jgilbert2017 for pointing this out

You can download an installer for your OS from the ffmpeg Website .

Or use a package manager:

On Ubuntu or Debian :

On Arch Linux :

On MacOS using Homebrew ( https://brew.sh/ ):

On Windows using Winget official documentation :

On Windows using Chocolatey ( https://chocolatey.org/ ):

On Windows using Scoop ( https://scoop.sh/ ):

Install PyTorch with CUDA support :

Quick Start

Basic usage:

Manual Recording

Start and stop of recording are manually triggered.

Automatic Recording

Recording based on voice activity detection.

When running recorder.text in a loop it is recommended to use a callback, allowing the transcription to be run asynchronously:

Keyword activation before detecting voice. Write the comma-separated list of your desired activation keywords into the wake_words parameter. You can choose wake words from these list: alexa, americano, blueberry, bumblebee, computer, grapefruits, grasshopper, hey google, hey siri, jarvis, ok google, picovoice, porcupine, terminator.

You can set callback functions to be executed on different events (see Configuration ) :

Feed chunks

If you don't want to use the local microphone set use_microphone parameter to false and provide raw PCM audiochunks in 16-bit mono (samplerate 16000) with this method:

You can shutdown the recorder safely by using the context manager protocol:

Or you can call the shutdown method manually (if using "with" is not feasible):

Testing the Library

The test subdirectory contains a set of scripts to help you evaluate and understand the capabilities of the RealtimeTTS library.

Test scripts depending on RealtimeTTS library may require you to enter your azure service region within the script. When using OpenAI-, Azure- or Elevenlabs-related demo scripts the API Keys should be provided in the environment variables OPENAI_API_KEY, AZURE_SPEECH_KEY and ELEVENLABS_API_KEY (see RealtimeTTS )

simple_test.py

• Description : A "hello world" styled demonstration of the library's simplest usage.

realtimestt_test.py

• Description : Showcasing live-transcription.

wakeword_test.py

• Description : A demonstration of the wakeword activation.

translator.py

• Dependencies : Run pip install openai realtimetts .
• Description : Real-time translations into six different languages.

openai_voice_interface.py

• Description : Wake word activated and voice based user interface to the OpenAI API.

advanced_talk.py

• Dependencies : Run pip install openai keyboard realtimetts .
• Description : Choose TTS engine and voice before starting AI conversation.

minimalistic_talkbot.py

• Description : A basic talkbot in 20 lines of code.

The example_app subdirectory contains a polished user interface application for the OpenAI API based on PyQt5.

Configuration

Initialization parameters for audiototextrecorder.

When you initialize the AudioToTextRecorder class, you have various options to customize its behavior.

General Parameters

model (str, default="tiny"): Model size or path for transcription.

• Options: 'tiny', 'tiny.en', 'base', 'base.en', 'small', 'small.en', 'medium', 'medium.en', 'large-v1', 'large-v2'.
• Note: If a size is provided, the model will be downloaded from the Hugging Face Hub.

language (str, default=""): Language code for transcription. If left empty, the model will try to auto-detect the language. Supported language codes are listed in Whisper Tokenizer library .

compute_type (str, default="default"): Specifies the type of computation to be used for transcription. See Whisper Quantization

input_device_index (int, default=0): Audio Input Device Index to use.

gpu_device_index (int, default=0): GPU Device Index to use. The model can also be loaded on multiple GPUs by passing a list of IDs (e.g. [0, 1, 2, 3]).

device (str, default="cuda"): Device for model to use. Can either be "cuda" or "cpu".

on_recording_start : A callable function triggered when recording starts.

on_recording_stop : A callable function triggered when recording ends.

on_transcription_start : A callable function triggered when transcription starts.

ensure_sentence_starting_uppercase (bool, default=True): Ensures that every sentence detected by the algorithm starts with an uppercase letter.

ensure_sentence_ends_with_period (bool, default=True): Ensures that every sentence that doesn't end with punctuation such as "?", "!" ends with a period

use_microphone (bool, default=True): Usage of local microphone for transcription. Set to False if you want to provide chunks with feed_audio method.

spinner (bool, default=True): Provides a spinner animation text with information about the current recorder state.

level (int, default=logging.WARNING): Logging level.

handle_buffer_overflow (bool, default=True): If set, the system will log a warning when an input overflow occurs during recording and remove the data from the buffer.

beam_size (int, default=5): The beam size to use for beam search decoding.

initial_prompt (str or iterable of int, default=None): Initial prompt to be fed to the transcription models.

suppress_tokens (list of int, default=[-1]): Tokens to be suppressed from the transcription output.

on_recorded_chunk : A callback function that is triggered when a chunk of audio is recorded. Submits the chunk data as parameter.

debug_mode (bool, default=False): If set, the system prints additional debug information to the console.

Real-time Transcription Parameters

Note : When enabling realtime description a GPU installation is strongly advised. Using realtime transcription may create high GPU loads.

enable_realtime_transcription (bool, default=False): Enables or disables real-time transcription of audio. When set to True, the audio will be transcribed continuously as it is being recorded.

realtime_model_type (str, default="tiny"): Specifies the size or path of the machine learning model to be used for real-time transcription.

• Valid options: 'tiny', 'tiny.en', 'base', 'base.en', 'small', 'small.en', 'medium', 'medium.en', 'large-v1', 'large-v2'.

realtime_processing_pause (float, default=0.2): Specifies the time interval in seconds after a chunk of audio gets transcribed. Lower values will result in more "real-time" (frequent) transcription updates but may increase computational load.

on_realtime_transcription_update : A callback function that is triggered whenever there's an update in the real-time transcription. The function is called with the newly transcribed text as its argument.

on_realtime_transcription_stabilized : A callback function that is triggered whenever there's an update in the real-time transcription and returns a higher quality, stabilized text as its argument.

beam_size_realtime (int, default=3): The beam size to use for real-time transcription beam search decoding.

Voice Activation Parameters

silero_sensitivity (float, default=0.6): Sensitivity for Silero's voice activity detection ranging from 0 (least sensitive) to 1 (most sensitive). Default is 0.6.

silero_use_onnx (bool, default=False): Enables usage of the pre-trained model from Silero in the ONNX (Open Neural Network Exchange) format instead of the PyTorch format. Default is False. Recommended for faster performance.

webrtc_sensitivity (int, default=3): Sensitivity for the WebRTC Voice Activity Detection engine ranging from 0 (least aggressive / most sensitive) to 3 (most aggressive, least sensitive). Default is 3.

post_speech_silence_duration (float, default=0.2): Duration in seconds of silence that must follow speech before the recording is considered to be completed. This ensures that any brief pauses during speech don't prematurely end the recording.

min_gap_between_recordings (float, default=1.0): Specifies the minimum time interval in seconds that should exist between the end of one recording session and the beginning of another to prevent rapid consecutive recordings.

min_length_of_recording (float, default=1.0): Specifies the minimum duration in seconds that a recording session should last to ensure meaningful audio capture, preventing excessively short or fragmented recordings.

pre_recording_buffer_duration (float, default=0.2): The time span, in seconds, during which audio is buffered prior to formal recording. This helps counterbalancing the latency inherent in speech activity detection, ensuring no initial audio is missed.

on_vad_detect_start : A callable function triggered when the system starts to listen for voice activity.

on_vad_detect_stop : A callable function triggered when the system stops to listen for voice activity.

Wake Word Parameters

wake_words (str, default=""): Wake words for initiating the recording. Multiple wake words can be provided as a comma-separated string. Supported wake words are: alexa, americano, blueberry, bumblebee, computer, grapefruits, grasshopper, hey google, hey siri, jarvis, ok google, picovoice, porcupine, terminator

wake_words_sensitivity (float, default=0.6): Sensitivity level for wake word detection (0 for least sensitive, 1 for most sensitive).

wake_word_activation_delay (float, default=0): Duration in seconds after the start of monitoring before the system switches to wake word activation if no voice is initially detected. If set to zero, the system uses wake word activation immediately.

wake_word_timeout (float, default=5): Duration in seconds after a wake word is recognized. If no subsequent voice activity is detected within this window, the system transitions back to an inactive state, awaiting the next wake word or voice activation.

on_wakeword_detected : A callable function triggered when a wake word is detected.

on_wakeword_timeout : A callable function triggered when the system goes back to an inactive state after when no speech was detected after wake word activation.

on_wakeword_detection_start : A callable function triggered when the system starts to listen for wake words

on_wakeword_detection_end : A callable function triggered when stopping to listen for wake words (e.g. because of timeout or wake word detected)

Contribution

Contributions are always welcome!

Shoutout to Steven Linn for providing docker support.

Kolja Beigel Email: [email protected] GitHub

Releases 11

Contributors 5.

• Python 93.9%
• JavaScript 3.0%
• Batchfile 1.6%

This browser is no longer supported.

Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.

Implement custom speech to text

This two-part guide describes various approaches for efficiently implementing high-quality speech-aware applications. It focuses on extending and customizing the baseline model of speech to text functionality that's provided by the AI Speech service .

This article describes the problem space and decision-making process for designing your solution. The second article, Deploy a custom speech to text solution , provides a use case for applying these instructions and recommended practices.

The pre-built and custom AI spectrum

The pre-built and custom AI spectrum represents multiple AI model customization and development effort tiers, ranging from ready-to-use pre-built models to fully customized AI solutions.

Pre-built and pre-trained models are on the left side, customized pre-built models are in the middle, and customized models tailored to your scenario and data are on the right side.

On the left side of the spectrum, Azure AI services enables a quick and low-friction implementation of AI capabilities into applications via pre-trained models. Microsoft curates extensive datasets to train and build these baseline models. As a result, you can use baseline models with no additional training data. They're consumed via enhanced-security programmatic API calls.

Azure AI services includes:

• Speech. Speech to text, text to speech, speech translation, and Speaker Recognition
• Language. Entity recognition, sentiment analysis, question answering, conversational language understanding, and translator
• Vision. Computer vision and Face API
• Decision. Anomaly detector, Content Moderator, and Personalizer
• OpenAI Service. Advanced language models

When the pre-built baseline models don't perform accurately enough on your data, you can customize them by adding training data that's relative to the problem domain. This customization requires the extra effort of gathering adequate data to train and evaluate an acceptable model. Azure AI services that are customizable include Custom Vision , Custom Translator , Custom Speech , and CLU . Extending pre-built Azure AI services models is in the center of the spectrum. Most of this article is focused on that central area.

Alternatively, when models and training data focus on a specific scenario and require a proprietary training dataset, Azure Machine Learning provides custom solution resources, tools, compute, and workflow guidance to support building entirely custom models. This scenario appears on the right side of the spectrum. These models are built from scratch. Developing a model by using Azure Machine Learning typically ranges from using visual tools like AutoML to programmatically developing the model by using notebooks .

Azure Speech service

Azure Speech service unifies speech to text, text to speech, speech translation, voice assistant, and Speaker Recognition functionality into a single subscription that's based on Azure AI services. You can enable an application for speech by integrating with Speech service via easy-to-use SDKs and APIs.

The Azure speech to text service analyzes audio in real time or asynchronously to transcribe the spoken word into text. Out of the box, Azure speech to text uses a Universal Language Model as a baseline that reflects commonly used spoken language. This baseline model is pre-trained with dialects and phonetics that represent a variety of common domains. As a result, consuming the baseline model requires no extra configuration and works well in most scenarios.

Note, however, that the baseline model might not be sufficient if the audio contains ambient noise or includes a lot of industry and domain-specific jargon. In these cases, building a custom speech model makes sense. You do that by training with additional data that's associated with the specific domain.

Depending on the size of the custom domain, it might also make sense to train multiple models and compartmentalize a model for an individual application. For example, Olympics commentators report on various sports, each with its own jargon. Because each sport has a vocabulary that differs significantly from the others, building a custom model specific to a sport increases accuracy by limiting the utterance data relative to that particular sport. As a result, the model can learn from a precise and targeted set of data.

So there are three approaches to implementing Azure speech to text:

• The baseline model is appropriate when the audio is clear of ambient noise and the transcribed speech consists of commonly spoken language.
• A custom model augments the baseline model to include domain-specific vocabulary that's shared across all areas of the custom domain.
• Multiple custom models make sense when the custom domain has numerous areas, each with a specific vocabulary.

Potential use cases

Here are some generic scenarios and use cases in which custom speech to text is helpful:

• Speech transcription for a specific domain, like medical transcription or call center transcription
• Live transcription, as in an app or to provide captions for live video streaming

Microsoft SDKs and open-source tools

When you're working with speech to text, you might find these resources helpful:

• Azure Speech SDK
• Speech Studio
• FFMpeg / SOX

Design considerations

This section describes some design considerations for building a speech-based application.

Baseline model vs. custom model

Azure Speech includes baseline models that support various languages. These models are pre-trained with a vast amount of vocabulary and domains. However, you might have a specialized vocabulary that needs recognition. In these situations, baseline models might fall short. The best way to determine whether the base model will suffice is to analyze the transcription that's produced from the baseline model and compare it to a human-generated transcript for the same audio. The deployment article in this guide describes using Speech Studio to compare the transcripts and obtain a word error rate (WER) score. If there are multiple incorrect word substitutions in the results, we recommend that you train a custom model to recognize those words.

One vs. many custom models

If your scenario will benefit from a custom model, you next need to determine how many models to build. One model is typically sufficient if the utterances are closely related to one area or domain. However, multiple models are best if the vocabulary is significantly different across the domain areas. In this scenario, you also need a variety of training data.

Let's return to Olympics example. Say you need to include the transcription of audio commentary for multiple sports, including ice hockey, luge, snowboarding, alpine skiing, and more. Building a custom speech model for each sport will improve accuracy because each sport has unique terminology. However, each model must have diverse training data. It's too restrictive and inextensible to create a model for each commentator for each sport. A more practical approach is to build a single model for each sport but include audio from a group of that includes commentators with different accents, of both genders, and of various ages. All domain-specific phrases related to the sport as captured by the diverse commentators reside in the same model.

You also need to consider which languages and locales to support. It might make sense to create these models by locale.

Acoustic and language model adaptation

Azure Speech provides three options for training a custom model:

Language model adaptation is the most commonly used customization. A language model helps to train how certain words are used together in a particular context or a specific domain. Building a language model is also relatively easy and fast. First, train the model by supplying a variety of utterances and phrases for the particular domain. For example, if the goal is to generate transcription for alpine skiing, collect human-generated transcripts of multiple skiing events. Clean and combine them to create one training data file with about 50 thousand phrases and sentences. For more details about the data requirements for custom language model training, see Training and testing datasets .

Pronunciation model customization is also one of the most commonly used customizations. A pronunciation model helps the custom model recognize uncommon words that don't have a standard pronunciation. For example, some of the terminology in alpine skiing borrows from other languages, like the terms schuss and mogul . These words are excellent candidates for training with a pronunciation dataset. For more details about improving recognition by using a pronunciation file, see Pronunciation data for training . For details about building a custom model by using Speech Studio, see What is Custom Speech? .

Acoustic model adaptation provides phonetic training on the pronunciation of certain words so that Azure Speech can properly recognize them. To build an acoustic model, you need audio samples and accompanying human-generated transcripts. If the recognition language matches common locales, like en-US, using the current baseline model should be sufficient. Baseline models have diverse training that uses the voices of native and non-native English speakers to cover a vast amount of English vocabulary. Therefore, building an acoustic model adaptation on the en-US base model might not provide much improvement. Training a custom acoustic model also takes a bit more time. For more information about the data requirements for custom acoustic training, see Training and testing datasets .

The final custom model can include datasets that use a combination of all three of the customizations described in this section.

Training a custom model

There are two approaches to training a custom model:

Train with numerous examples of phrases and utterances from the domain. For example, include transcripts of cleaned and normalized alpine skiing event audio and human-generated transcripts of previous events. Be sure that the transcripts include the terms used in alpine skiing and multiple examples of how commentators pronounce them. If you follow this process, the resulting custom model should be able to recognize domain-specific words and phrases.

Train with specific data that focuses on problem areas. This approach works well when there isn't much training data, for example, if new slang terms are used during alpine skiing events and need to be included in the model. This type of training uses the following approach:

• Use Speech Studio to generate a transcription and compare it with human-generated transcriptions.
• The contexts within which the problem word or utterance is applied.
• Different inflections and pronunciations of the word or utterance.
• Any unique commentator-specific applications of the word or utterance.

Training a custom model with specific data can be time-consuming. Steps include carefully analyzing the transcription gaps, manually adding training phrases, and repeating this process multiple times. However, in the end, this approach provides focused training for the problem areas that were previously incorrectly transcribed. And it's possible to iteratively build this model by selectively training on critical areas and then proceeding down the list in order of importance. Another benefit is that the dataset size will include a few hundred utterances rather than a few thousand, even after many iterations of building the training data.

After you build your model

After you build your model, keep the following recommendations in mind:

Be aware of the difference between lexical text and display text. Speech Studio produces WER based on lexical text. However, what the user sees is the display text with punctuation, capitalization, and numerical words represented as numbers. Following is an example of lexical text versus display text.

Lexical text: the speed is great and the time is even better fifty seven oh six three seconds for the German

Display text: The speed is great. And that time is even better. 57063 seconds for the German.

What's expected (implied) is: The speed is great. And that time is even better. 57.063 seconds for the German

The custom model has a low WER rate, but that doesn't mean that user-perceived error rate (errors in display text) is low. This problem occurs mainly in alphanumeric input because different applications can have alternative ways of representing the input. You shouldn't rely only on the WER. You also need to review the final recognition result.

When display text seems wrong, review the detailed recognition result from the SDK, which includes lexical text, in which everything is spelled out. If the lexical text is correct, the recognition is accurate. You can then resolve inaccuracies in the display text (the final recognized result) by adding post-processing rules.

Manage datasets, models, and their versions. In Speech Studio, when you create projects, datasets, and models, there are only two fields: name and description. When you build datasets and models iteratively, you need to follow a good naming and versioning scheme to make it easy to identify the contents of a dataset and which model reflects which version of the dataset. For more details about this recommendation, see Deploy a custom speech to text solution .

Go to part two of this guide: deployment

Contributors

This article is maintained by Microsoft. It was originally written by the following contributors.

Principal author:

• Pratyush Mishra | Principal Engineering Manager

Other contributors:

• Mick Alberts | Technical Writer
• Rania Bayoumy | Senior Technical Program Manager

To see non-public LinkedIn profiles, sign in to LinkedIn.

• What is Custom Speech?
• What is text to speech?
• Train a Custom Speech model
• Deploy a custom speech to text solution

Related resources

• Artificial intelligence (AI) architecture design
• Use a speech to text transcription pipeline to analyze recorded conversations
• Control IoT devices with a voice assistant app

Was this page helpful?

Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback .

Submit and view feedback for

Additional resources

Subscribe to the PwC Newsletter

Join the community, add a new evaluation result row, text-to-speech synthesis.

94 papers with code • 6 benchmarks • 17 datasets

Text-To-Speech Synthesis is a machine learning task that involves converting written text into spoken words. The goal is to generate synthetic speech that sounds natural and resembles human speech as closely as possible.

Benchmarks Add a Result

Most implemented papers

Fastspeech 2: fast and high-quality end-to-end text to speech.

In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e. g., pitch, energy and more accurate duration) as conditional inputs.

Tacotron: Towards End-to-End Speech Synthesis

A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module.

Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention

This paper describes a novel text-to-speech (TTS) technique based on deep convolutional neural networks (CNN), without use of any recurrent units.

FastSpeech: Fast, Robust and Controllable Text to Speech

In this work, we propose a novel feed-forward network based on Transformer to generate mel-spectrogram in parallel for TTS.

Efficient Neural Audio Synthesis

The small number of weights in a Sparse WaveRNN makes it possible to sample high-fidelity audio on a mobile CPU in real time.

Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram

We propose Parallel WaveGAN, a distillation-free, fast, and small-footprint waveform generation method using a generative adversarial network.

Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis

In this work, we propose "global style tokens" (GSTs), a bank of embeddings that are jointly trained within Tacotron, a state-of-the-art end-to-end speech synthesis system.

Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis

Clone a voice in 5 seconds to generate arbitrary speech in real-time

FastSpeech: Fast,Robustand Controllable Text-to-Speech

Compared with traditional concatenative and statistical parametric approaches, neural network based end-to-end models suffer from slow inference speed, and the synthesized speech is usually not robust (i. e., some words are skipped or repeated) and lack of controllability (voice speed or prosody control).

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism

Singing voice synthesis (SVS) systems are built to synthesize high-quality and expressive singing voice, in which the acoustic model generates the acoustic features (e. g., mel-spectrogram) given a music score.

• For Mobile & Edge

Retrain a speech recognition model with TensorFlow Lite Model Maker

In this colab notebook, you'll learn how to use the TensorFlow Lite Model Maker to train a speech recognition model that can classify spoken words or short phrases using one-second sound samples. The Model Maker library uses transfer learning to retrain an existing TensorFlow model with a new dataset, which reduces the amount of sample data and time required for training.

By default, this notebook retrains the model (BrowserFft, from the TFJS Speech Command Recognizer ) using a subset of words from the speech commands dataset (such as "up," "down," "left," and "right"). Then it exports a TFLite model that you can run on a mobile device or embedded system (such as a Raspberry Pi). It also exports the trained model as a TensorFlow SavedModel.

This notebook is also designed to accept a custom dataset of WAV files, uploaded to Colab in a ZIP file. The more samples you have for each class, the better your accuracy will be, but because the transfer learning process uses feature embeddings from the pre-trained model, you can still get a fairly accurate model with only a few dozen samples in each of your classes.

If you want to run the notebook with the default speech dataset, you can run the whole thing now by clicking Runtime > Run all in the Colab toolbar. However, if you want to use your own dataset, then continue down to Prepare the dataset and follow the instructions there.

Import the required packages

You'll need TensorFlow, TFLite Model Maker, and some modules for audio manipulation, playback, and visualizations.

Prepare the dataset

To train with the default speech dataset, just run all the code below as-is.

But if you want to train with your own speech dataset, follow these steps:

• Be sure each sample in your dataset is in WAV file format, about one second long . Then create a ZIP file with all your WAV files, organized into separate subfolders for each classification. For example, each sample for a speech command "yes" should be in a subfolder named "yes". Even if you have only one class, the samples must be saved in a subdirectory with the class name as the directory name. (This script assumes your dataset is not split into train/validation/test sets and performs that split for you.)
• Click the Files tab in the left panel and just drag-drop your ZIP file there to upload it.
• Use the following drop-down option to set use_custom_dataset to True.
• Then skip to Prepare a custom audio dataset to specify your ZIP filename and dataset directory name.

Toggle code

Generate a background noise dataset

Whether you're using the default speech dataset or a custom dataset, you should have a good set of background noises so your model can distinguish speech from other noises (including silence).

Because the following background samples are provided in WAV files that are a minute long or longer, we need to split them up into smaller one-second samples so we can reserve some for our test dataset. We'll also combine a couple different sample sources to build a comprehensive set of background noises and silence:

Prepare the speech commands dataset

We already downloaded the speech commands dataset, so now we just need to prune the number of classes for our model.

This dataset includes over 30 speech command classifications, and most of them have over 2,000 samples. But because we're using transfer learning, we don't need that many samples. So the following code does a few things:

• Specify which classifications we want to use, and delete the rest.
• Keep only 150 samples of each class for training (to prove that transfer learning works well with smaller datasets and simply to reduce the training time).
• Create a separate directory for a test dataset so we can easily run inference with them later.

Prepare a custom dataset

If you want to train the model with our own speech dataset, you need to upload your samples as WAV files in a ZIP ( as described above ) and modify the following variables to specify your dataset:

After changing the filename and path name above, you're ready to train the model with your custom dataset. In the Colab toolbar, select Runtime > Run all to run the whole notebook.

The following code integrates our new background noise samples into your dataset and then separates a portion of all samples to create a test set.

Play a sample

To be sure the dataset looks correct, let's play at a random sample from the test set:

Define the model

When using Model Maker to retrain any model, you have to start by defining a model spec. The spec defines the base model from which your new model will extract feature embeddings to begin learning new classes. The spec for this speech recognizer is based on the pre-trained BrowserFft model from TFJS .

The model expects input as an audio sample that's 44.1 kHz, and just under a second long: the exact sample length must be 44034 frames.

You don't need to do any resampling with your training dataset. Model Maker takes care of that for you. But when you later run inference, you must be sure that your input matches that expected format.

All you need to do here is instantiate the BrowserFftSpec :

Load your dataset

Now you need to load your dataset according to the model specifications. Model Maker includes the DataLoader API, which will load your dataset from a folder and ensure it's in the expected format for the model spec.

We already reserved some test files by moving them to a separate directory, which makes it easier to run inference with them later. Now we'll create a DataLoader for each split: the training set, the validation set, and the test set.

Load the speech commands dataset

Load a custom dataset, train the model.

Now we'll use the Model Maker create() function to create a model based on our model spec and training dataset, and begin training.

If you're using a custom dataset, you might want to change the batch size as appropriate for the number of samples in your train set.

Review the model performance

Even if the accuracy/loss looks good from the training output above, it's important to also run the model using test data that the model has not seen yet, which is what the evaluate() method does here:

View the confusion matrix

When training a classification model such as this one, it's also useful to inspect the confusion matrix . The confusion matrix gives you detailed visual representation of how well your classifier performs for each classification in your test data.

Export the model

The last step is exporting your model into the TensorFlow Lite format for execution on mobile/embedded devices and into the SavedModel format for execution elsewhere.

When exporting a .tflite file from Model Maker, it includes model metadata that describes various details that can later help during inference. It even includes a copy of the classification labels file, so you don't need to a separate labels.txt file. (In the next section, we show how to use this metadata to run an inference.)

Run inference with TF Lite model

Now your TFLite model can be deployed and run using any of the supported inferencing libraries or with the new TFLite AudioClassifier Task API . The following code shows how you can run inference with the .tflite model in Python.

To observe how well the model performs with real samples, run the following code block over and over. Each time, it will fetch a new test sample and run inference with it, and you can listen to the audio sample below.

Download the TF Lite model

Now you can deploy the TF Lite model to your mobile or embedded device. You don't need to download the labels file because you can instead retrieve the labels from .tflite file metadata, as shown in the previous inferencing example.

Check out our end-to-end example apps that perform inferencing with TFLite audio models on Android and iOS .

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License , and code samples are licensed under the Apache 2.0 License . For details, see the Google Developers Site Policies . Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2024-04-21 UTC.

Transforming customer feedback: analyzing audio customer reviews with BigQuery ML’s speech-to-text

Nivedita kumari.

Data Analytics Customer Engineer

Michael Kilberry

Head of Product - AI/ML, Data Analytics

Try Gemini 1.5 models

Google's most advanced multimodal models in Vertex AI

BigQuery's integrated speech-to-text functionality offers a powerful tool for unlocking valuable insights hidden within audio data. This service transcribes audio files, such as customer review calls, into text format, making them ready for analysis within BigQuery's robust data platform. By combining speech-to-text with BigQuery's analytics capabilities, you can delve into customer sentiment, identify recurring product issues, and gain a better understanding of the voice of your customer.

BigQuery speech-to-text transforms audio data into actionable insights, offering potential benefits across industries and enabling a deeper understanding of customer interactions across multiple channels. You can also use BigQuery ML to leverage Gemini 1.0 Pro to gain additional insights & data formatting such as entity extraction and sentiment analysis to the text extracted from audio files using BigQuery ML’s native speech-to-text capability. Below are some use cases and the business value for specific industries:

Using advanced AI features such as BigQuery ML, you still have access to all of the built-in governance features of BigQuery, which give you the ability to have access control passthrough, so you can restrict insights from customer audio files based upon row-level security you have on your BigQuery Object Table. 

Ready to turn your audio data into insights? Let's dive into how you can use speech-to-text in BigQuery:

Imagine you have a collection of customer feedback calls stored as audio files in a Google Cloud Storage bucket. BigQuery's ML.TRANSCRIBE function, connected to a pre-trained speech-to-text model hosted on Google's Vertex AI platform, lets you automatically convert these audio files into readable text within BigQuery. Think of it as a specialized translator for audio data. You tell the ML.TRANSCRIBE function where your audio files are located (in your object table) and which speech-to-text model to use. It then handles the transcription process, using the power of machine learning, and delivers the text results directly into BigQuery. This makes it easy to analyze customer conversations alongside other business data.

Let's walk through the process together in BigQuery.

Setup instructions:

Before starting, choose your Google Cloud project, link a billing account, and enable the necessary API, full instructions here

Create a recognizer , a recognizer stores the configuration for speech recognition and is optional to create

Create a cloud resource connection and get the connection's service account, full guide here

Grant access to the service account by following the steps here . 

Create a dataset that will contain the model and the object table by following the steps here

Download and store the audio files in the Google Cloud Storage 

Download 5 audio files from here

Create a bucket in Google Cloud Storage and a folder within the bucket

Upload the downloaded audio files in the folder

Create a model

Create a remote model with a REMOTE_SERVICE_TYPE of CLOUD_AI_SPEECH_TO_TEXT_V2. A model makes the speech to text API available within BigQuery.

Example query:

Create an object table to reference the audio files

Sample code:

Please replace 'BUCKET_PATH'   with your Google Cloud Storage bucket/folder path where audio files are stored

Transcribe audio files using BigQuery ML

Sample query:

The results of ML.TRANSCRIBE include these columns:

transcripts: Contains the text transcription of the processed audio files

ml_transcribe_result: JSON value that contains the result from the Speech-to-Text API

ml_transcribe_status: Contains a string value that indicates the success or failure of the transcription process for each row. It will be empty if the process is successful

The object table columns

The ML.TRANSCRIBE function eliminates the need for manual transcription, saving time and effort. Transcribed text becomes easily searchable and analyzable within BigQuery, enabling you to extract valuable insights from your audio data.

Follow-up Ideas

Take the text extracted from the audio files, and use Gemini 1.0 Pro with BigQuery ML’s ML.generate_text function, to extract entities such as product names, stock prices, or other types of entity data you are looking to extract and structure them in JSON.

Use Gemini 1.0 Pro with BigQuery ML to measure sentiment analysis of the extracted text, and structure positive & negative sentiments in JSON.

Join customer feedback verbatims & sentiment scores with Customer Lifetime Total Value score or other relevant customer data to see how quantitative data & qualitative data relate to each other. 

Generate embeddings over the extracted text, and use vector search to search the audio files for specific content.

Curious to learn more? The official Google Cloud documentation on ML.TRANSCRIBE has all the details. Please also check out the blog on Gemini 1.0 Pro support for BigQuery ML to see other GenAI use cases as outlined in the Follow-up ideas.

• Data Analytics
• AI & Machine Learning
• Developers & Practitioners

Related articles

How to integrate Gemini and Sheets with BigQuery

By Karl Weinmeister • 6-minute read

Chugai Pharmaceutical: Accelerating drug discovery through AI, machine learning and data analysis

By Kento Tokuyama • 4-minute read

Helping marketers access data warehousing with Supermetrics Storage

By Tea Korpi • 6-minute read

Hands on with Gemini models in BigQuery: Decoding sentiment in customer reviews

By Nivedita Kumari • 7-minute read

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Welcome to the Purdue Online Writing Lab

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

The Online Writing Lab at Purdue University houses writing resources and instructional material, and we provide these as a free service of the Writing Lab at Purdue. Students, members of the community, and users worldwide will find information to assist with many writing projects. Teachers and trainers may use this material for in-class and out-of-class instruction.

The Purdue On-Campus Writing Lab and Purdue Online Writing Lab assist clients in their development as writers—no matter what their skill level—with on-campus consultations, online participation, and community engagement. The Purdue Writing Lab serves the Purdue, West Lafayette, campus and coordinates with local literacy initiatives. The Purdue OWL offers global support through online reference materials and services.

A Message From the Assistant Director of Content Development 

The Purdue OWL® is committed to supporting  students, instructors, and writers by offering a wide range of resources that are developed and revised with them in mind. To do this, the OWL team is always exploring possibilties for a better design, allowing accessibility and user experience to guide our process. As the OWL undergoes some changes, we welcome your feedback and suggestions by email at any time.

Please don't hesitate to contact us via our contact page  if you have any questions or comments.

All the best,

Social Media

Facebook twitter.