Transformers documentation

OmniASR CTC

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.23.0).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

This model was published in HF papers on 2025-11-12 and contributed to Hugging Face Transformers on 2026-11-16.

OmniASR CTC

FlashAttention SDPA

Overview

Check the OmniASR page for more information about this model family and checkpoints.

Usage

The CTC variant transcribes in lowercase without punctuation, and does not take a language hint.

Simple transcription

from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor

model_id = "hf-audio/omniasr-ctc-300m-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id, device_map="auto")

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))

inputs = processor(ds[0]["audio"]["array"], sampling_rate=processor.feature_extractor.sampling_rate)
inputs.to(model.device, dtype=model.dtype)
outputs = model.generate(**inputs)
print(processor.decode(outputs, skip_special_tokens=True)[0])

Batch inference

Batch inference is possible by passing a list of audios, which the processor pads to the longest one.

from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor

model_id = "hf-audio/omniasr-ctc-300m-v2"
NUM_SAMPLES = 5

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id, device_map="auto")

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el["array"] for el in ds["audio"][:NUM_SAMPLES]]

inputs = processor(speech_samples, sampling_rate=processor.feature_extractor.sampling_rate)
inputs.to(model.device, dtype=model.dtype)
outputs = model.generate(**inputs)
for i, text in enumerate(processor.decode(outputs, skip_special_tokens=True)):
    print(f"Audio {i + 1}: {text}")

Training

The model can be trained with the loss it outputs. Passing text to the processor prepares the CTC labels, with padding positions masked out. Note that OmniASR transcribes in lowercase, so the (uppercase) LibriSpeech transcripts are lowercased below.

from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor

model_id = "hf-audio/omniasr-ctc-300m-v2"
NUM_SAMPLES = 5

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id, device_map="auto")
model.train()

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el["array"] for el in ds["audio"][:NUM_SAMPLES]]
text_samples = [text.lower() for text in ds["text"][:NUM_SAMPLES]]

# passing `text` to the processor will prepare inputs' `labels` key
inputs = processor(audio=speech_samples, text=text_samples, sampling_rate=processor.feature_extractor.sampling_rate)
inputs.to(model.device, dtype=model.dtype)

outputs = model(**inputs)
print("Loss:", outputs.loss.item())
outputs.loss.backward()

Torch compile

The CTC variant runs a single forward pass, which is compiled by passing a CompileConfig to generate. Padding all inputs to the same length (here 30 seconds) keeps the input shapes static, so that new inputs don’t trigger a recompilation.

On an A100, we observed a speed-up of ~1.3 for a batch size of 4 (script).

import torch
from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor, CompileConfig

model_id = "hf-audio/omniasr-ctc-300m-v2"
NUM_SAMPLES = 4
num_warmup = 3

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el["array"] for el in ds["audio"][:NUM_SAMPLES]]

inputs = processor(
    speech_samples,
    sampling_rate=processor.feature_extractor.sampling_rate,
    padding="max_length",
    max_length=30 * processor.feature_extractor.sampling_rate,
).to(model.device, dtype=model.dtype)

compile_config = CompileConfig(fullgraph=True, mode="reduce-overhead")

# Warmup
for _ in range(num_warmup):
    _ = model.generate(**inputs, compile_config=compile_config)
torch.cuda.synchronize()

# Apply model
outputs = model.generate(**inputs, compile_config=compile_config)
print(processor.decode(outputs, skip_special_tokens=True)[0])

OmniASRCTCConfig

class transformers.OmniASRCTCConfig

< >

( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonevocab_size: int = 10288ctc_loss_reduction: str = 'mean'ctc_zero_infinity: bool = Falseinitializer_range: float = 0.02audio_config: dict | transformers.configuration_utils.PreTrainedConfig | None = Nonebos_token_id: int | None = 0pad_token_id: int | None = 1eos_token_id: int | None = 2 )

Parameters

  • vocab_size (int, optional, defaults to 10288) — Vocabulary size of the model. Defines the number of different tokens that can be represented by the input_ids.
  • ctc_loss_reduction (str, optional, defaults to "mean") — Specifies the reduction to apply to the output of torch.nn.CTCLoss. Only relevant when training an instance of OmniASRForCTC.
  • ctc_zero_infinity (bool, optional, defaults to False) — Whether to zero infinite losses and the associated gradients of torch.nn.CTCLoss. Infinite losses mainly occur when the inputs are too short to be aligned to the targets. Only relevant when training an instance of OmniASRForCTC.
  • initializer_range (float, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
  • audio_config (Union[dict, PreTrainedConfig], optional) — The config object or dictionary of the audio encoder. Defaults to OmniASRAudioConfig.
  • bos_token_id (int, optional, defaults to 0) — Token id used for beginning-of-stream in the vocabulary.
  • pad_token_id (int, optional, defaults to 1) — Token id used for padding in the vocabulary.
  • eos_token_id (int, optional, defaults to 2) — Token id used for end-of-stream in the vocabulary.

This is the configuration class to store the configuration of a OmniASRCTCModel. It is used to instantiate a Omniasr Ctc model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the hf-audio/omniasr-ctc-300m-v2

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

Example:

>>> from transformers import OmniASRForCTC, OmniASRCTCConfig

>>> # Initializing an OmniASR-CTC configuration
>>> configuration = OmniASRCTCConfig()

>>> # Initializing a model (with random weights) from the configuration
>>> model = OmniASRForCTC(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config

OmniASRCTCProcessor

class transformers.OmniASRCTCProcessor

< >

( feature_extractortokenizer )

Parameters

  • feature_extractor (OmniASRFeatureExtractor) — The feature extractor is a required input.
  • tokenizer (ParakeetTokenizer) — The tokenizer is a required input.

Constructs a OmniASRCTCProcessor which wraps a feature extractor and a tokenizer into a single processor.

OmniASRCTCProcessor offers all the functionalities of OmniASRFeatureExtractor and ParakeetTokenizer. See the ~OmniASRFeatureExtractor and ~ParakeetTokenizer for more information.

OmniASRCTCModel

class transformers.OmniASRCTCModel

< >

( config: OmniASRCTCConfig )

Parameters

  • config (OmniASRCTCConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The OmniASR CTC model, i.e. the speech encoder without the CTC head.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< >

( input_values: Tensorpadding_mask: typing.Optional[torch.Tensor] = None**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → BaseModelOutputWithPooling or tuple(torch.Tensor)

Parameters

  • input_values (torch.Tensor of shape (batch_size, sequence_length)) — Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file into an array of type list[float], a numpy.ndarray or a torch.Tensor, e.g. via the torchcodec library (pip install torchcodec) or the soundfile library (pip install soundfile). To prepare the array into input_values, the AutoProcessor should be used for padding and conversion into a tensor of type torch.Tensor. See OmniASRCTCProcessor.__call__() for details.
  • padding_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Padding mask used to pad input_values.

Returns

BaseModelOutputWithPooling or tuple(torch.Tensor)

A BaseModelOutputWithPooling or a tuple of torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (OmniASRCTCConfig) and inputs.

The OmniASRCTCModel forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

  • last_hidden_state (torch.Tensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.

  • pooler_output (torch.Tensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

  • hidden_states (tuple(torch.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple(torch.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Example:

>>> from transformers import AutoProcessor, OmniASRCTCModel
>>> from datasets import load_dataset
>>> import torch

>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate

>>> processor = AutoProcessor.from_pretrained("hf-audio/omniasr-ctc-300m-v2")
>>> model = OmniASRCTCModel.from_pretrained("hf-audio/omniasr-ctc-300m-v2")

>>> # audio file is decoded on the fly
>>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_ids = torch.argmax(logits, dim=-1)

>>> # transcribe speech
>>> transcription = processor.batch_decode(predicted_ids)
>>> transcription[0]
...

>>> inputs["labels"] = processor(text=dataset[0]["text"], return_tensors="pt").input_ids

>>> # compute loss
>>> loss = model(**inputs).loss
>>> round(loss.item(), 2)
...

OmniASRForCTC

class transformers.OmniASRForCTC

< >

( config: OmniASRCTCConfig )

Parameters

  • config (OmniASRCTCConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

OmniASR Encoder with a Connectionist Temporal Classification (CTC) head.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< >

( input_values: Tensorpadding_mask: typing.Optional[torch.Tensor] = Nonelabels: typing.Optional[torch.Tensor] = None**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → CausalLMOutput or tuple(torch.Tensor)

Parameters

  • input_values (torch.Tensor of shape (batch_size, sequence_length)) — Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file into an array of type list[float], a numpy.ndarray or a torch.Tensor, e.g. via the torchcodec library (pip install torchcodec) or the soundfile library (pip install soundfile). To prepare the array into input_values, the AutoProcessor should be used for padding and conversion into a tensor of type torch.Tensor. See processor_class.__call__ for details.
  • padding_mask (torch.Tensor of shape (batch_size, 1, sequence_length)) — Padding mask used to pad input_values.
  • labels (torch.Tensor of shape (batch_size, sequence_length), optional) — Labels for computing the masked language modeling loss. Indices should either be in [0, ..., config.vocab_size] or -100 (see input_ids docstring). Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size].

Returns

CausalLMOutput or tuple(torch.Tensor)

A CausalLMOutput or a tuple of torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (None) and inputs.

The OmniASRForCTC forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

  • loss (torch.Tensor of shape (1,), optional, returned when labels is provided) — Language modeling loss (for next-token prediction).

  • logits (torch.Tensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

  • hidden_states (tuple(torch.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple(torch.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Example:

>>> from transformers import AutoProcessor, OmniASRForCTC
>>> from datasets import load_dataset, Audio

>>> model_id = "hf-audio/omniasr-ctc-300m-v2"
>>> processor = AutoProcessor.from_pretrained(model_id)
>>> model = OmniASRForCTC.from_pretrained(model_id)

>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))

>>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"])
>>> outputs = model(**inputs)

>>> print(outputs.loss)
Update on GitHub