Transformers documentation
OmniASR CTC
This model was published in HF papers on 2025-11-12 and contributed to Hugging Face Transformers on 2026-11-16.
OmniASR CTC
Overview
Check the OmniASR page for more information about this model family and checkpoints.
Usage
The CTC variant transcribes in lowercase without punctuation, and does not take a language hint.
Simple transcription
from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor
model_id = "hf-audio/omniasr-ctc-300m-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id, device_map="auto")
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
inputs = processor(ds[0]["audio"]["array"], sampling_rate=processor.feature_extractor.sampling_rate)
inputs.to(model.device, dtype=model.dtype)
outputs = model.generate(**inputs)
print(processor.decode(outputs, skip_special_tokens=True)[0])Batch inference
Batch inference is possible by passing a list of audios, which the processor pads to the longest one.
from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor
model_id = "hf-audio/omniasr-ctc-300m-v2"
NUM_SAMPLES = 5
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id, device_map="auto")
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el["array"] for el in ds["audio"][:NUM_SAMPLES]]
inputs = processor(speech_samples, sampling_rate=processor.feature_extractor.sampling_rate)
inputs.to(model.device, dtype=model.dtype)
outputs = model.generate(**inputs)
for i, text in enumerate(processor.decode(outputs, skip_special_tokens=True)):
print(f"Audio {i + 1}: {text}")Training
The model can be trained with the loss it outputs. Passing text to the processor prepares the CTC labels, with
padding positions masked out. Note that OmniASR transcribes in lowercase, so the (uppercase) LibriSpeech transcripts
are lowercased below.
from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor
model_id = "hf-audio/omniasr-ctc-300m-v2"
NUM_SAMPLES = 5
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id, device_map="auto")
model.train()
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el["array"] for el in ds["audio"][:NUM_SAMPLES]]
text_samples = [text.lower() for text in ds["text"][:NUM_SAMPLES]]
# passing `text` to the processor will prepare inputs' `labels` key
inputs = processor(audio=speech_samples, text=text_samples, sampling_rate=processor.feature_extractor.sampling_rate)
inputs.to(model.device, dtype=model.dtype)
outputs = model(**inputs)
print("Loss:", outputs.loss.item())
outputs.loss.backward()Torch compile
The CTC variant runs a single forward pass, which is compiled by passing a CompileConfig to generate. Padding all
inputs to the same length (here 30 seconds) keeps the input shapes static, so that new inputs don’t trigger a
recompilation.
On an A100, we observed a speed-up of ~1.3 for a batch size of 4 (script).
import torch
from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor, CompileConfig
model_id = "hf-audio/omniasr-ctc-300m-v2"
NUM_SAMPLES = 4
num_warmup = 3
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el["array"] for el in ds["audio"][:NUM_SAMPLES]]
inputs = processor(
speech_samples,
sampling_rate=processor.feature_extractor.sampling_rate,
padding="max_length",
max_length=30 * processor.feature_extractor.sampling_rate,
).to(model.device, dtype=model.dtype)
compile_config = CompileConfig(fullgraph=True, mode="reduce-overhead")
# Warmup
for _ in range(num_warmup):
_ = model.generate(**inputs, compile_config=compile_config)
torch.cuda.synchronize()
# Apply model
outputs = model.generate(**inputs, compile_config=compile_config)
print(processor.decode(outputs, skip_special_tokens=True)[0])OmniASRCTCConfig
class transformers.OmniASRCTCConfig
< source >( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonevocab_size: int = 10288ctc_loss_reduction: str = 'mean'ctc_zero_infinity: bool = Falseinitializer_range: float = 0.02audio_config: dict | transformers.configuration_utils.PreTrainedConfig | None = Nonebos_token_id: int | None = 0pad_token_id: int | None = 1eos_token_id: int | None = 2 )
Parameters
- vocab_size (
int, optional, defaults to10288) — Vocabulary size of the model. Defines the number of different tokens that can be represented by theinput_ids. - ctc_loss_reduction (
str, optional, defaults to"mean") — Specifies the reduction to apply to the output oftorch.nn.CTCLoss. Only relevant when training an instance of OmniASRForCTC. - ctc_zero_infinity (
bool, optional, defaults toFalse) — Whether to zero infinite losses and the associated gradients oftorch.nn.CTCLoss. Infinite losses mainly occur when the inputs are too short to be aligned to the targets. Only relevant when training an instance of OmniASRForCTC. - initializer_range (
float, optional, defaults to0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices. - audio_config (
Union[dict, PreTrainedConfig], optional) — The config object or dictionary of the audio encoder. Defaults to OmniASRAudioConfig. - bos_token_id (
int, optional, defaults to0) — Token id used for beginning-of-stream in the vocabulary. - pad_token_id (
int, optional, defaults to1) — Token id used for padding in the vocabulary. - eos_token_id (
int, optional, defaults to2) — Token id used for end-of-stream in the vocabulary.
This is the configuration class to store the configuration of a OmniASRCTCModel. It is used to instantiate a Omniasr Ctc model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the hf-audio/omniasr-ctc-300m-v2
Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.
Example:
>>> from transformers import OmniASRForCTC, OmniASRCTCConfig
>>> # Initializing an OmniASR-CTC configuration
>>> configuration = OmniASRCTCConfig()
>>> # Initializing a model (with random weights) from the configuration
>>> model = OmniASRForCTC(configuration)
>>> # Accessing the model configuration
>>> configuration = model.configOmniASRCTCProcessor
class transformers.OmniASRCTCProcessor
< source >( feature_extractortokenizer )
Constructs a OmniASRCTCProcessor which wraps a feature extractor and a tokenizer into a single processor.
OmniASRCTCProcessor offers all the functionalities of OmniASRFeatureExtractor and ParakeetTokenizer. See the ~OmniASRFeatureExtractor and ~ParakeetTokenizer for more information.
OmniASRCTCModel
class transformers.OmniASRCTCModel
< source >( config: OmniASRCTCConfig )
Parameters
- config (OmniASRCTCConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.
The OmniASR CTC model, i.e. the speech encoder without the CTC head.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)
This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
forward
< source >( input_values: Tensorpadding_mask: typing.Optional[torch.Tensor] = None**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → BaseModelOutputWithPooling or tuple(torch.Tensor)
Parameters
- input_values (
torch.Tensorof shape(batch_size, sequence_length)) — Float values of input raw speech waveform. Values can be obtained by loading a.flacor.wavaudio file into an array of typelist[float], anumpy.ndarrayor atorch.Tensor, e.g. via the torchcodec library (pip install torchcodec) or the soundfile library (pip install soundfile). To prepare the array intoinput_values, the AutoProcessor should be used for padding and conversion into a tensor of typetorch.Tensor. SeeOmniASRCTCProcessor.__call__()for details. - padding_mask (
torch.Tensorof shape(batch_size, sequence_length), optional) — Padding mask used to padinput_values.
Returns
BaseModelOutputWithPooling or tuple(torch.Tensor)
A BaseModelOutputWithPooling or a tuple of
torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (OmniASRCTCConfig) and inputs.
The OmniASRCTCModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
last_hidden_state (
torch.Tensorof shape(batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.pooler_output (
torch.Tensorof shape(batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.hidden_states (
tuple(torch.Tensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.Tensor(one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (
tuple(torch.Tensor), optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.Tensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
Example:
>>> from transformers import AutoProcessor, OmniASRCTCModel
>>> from datasets import load_dataset
>>> import torch
>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> processor = AutoProcessor.from_pretrained("hf-audio/omniasr-ctc-300m-v2")
>>> model = OmniASRCTCModel.from_pretrained("hf-audio/omniasr-ctc-300m-v2")
>>> # audio file is decoded on the fly
>>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> with torch.no_grad():
... logits = model(**inputs).logits
>>> predicted_ids = torch.argmax(logits, dim=-1)
>>> # transcribe speech
>>> transcription = processor.batch_decode(predicted_ids)
>>> transcription[0]
...
>>> inputs["labels"] = processor(text=dataset[0]["text"], return_tensors="pt").input_ids
>>> # compute loss
>>> loss = model(**inputs).loss
>>> round(loss.item(), 2)
...OmniASRForCTC
class transformers.OmniASRForCTC
< source >( config: OmniASRCTCConfig )
Parameters
- config (OmniASRCTCConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.
OmniASR Encoder with a Connectionist Temporal Classification (CTC) head.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)
This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
forward
< source >( input_values: Tensorpadding_mask: typing.Optional[torch.Tensor] = Nonelabels: typing.Optional[torch.Tensor] = None**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → CausalLMOutput or tuple(torch.Tensor)
Parameters
- input_values (
torch.Tensorof shape(batch_size, sequence_length)) — Float values of input raw speech waveform. Values can be obtained by loading a.flacor.wavaudio file into an array of typelist[float], anumpy.ndarrayor atorch.Tensor, e.g. via the torchcodec library (pip install torchcodec) or the soundfile library (pip install soundfile). To prepare the array intoinput_values, the AutoProcessor should be used for padding and conversion into a tensor of typetorch.Tensor. Seeprocessor_class.__call__for details. - padding_mask (
torch.Tensorof shape(batch_size, 1, sequence_length)) — Padding mask used to padinput_values. - labels (
torch.Tensorof shape(batch_size, sequence_length), optional) — Labels for computing the masked language modeling loss. Indices should either be in[0, ..., config.vocab_size]or -100 (seeinput_idsdocstring). Tokens with indices set to-100are ignored (masked), the loss is only computed for the tokens with labels in[0, ..., config.vocab_size].
Returns
CausalLMOutput or tuple(torch.Tensor)
A CausalLMOutput or a tuple of
torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (None) and inputs.
The OmniASRForCTC forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
loss (
torch.Tensorof shape(1,), optional, returned whenlabelsis provided) — Language modeling loss (for next-token prediction).logits (
torch.Tensorof shape(batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).hidden_states (
tuple(torch.Tensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.Tensor(one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (
tuple(torch.Tensor), optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.Tensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
Example:
>>> from transformers import AutoProcessor, OmniASRForCTC
>>> from datasets import load_dataset, Audio
>>> model_id = "hf-audio/omniasr-ctc-300m-v2"
>>> processor = AutoProcessor.from_pretrained(model_id)
>>> model = OmniASRForCTC.from_pretrained(model_id)
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
>>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"])
>>> outputs = model(**inputs)
>>> print(outputs.loss)