Transformers documentation

OmniASR

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.23.0).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

This model was published in HF papers on 2025-11-12 and contributed to Hugging Face Transformers on 2026-11-16.

OmniASR

FlashAttention SDPA

Overview

OmniASR (Omnilingual ASR) is a suite of multilingual speech-recognition models from Meta FAIR, introduced in Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages, that covers more than 1,600 languages — including hundreds never previously supported by any ASR system. Each model pairs a Wav2Vec2-style audio encoder (convolutional layers that downsample the 16 kHz waveform ~320× to a 50 Hz frame rate, followed by a pre-norm Transformer encoder) with one of two heads:

  • CTC variant (OmniASRForCTC, see OmniASR CTC): a single linear projection to the vocabulary, decoded non-autoregressively with greedy CTC.
  • ALM (audio language model) variant (OmniASRForConditionalGeneration, this page): the audio embeddings are linearly projected into a Llama decoder which autoregressively generates the transcription. This variant additionally supports optional language conditioning: passing a language code such as "eng_Latn" makes the processor write the matching language token into the decoder context, which generally improves transcription quality.

Checkpoints can be found in this collection: 4x ALM and 4x CTC checkpoints of various sizes (300M, 1B, 3B, 7B). Original weights are released by Meta, and can be found on their GitHub repo.

This model was contributed by Eric Bezzam.

Usage

The simplest way to transcribe audio is with apply_transcription_request, which handles the chat template formatting for you, namely it is a convenience wrapper for apply_chat_template (see Chat template below).

Simple transcription

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "hf-audio/omniasr-llm-300m-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
)
inputs.to(model.device, dtype=model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
print(processor.decode(generated_ids, skip_special_tokens=True)[0])

Language hint

Without a language, the model runs in its language-agnostic mode. Passing a language code (e.g. "eng_Latn", "fra_Latn", "cmn_Hans") writes the matching language token into the decoder prompt, which generally improves transcription quality. The list of supported languages and naming convention follows the original here. For the languages that have one, the more common ISO 639-1 codes (e.g. "en", "fr", "zh") are also accepted.

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "hf-audio/omniasr-llm-300m-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

audio = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav"

# Without language hint (language-agnostic)
inputs = processor.apply_transcription_request(audio=audio).to(model.device, dtype=model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
print(f"Language-agnostic: {processor.decode(generated_ids, skip_special_tokens=True)[0]}")

# With language hint
inputs = processor.apply_transcription_request(audio=audio, language="zh").to(model.device, dtype=model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
print(f"Language hint:     {processor.decode(generated_ids, skip_special_tokens=True)[0]}")

Batch inference

Batch inference is possible by passing a list of audios and, if provided, a single language for the whole batch or a list of languages (one per audio, None for the language-agnostic mode).

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "hf-audio/omniasr-llm-300m-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

audio = [
    "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
    "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav",
]

inputs = processor.apply_transcription_request(audio, language=["en", None])
inputs.to(model.device, dtype=model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
for i, text in enumerate(processor.decode(generated_ids, skip_special_tokens=True)):
    print(f"Audio {i + 1}: {text}")

Chat template

The model also accepts chat template inputs. The apply_transcription_request usage above is a convenience wrapper for apply_chat_template. The language hint is given as a {"type": "language", ...} item of the user turn; leaving it out selects the language-agnostic mode.

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "hf-audio/omniasr-llm-300m-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

conversations = [
    [
        {
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
                },
                {"type": "language", "language": "en"},
            ],
        },
    ],
    [
        {
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav",
                },
                {"type": "language", "language": "zh"},
            ],
        },
    ],
]

inputs = processor.apply_chat_template(
    conversations, tokenize=True, add_generation_prompt=True, return_dict=True
)
inputs.to(model.device, dtype=model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
for i, text in enumerate(processor.decode(generated_ids, skip_special_tokens=True)):
    print(f"Audio {i + 1}: {text}")

Training

OmniASR can be trained with the loss outputted by the model. Put the target transcript in the assistant turn of the chat template and pass output_labels=True. Audio and padding positions are masked automatically.

Note that OmniASR transcribes in lowercase, so the (uppercase) LibriSpeech transcripts are lowercased below.

from datasets import Audio, load_dataset
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "hf-audio/omniasr-llm-300m-v2"
NUM_SAMPLES = 2

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
model.train()

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
conversation = [
    [
        {
            "role": "user",
            "content": [
                {"type": "audio", "audio": sample["audio"]["array"]},
                # any supported language code, or no `language` item to train the language-agnostic mode
                {"type": "language", "language": "en"},
            ],
        },
        {"role": "assistant", "content": [{"type": "text", "text": sample["text"].lower()}]},
    ]
    for sample in ds.select(range(NUM_SAMPLES))
]

inputs = processor.apply_chat_template(
    conversation,
    tokenize=True,
    return_dict=True,
    # only train on the assistant outputs: mask the labels outside the assistant turns
    return_assistant_tokens_mask=True,
    return_tensors="pt",
    processor_kwargs={"output_labels": True},
).to(model.device, dtype=model.dtype)
inputs["labels"][inputs.pop("assistant_masks") == 0] = -100

loss = model(**inputs).loss
print("Loss:", loss.item())
loss.backward()

Torch compile

For autoregressive transcription, torch.compile accelerates the per-token forward passes inside generate by providing a CompileConfig object, which requires a static cache.

Note that the StaticCache is created once and reused across generate calls to avoid recompiles.

On an A100, we observed a speed-up of ~2.3 for a batch size of 4 (script).

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor, CompileConfig, StaticCache

model_id = "hf-audio/omniasr-llm-300m-v2"
NUM_SAMPLES = 4
num_warmup = 3
max_new_tokens = 256

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

audio_url = "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav"
inputs = processor.apply_transcription_request(
    audio=[audio_url] * NUM_SAMPLES,
    language="en",
).to(model.device, dtype=model.dtype)

# Created once and reused across `generate` calls, to avoid recompiling at every call
static_cache = StaticCache(
    config=model.config,
    max_cache_len=inputs["input_ids"].shape[1] + max_new_tokens,
)
compile_config = CompileConfig()

# Warmup
with torch.inference_mode():
    for _ in range(num_warmup):
        static_cache.reset()
        _ = model.generate(
            **inputs, max_new_tokens=max_new_tokens, do_sample=False,
            cache=static_cache, compile_config=compile_config,
        )
torch.cuda.synchronize()

# Apply model
with torch.inference_mode():
    static_cache.reset()
    output_ids = model.generate(
        **inputs, max_new_tokens=max_new_tokens, do_sample=False,
        cache=static_cache, compile_config=compile_config,
    )
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
print(processor.decode(generated_ids, skip_special_tokens=True)[0])

OmniASRAudioConfig

class transformers.OmniASRAudioConfig

< >

( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonehidden_size: int = 1024conv_dim: list[int] | tuple[int, ...] = (512, 512, 512, 512, 512, 512, 512)conv_kernel: list[int] | tuple[int, ...] = (10, 3, 3, 3, 3, 2, 2)conv_stride: list[int] | tuple[int, ...] = (5, 2, 2, 2, 2, 2, 2)conv_bias: bool = Truenum_attention_heads: int = 16num_hidden_layers: int = 24num_conv_pos_embeddings: int = 128num_conv_pos_embedding_groups: int = 16intermediate_size: int = 4096attention_dropout: float | int = 0.0hidden_dropout: float | int = 0.1layerdrop: float | int = 0.1activation_dropout: float | int = 0.1initializer_range: float = 0.02layer_norm_eps: float = 1e-05hidden_act: str = 'gelu' )

Parameters

  • hidden_size (int, optional, defaults to 1024) — Dimension of the hidden representations.
  • conv_dim (tuple[int] or list[int], optional, defaults to (512, 512, 512, 512, 512, 512, 512)) — A tuple of integers defining the number of input and output channels of each 1D convolutional layer in the feature encoder. The length of conv_dim defines the number of 1D convolutional layers.
  • conv_kernel (tuple[int] or list[int], optional, defaults to (10, 3, 3, 3, 3, 2, 2)) — A tuple of integers defining the kernel size of each 1D convolutional layer in the feature encoder. The length of conv_kernel defines the number of convolutional layers and has to match the length of conv_dim.
  • conv_stride (tuple[int] or list[int], optional, defaults to (5, 2, 2, 2, 2, 2, 2)) — A tuple of integers defining the stride of each 1D convolutional layer in the feature encoder. The length of conv_stride defines the number of convolutional layers and has to match the length of conv_dim.
  • conv_bias (bool, optional, defaults to True) — Whether the 1D convolutional layers have a bias.
  • num_attention_heads (int, optional, defaults to 16) — Number of attention heads for each attention layer in the Transformer decoder.
  • num_hidden_layers (int, optional, defaults to 24) — Number of hidden layers in the Transformer decoder.
  • num_conv_pos_embeddings (int, optional, defaults to 128) — Number of convolutional positional embeddings. Defines the kernel size of the 1D convolutional positional embeddings layer.
  • num_conv_pos_embedding_groups (int, optional, defaults to 16) — Number of groups of the 1D convolutional positional embeddings layer.
  • intermediate_size (int, optional, defaults to 4096) — Dimension of the MLP representations.
  • attention_dropout (Union[float, int], optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
  • hidden_dropout (Union[float, int], optional, defaults to 0.1) — The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
  • layerdrop (Union[float, int], optional, defaults to 0.1) — The LayerDrop probability. See the [LayerDrop paper](see https://huggingface.co/papers/1909.11556) for more details.
  • activation_dropout (Union[float, int], optional, defaults to 0.1) — The dropout ratio for activations inside the fully connected layer.
  • initializer_range (float, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
  • layer_norm_eps (float, optional, defaults to 1e-05) — The epsilon used by the layer normalization layers.
  • hidden_act (str, optional, defaults to gelu) — The non-linear activation function (function or string) in the decoder. For example, "gelu", "relu", "silu", etc.

This is the configuration class to store the configuration of a OmniASRModel. It is used to instantiate a Omniasr model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the hf-audio/omniasr-ctc-300m-v2

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

Example:

>>> from transformers import OmniASRAudioConfig, OmniASRAudioModel

>>> # Initializing an OmniASR encoder configuration
>>> configuration = OmniASRAudioConfig()

>>> # Initializing a model (with random weights) from the configuration
>>> model = OmniASRAudioModel(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config

OmniASRConfig

class transformers.OmniASRConfig

< >

( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Noneaudio_config: dict | transformers.configuration_utils.PreTrainedConfig | None = Nonetext_config: dict | transformers.configuration_utils.PreTrainedConfig | None = Noneaudio_token_id: int = 10289initializer_range: float = 0.02bos_token_id: int | None = 0pad_token_id: int | None = 1eos_token_id: int | None = 2 )

Parameters

  • audio_config (Union[dict, ~configuration_utils.PreTrainedConfig], optional) — The config object or dictionary of the audio backbone.
  • text_config (Union[dict, ~configuration_utils.PreTrainedConfig], optional) — The config object or dictionary of the text backbone.
  • audio_token_id (int, optional, defaults to 10289) — The audio token index used as a placeholder for input audio.
  • initializer_range (float, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
  • bos_token_id (int, optional, defaults to 0) — Token id used for beginning-of-stream in the vocabulary.
  • pad_token_id (int, optional, defaults to 1) — Token id used for padding in the vocabulary.
  • eos_token_id (int, optional, defaults to 2) — Token id used for end-of-stream in the vocabulary.

This is the configuration class to store the configuration of a OmniASRModel. It is used to instantiate a Omniasr model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the hf-audio/omniasr-llm-300m-v2

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

Example:

>>> from transformers import OmniASRForConditionalGeneration, OmniASRConfig

>>> # Initializing an OmniASR-LLM configuration
>>> configuration = OmniASRConfig()

>>> # Initializing a model (with random weights) from the configuration
>>> model = OmniASRForConditionalGeneration(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config

OmniASRFeatureExtractor

class transformers.OmniASRFeatureExtractor

< >

( feature_size = 1sampling_rate = 16000padding_value = 0.0return_attention_mask = Truedo_normalize = True**kwargs )

Parameters

  • feature_size (int, optional, defaults to 1) — The feature dimension of the extracted features.
  • sampling_rate (int, optional, defaults to 16000) — The sampling rate at which the audio files should be digitalized expressed in hertz (Hz).
  • padding_value (float, optional, defaults to 0.0) — The value that is used to fill the padding values.
  • return_attention_mask (bool, optional, defaults to True) — Whether or not __call__() should return attention_mask. OmniASR needs the mask to know which frames are padding, so batched inputs are only encoded correctly when it is returned.
  • do_normalize (bool, optional, defaults to True) — Whether or not to normalize each audio sample to zero mean and unit variance. The statistics are computed per sample, over its non-padded samples, to match the original implementation.

Constructs an OmniASR feature extractor.

OmniASRProcessor

class transformers.OmniASRProcessor

< >

( feature_extractortokenizerchat_template = Noneaudio_token = '<extra_id_1>'conv_kernel = Noneconv_stride = None )

Parameters

  • feature_extractor (OmniASRFeatureExtractor) — The feature extractor is a required input.
  • tokenizer (TokenizersBackend) — The tokenizer is a required input.
  • chat_template (str) — A Jinja template to convert lists of messages in a chat into a tokenizable string.
  • audio_token (str, optional, defaults to "<extra_id_1>") — The placeholder token that stands for one speech encoder frame in the prompt, i.e. the token of OmniASRConfig.audio_token_id().
  • conv_kernel (list[int], optional) — Kernel size of each convolution of the speech encoder’s feature encoder, i.e. OmniASRAudioConfig.conv_kernel(). Needed to count the frames an audio input is subsampled to, and therefore how many audio placeholders its prompt holds. Defaults to [10, 3, 3, 3, 3, 2, 2].
  • conv_stride (list[int], optional) — Stride of each convolution of the speech encoder’s feature encoder, i.e. OmniASRAudioConfig.conv_stride(). Defaults to [5, 2, 2, 2, 2, 2, 2].

Constructs a OmniASRProcessor which wraps a feature extractor and a tokenizer into a single processor.

OmniASRProcessor offers all the functionalities of OmniASRFeatureExtractor and TokenizersBackend. See the ~OmniASRFeatureExtractor and ~TokenizersBackend for more information.

apply_chat_template

< >

( conversation*args**kwargs )

Same as apply_chat_template(), but resolves the language codes of the conversation first. The chat template writes the language token as is, so this override is needed to accept ISO 639-1 codes (e.g. "en" for "eng_Latn"), and to raise on unsupported codes instead of silently writing an unknown token.

apply_transcription_request

< >

( audio: typing.Union[str, list[str], numpy.ndarray, ForwardRef('torch.Tensor'), collections.abc.Sequence[numpy.ndarray], collections.abc.Sequence['torch.Tensor']]language: str | list[str] | None = Nonetokenize: bool = True**kwargs: *<class 'transformers.processing_utils.ProcessingKwargs'> ) → BatchFeature or list[str]

Parameters

  • audio (str, list[str], np.ndarray, torch.Tensor, list[np.ndarray], list[torch.Tensor]) — Audio to transcribe. Strings are interpreted as local paths or URLs and will be loaded automatically by the chat template loader; NumPy arrays and PyTorch tensors are forwarded directly.
  • language (str or list[str], optional) — Language code(s) (e.g. "eng_Latn" or ["eng_Latn", "fra_Latn"]), either one for the whole batch or one per audio. ISO 639-1 codes (e.g. "en") are also accepted for the languages that have one. None or "auto" selects the model’s language-agnostic mode; naming the language explicitly gives better transcription quality.
  • tokenize (bool, optional, defaults to True) — Whether to tokenize the prompts and process the audio. If False, the audio is not loaded and the prompts are returned as strings with unexpanded audio placeholders. Passing them with the audio to the processor, processor(text=prompts, audio=audio), gives the same outputs as tokenize=True.
  • **kwargs — Additional keyword arguments forwarded to apply_chat_template(). Processor kwargs (for example text_kwargs, audio_kwargs, …) should be passed as processor_kwargs={...}.

Returns

BatchFeature or list[str]

One prompt per audio when tokenize=False, otherwise processor outputs ready to be passed to OmniASRForConditionalGeneration.generate().

Prepare inputs for speech recognition without manually writing the chat template.

OmniASRAudioModel

class transformers.OmniASRAudioModel

< >

( config: OmniASRAudioConfig )

Parameters

  • config (OmniASRAudioConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The OmniASR speech encoder, which is a Wav2Vec2-style encoder.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< >

( input_values: typing.Optional[torch.Tensor]padding_mask: typing.Optional[torch.Tensor] = Noneoutput_attention_mask: bool = True**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → OmniASREncoderModelOutput or tuple(torch.Tensor)

Parameters

  • input_values (torch.Tensor of shape (batch_size, sequence_length), optional) — Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file into an array of type list[float], a numpy.ndarray or a torch.Tensor, e.g. via the torchcodec library (pip install torchcodec) or the soundfile library (pip install soundfile). To prepare the array into input_values, the AutoProcessor should be used for padding and conversion into a tensor of type torch.Tensor. See processor_class.__call__ for details.
  • padding_mask (torch.Tensor of shape (batch_size, 1, sequence_length)) — Padding mask used to pad input_values.
  • output_attention_mask (bool, optional, defaults to True) — Whether to return the subsampled attention mask. Only effective when padding_mask is provided.

Returns

OmniASREncoderModelOutput or tuple(torch.Tensor)

A OmniASREncoderModelOutput or a tuple of torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (None) and inputs.

The OmniASRAudioModel forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

  • last_hidden_state (torch.Tensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.

  • pooler_output (torch.Tensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

  • hidden_states (tuple(torch.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple(torch.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

  • attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices after sequence compression. Returned because the sequence length may differ from the input sequence length. Mask values selected in [0, 1]:

    • 1 for tokens that are not masked,
    • 0 for tokens that are masked.

OmniASRModel

class transformers.OmniASRModel

< >

( config: OmniASRConfig )

Parameters

  • config (OmniASRConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The OmniASR model, which consists of a Wav2Vec2 encoder, a multi-modal projector and a LLama language model, without a language modeling head.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< >

( input_ids: typing.Optional[torch.LongTensor] = Noneinputs_embeds: typing.Optional[torch.Tensor] = Noneinput_values: typing.Optional[torch.Tensor] = Nonepadding_mask: typing.Optional[torch.Tensor] = None**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → OmniASRBaseModelOutputWithPast or tuple(torch.Tensor)

Parameters

  • input_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

    Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

    What are input IDs?

  • inputs_embeds (torch.Tensor of shape (batch_size, sequence_length, hidden_size), optional) — Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
  • input_values (torch.Tensor of shape (batch_size, sequence_length), optional) — Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file into an array of type list[float], a numpy.ndarray or a torch.Tensor, e.g. via the torchcodec library (pip install torchcodec) or the soundfile library (pip install soundfile). To prepare the array into input_values, the AutoProcessor should be used for padding and conversion into a tensor of type torch.Tensor. See OmniASRProcessor.__call__() for details.
  • padding_mask (torch.Tensor of shape (batch_size, num_samples), optional) — Padding mask to remove padded parts of audio.

Returns

OmniASRBaseModelOutputWithPast or tuple(torch.Tensor)

A OmniASRBaseModelOutputWithPast or a tuple of torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (OmniASRConfig) and inputs.

The OmniASRModel forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

  • last_hidden_state (torch.Tensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.

    If cache is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.

  • cache (Cache, optional, returned when use_cache=True is passed or when config.use_cache=True) — It is a Cache instance. For more details, see our kv cache guide.

    Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if config.is_encoder_decoder=True in the cross-attention blocks) that can be used (see cache input) to speed up sequential decoding.

  • hidden_states (tuple(torch.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple(torch.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

  • audio_features (torch.Tensor of shape (num_audio_frames, hidden_size), optional) — Encoder output frames, before the multi-modal projector, for the non-padded audio frames.

OmniASRForConditionalGeneration

class transformers.OmniASRForConditionalGeneration

< >

( config: OmniASRConfig )

Parameters

  • config (OmniASRConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

OmniASR model, which consists of a Wav2Vec2 encoder, a multi-modal projector and a LLama language model.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< >

( input_ids: typing.Optional[torch.LongTensor] = Noneinput_values: typing.Optional[torch.Tensor] = Nonepadding_mask: typing.Optional[torch.Tensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneposition_ids: typing.Optional[torch.LongTensor] = Nonecache: transformers.cache_utils.Cache | None = Noneinputs_embeds: typing.Optional[torch.FloatTensor] = Nonelabels: typing.Optional[torch.Tensor] = Noneuse_cache: bool | None = Nonelogits_to_keep: typing.Union[int, torch.Tensor] = 0**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → CausalLMOutputWithPast or tuple(torch.Tensor)

Parameters

  • input_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

    Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

    What are input IDs?

  • input_values (torch.Tensor of shape (batch_size, 1, num_samples), optional) — Float values of the raw audio waveform, scattered over the audio placeholders of input_ids. See OmniASRModel.forward().
  • padding_mask (torch.Tensor of shape (batch_size, num_samples), optional) — Mask to avoid running the speech encoder over padding samples of input_values, with 1 for samples that are not padding. Distinct from attention_mask, which covers the decoder sequence.
  • attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
    • 1 for tokens that are not masked,
    • 0 for tokens that are masked.

    What are attention masks?

  • position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.n_positions - 1].

    What are position IDs?

  • cache (~cache_utils.Cache, optional) — Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the cache returned by the model at a previous stage of decoding, when use_cache=True or config.use_cache=True.

    Only Cache instance is allowed as input, see our kv cache guide. If no cache are passed, DynamicCache will be initialized by default.

    The model will output the same cache format that is fed as input.

    If cache are used, the user is expected to input only unprocessed input_ids (those that don’t have their past key value states given to this model) of shape (batch_size, unprocessed_length) instead of all input_ids of shape (batch_size, sequence_length).

  • inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
  • labels (torch.LongTensor of shape (batch_size, sequence_length), optional) — Labels for computing the causal language modeling loss. They are shifted internally, so they must be aligned with the decoder sequence — i.e. as long as input_ids / inputs_embeds, with -100 on the positions that should not contribute to the loss (the audio context, and padding).
  • use_cache (bool, optional) — If set to True, cache key value states are returned and can be used to speed up decoding (see cache).
  • logits_to_keep (Union[int, torch.Tensor], optional, defaults to 0) — If an int, compute logits for the last logits_to_keep tokens. If 0, calculate logits for all input_ids (special case). Only last token logits are needed for generation, and calculating them only for that token can save memory, which becomes pretty significant for long sequences or large vocabulary size. If a torch.Tensor, must be 1D corresponding to the indices to keep in the sequence length dimension. This is useful when using packed tensor format (single dimension for batch and sequence length).

Returns

CausalLMOutputWithPast or tuple(torch.Tensor)

A CausalLMOutputWithPast or a tuple of torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (OmniASRConfig) and inputs.

The OmniASRForConditionalGeneration forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

  • loss (torch.Tensor of shape (1,), optional, returned when labels is provided) — Language modeling loss (for next-token prediction).

  • logits (torch.Tensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

  • cache (Cache, optional, returned when use_cache=True is passed or when config.use_cache=True) — It is a Cache instance. For more details, see our kv cache guide.

    Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see cache input) to speed up sequential decoding.

  • hidden_states (tuple(torch.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple(torch.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Example:

>>> from transformers import AutoProcessor, OmniASRForConditionalGeneration
>>> from datasets import load_dataset, Audio

>>> model_id = "hf-audio/omniasr-llm-300m-v2"
>>> processor = AutoProcessor.from_pretrained(model_id)
>>> model = OmniASRForConditionalGeneration.from_pretrained(model_id)

>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))

>>> inputs = processor.apply_transcription_request(ds[0]["audio"]["array"], language="eng_Latn")
>>> generated_ids = model.generate(**inputs, max_new_tokens=256)
>>> transcription = processor.decode(generated_ids, skip_special_tokens=True)
Update on GitHub