Transformers documentation
OmniASR
This model was published in HF papers on 2025-11-12 and contributed to Hugging Face Transformers on 2026-11-16.
OmniASR
Overview
OmniASR (Omnilingual ASR) is a suite of multilingual speech-recognition models from Meta FAIR, introduced in Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages, that covers more than 1,600 languages — including hundreds never previously supported by any ASR system. Each model pairs a Wav2Vec2-style audio encoder (convolutional layers that downsample the 16 kHz waveform ~320× to a 50 Hz frame rate, followed by a pre-norm Transformer encoder) with one of two heads:
- CTC variant (
OmniASRForCTC, see OmniASR CTC): a single linear projection to the vocabulary, decoded non-autoregressively with greedy CTC. - ALM (audio language model) variant (
OmniASRForConditionalGeneration, this page): the audio embeddings are linearly projected into a Llama decoder which autoregressively generates the transcription. This variant additionally supports optional language conditioning: passing a language code such as"eng_Latn"makes the processor write the matching language token into the decoder context, which generally improves transcription quality.
Checkpoints can be found in this collection: 4x ALM and 4x CTC checkpoints of various sizes (300M, 1B, 3B, 7B). Original weights are released by Meta, and can be found on their GitHub repo.
This model was contributed by Eric Bezzam.
Usage
The simplest way to transcribe audio is with
apply_transcription_request, which handles the chat template formatting for you, namely it is a convenience wrapper forapply_chat_template(see Chat template below).
Simple transcription
from transformers import AutoModelForMultimodalLM, AutoProcessor
model_id = "hf-audio/omniasr-llm-300m-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
inputs = processor.apply_transcription_request(
audio="https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
)
inputs.to(model.device, dtype=model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
print(processor.decode(generated_ids, skip_special_tokens=True)[0])Language hint
Without a language, the model runs in its language-agnostic mode. Passing a language code (e.g. "eng_Latn", "fra_Latn", "cmn_Hans") writes the matching language token into the decoder prompt, which generally improves
transcription quality. The list of supported languages and naming convention follows the original here.
For the languages that have one, the more common ISO 639-1 codes (e.g. "en", "fr", "zh") are also accepted.
from transformers import AutoModelForMultimodalLM, AutoProcessor
model_id = "hf-audio/omniasr-llm-300m-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
audio = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav"
# Without language hint (language-agnostic)
inputs = processor.apply_transcription_request(audio=audio).to(model.device, dtype=model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
print(f"Language-agnostic: {processor.decode(generated_ids, skip_special_tokens=True)[0]}")
# With language hint
inputs = processor.apply_transcription_request(audio=audio, language="zh").to(model.device, dtype=model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
print(f"Language hint: {processor.decode(generated_ids, skip_special_tokens=True)[0]}")Batch inference
Batch inference is possible by passing a list of audios and, if provided, a single language for the whole batch or a
list of languages (one per audio, None for the language-agnostic mode).
from transformers import AutoModelForMultimodalLM, AutoProcessor
model_id = "hf-audio/omniasr-llm-300m-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
audio = [
"https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
"https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav",
]
inputs = processor.apply_transcription_request(audio, language=["en", None])
inputs.to(model.device, dtype=model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
for i, text in enumerate(processor.decode(generated_ids, skip_special_tokens=True)):
print(f"Audio {i + 1}: {text}")Chat template
The model also accepts chat template inputs. The apply_transcription_request usage above is a convenience wrapper for apply_chat_template. The language hint is given as a {"type": "language", ...} item of the user turn; leaving it out selects the language-agnostic mode.
from transformers import AutoModelForMultimodalLM, AutoProcessor
model_id = "hf-audio/omniasr-llm-300m-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
conversations = [
[
{
"role": "user",
"content": [
{
"type": "audio",
"path": "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
},
{"type": "language", "language": "en"},
],
},
],
[
{
"role": "user",
"content": [
{
"type": "audio",
"path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav",
},
{"type": "language", "language": "zh"},
],
},
],
]
inputs = processor.apply_chat_template(
conversations, tokenize=True, add_generation_prompt=True, return_dict=True
)
inputs.to(model.device, dtype=model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
for i, text in enumerate(processor.decode(generated_ids, skip_special_tokens=True)):
print(f"Audio {i + 1}: {text}")Training
OmniASR can be trained with the loss outputted by the model. Put the target transcript in the assistant turn of the chat template and pass output_labels=True. Audio and padding positions are masked automatically.
Note that OmniASR transcribes in lowercase, so the (uppercase) LibriSpeech transcripts are lowercased below.
from datasets import Audio, load_dataset
from transformers import AutoModelForMultimodalLM, AutoProcessor
model_id = "hf-audio/omniasr-llm-300m-v2"
NUM_SAMPLES = 2
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
model.train()
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
conversation = [
[
{
"role": "user",
"content": [
{"type": "audio", "audio": sample["audio"]["array"]},
# any supported language code, or no `language` item to train the language-agnostic mode
{"type": "language", "language": "en"},
],
},
{"role": "assistant", "content": [{"type": "text", "text": sample["text"].lower()}]},
]
for sample in ds.select(range(NUM_SAMPLES))
]
inputs = processor.apply_chat_template(
conversation,
tokenize=True,
return_dict=True,
# only train on the assistant outputs: mask the labels outside the assistant turns
return_assistant_tokens_mask=True,
return_tensors="pt",
processor_kwargs={"output_labels": True},
).to(model.device, dtype=model.dtype)
inputs["labels"][inputs.pop("assistant_masks") == 0] = -100
loss = model(**inputs).loss
print("Loss:", loss.item())
loss.backward()Torch compile
For autoregressive transcription, torch.compile accelerates the per-token forward passes inside generate by
providing a CompileConfig object, which requires a static cache.
Note that the StaticCache is created once and reused across generate calls to avoid recompiles.
On an A100, we observed a speed-up of ~2.3 for a batch size of 4 (script).
import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor, CompileConfig, StaticCache
model_id = "hf-audio/omniasr-llm-300m-v2"
NUM_SAMPLES = 4
num_warmup = 3
max_new_tokens = 256
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()
audio_url = "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav"
inputs = processor.apply_transcription_request(
audio=[audio_url] * NUM_SAMPLES,
language="en",
).to(model.device, dtype=model.dtype)
# Created once and reused across `generate` calls, to avoid recompiling at every call
static_cache = StaticCache(
config=model.config,
max_cache_len=inputs["input_ids"].shape[1] + max_new_tokens,
)
compile_config = CompileConfig()
# Warmup
with torch.inference_mode():
for _ in range(num_warmup):
static_cache.reset()
_ = model.generate(
**inputs, max_new_tokens=max_new_tokens, do_sample=False,
cache=static_cache, compile_config=compile_config,
)
torch.cuda.synchronize()
# Apply model
with torch.inference_mode():
static_cache.reset()
output_ids = model.generate(
**inputs, max_new_tokens=max_new_tokens, do_sample=False,
cache=static_cache, compile_config=compile_config,
)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
print(processor.decode(generated_ids, skip_special_tokens=True)[0])OmniASRAudioConfig
class transformers.OmniASRAudioConfig
< source >( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonehidden_size: int = 1024conv_dim: list[int] | tuple[int, ...] = (512, 512, 512, 512, 512, 512, 512)conv_kernel: list[int] | tuple[int, ...] = (10, 3, 3, 3, 3, 2, 2)conv_stride: list[int] | tuple[int, ...] = (5, 2, 2, 2, 2, 2, 2)conv_bias: bool = Truenum_attention_heads: int = 16num_hidden_layers: int = 24num_conv_pos_embeddings: int = 128num_conv_pos_embedding_groups: int = 16intermediate_size: int = 4096attention_dropout: float | int = 0.0hidden_dropout: float | int = 0.1layerdrop: float | int = 0.1activation_dropout: float | int = 0.1initializer_range: float = 0.02layer_norm_eps: float = 1e-05hidden_act: str = 'gelu' )
Parameters
- hidden_size (
int, optional, defaults to1024) — Dimension of the hidden representations. - conv_dim (
tuple[int]orlist[int], optional, defaults to(512, 512, 512, 512, 512, 512, 512)) — A tuple of integers defining the number of input and output channels of each 1D convolutional layer in the feature encoder. The length of conv_dim defines the number of 1D convolutional layers. - conv_kernel (
tuple[int]orlist[int], optional, defaults to(10, 3, 3, 3, 3, 2, 2)) — A tuple of integers defining the kernel size of each 1D convolutional layer in the feature encoder. The length of conv_kernel defines the number of convolutional layers and has to match the length of conv_dim. - conv_stride (
tuple[int]orlist[int], optional, defaults to(5, 2, 2, 2, 2, 2, 2)) — A tuple of integers defining the stride of each 1D convolutional layer in the feature encoder. The length of conv_stride defines the number of convolutional layers and has to match the length of conv_dim. - conv_bias (
bool, optional, defaults toTrue) — Whether the 1D convolutional layers have a bias. - num_attention_heads (
int, optional, defaults to16) — Number of attention heads for each attention layer in the Transformer decoder. - num_hidden_layers (
int, optional, defaults to24) — Number of hidden layers in the Transformer decoder. - num_conv_pos_embeddings (
int, optional, defaults to 128) — Number of convolutional positional embeddings. Defines the kernel size of the 1D convolutional positional embeddings layer. - num_conv_pos_embedding_groups (
int, optional, defaults to 16) — Number of groups of the 1D convolutional positional embeddings layer. - intermediate_size (
int, optional, defaults to4096) — Dimension of the MLP representations. - attention_dropout (
Union[float, int], optional, defaults to0.0) — The dropout ratio for the attention probabilities. - hidden_dropout (
Union[float, int], optional, defaults to0.1) — The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. - layerdrop (
Union[float, int], optional, defaults to0.1) — The LayerDrop probability. See the [LayerDrop paper](see https://huggingface.co/papers/1909.11556) for more details. - activation_dropout (
Union[float, int], optional, defaults to0.1) — The dropout ratio for activations inside the fully connected layer. - initializer_range (
float, optional, defaults to0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices. - layer_norm_eps (
float, optional, defaults to1e-05) — The epsilon used by the layer normalization layers. - hidden_act (
str, optional, defaults togelu) — The non-linear activation function (function or string) in the decoder. For example,"gelu","relu","silu", etc.
This is the configuration class to store the configuration of a OmniASRModel. It is used to instantiate a Omniasr model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the hf-audio/omniasr-ctc-300m-v2
Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.
Example:
>>> from transformers import OmniASRAudioConfig, OmniASRAudioModel
>>> # Initializing an OmniASR encoder configuration
>>> configuration = OmniASRAudioConfig()
>>> # Initializing a model (with random weights) from the configuration
>>> model = OmniASRAudioModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.configOmniASRConfig
class transformers.OmniASRConfig
< source >( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Noneaudio_config: dict | transformers.configuration_utils.PreTrainedConfig | None = Nonetext_config: dict | transformers.configuration_utils.PreTrainedConfig | None = Noneaudio_token_id: int = 10289initializer_range: float = 0.02bos_token_id: int | None = 0pad_token_id: int | None = 1eos_token_id: int | None = 2 )
Parameters
- audio_config (
Union[dict, ~configuration_utils.PreTrainedConfig], optional) — The config object or dictionary of the audio backbone. - text_config (
Union[dict, ~configuration_utils.PreTrainedConfig], optional) — The config object or dictionary of the text backbone. - audio_token_id (
int, optional, defaults to10289) — The audio token index used as a placeholder for input audio. - initializer_range (
float, optional, defaults to0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices. - bos_token_id (
int, optional, defaults to0) — Token id used for beginning-of-stream in the vocabulary. - pad_token_id (
int, optional, defaults to1) — Token id used for padding in the vocabulary. - eos_token_id (
int, optional, defaults to2) — Token id used for end-of-stream in the vocabulary.
This is the configuration class to store the configuration of a OmniASRModel. It is used to instantiate a Omniasr model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the hf-audio/omniasr-llm-300m-v2
Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.
Example:
>>> from transformers import OmniASRForConditionalGeneration, OmniASRConfig
>>> # Initializing an OmniASR-LLM configuration
>>> configuration = OmniASRConfig()
>>> # Initializing a model (with random weights) from the configuration
>>> model = OmniASRForConditionalGeneration(configuration)
>>> # Accessing the model configuration
>>> configuration = model.configOmniASRFeatureExtractor
class transformers.OmniASRFeatureExtractor
< source >( feature_size = 1sampling_rate = 16000padding_value = 0.0return_attention_mask = Truedo_normalize = True**kwargs )
Parameters
- feature_size (
int, optional, defaults to 1) — The feature dimension of the extracted features. - sampling_rate (
int, optional, defaults to 16000) — The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). - padding_value (
float, optional, defaults to 0.0) — The value that is used to fill the padding values. - return_attention_mask (
bool, optional, defaults toTrue) — Whether or not__call__()should returnattention_mask. OmniASR needs the mask to know which frames are padding, so batched inputs are only encoded correctly when it is returned. - do_normalize (
bool, optional, defaults toTrue) — Whether or not to normalize each audio sample to zero mean and unit variance. The statistics are computed per sample, over its non-padded samples, to match the original implementation.
Constructs an OmniASR feature extractor.
OmniASRProcessor
class transformers.OmniASRProcessor
< source >( feature_extractortokenizerchat_template = Noneaudio_token = '<extra_id_1>'conv_kernel = Noneconv_stride = None )
Parameters
- feature_extractor (
OmniASRFeatureExtractor) — The feature extractor is a required input. - tokenizer (
TokenizersBackend) — The tokenizer is a required input. - chat_template (
str) — A Jinja template to convert lists of messages in a chat into a tokenizable string. - audio_token (
str, optional, defaults to"<extra_id_1>") — The placeholder token that stands for one speech encoder frame in the prompt, i.e. the token ofOmniASRConfig.audio_token_id(). - conv_kernel (
list[int], optional) — Kernel size of each convolution of the speech encoder’s feature encoder, i.e.OmniASRAudioConfig.conv_kernel(). Needed to count the frames an audio input is subsampled to, and therefore how many audio placeholders its prompt holds. Defaults to[10, 3, 3, 3, 3, 2, 2]. - conv_stride (
list[int], optional) — Stride of each convolution of the speech encoder’s feature encoder, i.e.OmniASRAudioConfig.conv_stride(). Defaults to[5, 2, 2, 2, 2, 2, 2].
Constructs a OmniASRProcessor which wraps a feature extractor and a tokenizer into a single processor.
OmniASRProcessor offers all the functionalities of OmniASRFeatureExtractor and TokenizersBackend. See the ~OmniASRFeatureExtractor and ~TokenizersBackend for more information.
Same as apply_chat_template(), but resolves the language codes of the conversation first.
The chat template writes the language token as is, so this override is needed to accept ISO 639-1 codes (e.g. "en" for "eng_Latn"), and to raise on unsupported codes instead of silently writing an unknown token.
apply_transcription_request
< source >( audio: typing.Union[str, list[str], numpy.ndarray, ForwardRef('torch.Tensor'), collections.abc.Sequence[numpy.ndarray], collections.abc.Sequence['torch.Tensor']]language: str | list[str] | None = Nonetokenize: bool = True**kwargs: *<class 'transformers.processing_utils.ProcessingKwargs'> ) → BatchFeature or list[str]
Parameters
- audio (
str,list[str],np.ndarray,torch.Tensor,list[np.ndarray],list[torch.Tensor]) — Audio to transcribe. Strings are interpreted as local paths or URLs and will be loaded automatically by the chat template loader; NumPy arrays and PyTorch tensors are forwarded directly. - language (
strorlist[str], optional) — Language code(s) (e.g."eng_Latn"or["eng_Latn", "fra_Latn"]), either one for the whole batch or one per audio. ISO 639-1 codes (e.g."en") are also accepted for the languages that have one.Noneor"auto"selects the model’s language-agnostic mode; naming the language explicitly gives better transcription quality. - tokenize (
bool, optional, defaults toTrue) — Whether to tokenize the prompts and process the audio. IfFalse, the audio is not loaded and the prompts are returned as strings with unexpanded audio placeholders. Passing them with the audio to the processor,processor(text=prompts, audio=audio), gives the same outputs astokenize=True. - **kwargs —
Additional keyword arguments forwarded to apply_chat_template().
Processor kwargs (for example
text_kwargs,audio_kwargs, …) should be passed asprocessor_kwargs={...}.
Returns
BatchFeature or list[str]
One prompt per audio when tokenize=False, otherwise processor
outputs ready to be passed to OmniASRForConditionalGeneration.generate().
Prepare inputs for speech recognition without manually writing the chat template.
OmniASRAudioModel
class transformers.OmniASRAudioModel
< source >( config: OmniASRAudioConfig )
Parameters
- config (OmniASRAudioConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.
The OmniASR speech encoder, which is a Wav2Vec2-style encoder.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)
This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
forward
< source >( input_values: typing.Optional[torch.Tensor]padding_mask: typing.Optional[torch.Tensor] = Noneoutput_attention_mask: bool = True**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → OmniASREncoderModelOutput or tuple(torch.Tensor)
Parameters
- input_values (
torch.Tensorof shape(batch_size, sequence_length), optional) — Float values of input raw speech waveform. Values can be obtained by loading a.flacor.wavaudio file into an array of typelist[float], anumpy.ndarrayor atorch.Tensor, e.g. via the torchcodec library (pip install torchcodec) or the soundfile library (pip install soundfile). To prepare the array intoinput_values, the AutoProcessor should be used for padding and conversion into a tensor of typetorch.Tensor. Seeprocessor_class.__call__for details. - padding_mask (
torch.Tensorof shape(batch_size, 1, sequence_length)) — Padding mask used to padinput_values. - output_attention_mask (
bool, optional, defaults toTrue) — Whether to return the subsampled attention mask. Only effective whenpadding_maskis provided.
Returns
OmniASREncoderModelOutput or tuple(torch.Tensor)
A OmniASREncoderModelOutput or a tuple of
torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (None) and inputs.
The OmniASRAudioModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
last_hidden_state (
torch.Tensorof shape(batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.pooler_output (
torch.Tensorof shape(batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.hidden_states (
tuple(torch.Tensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.Tensor(one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (
tuple(torch.Tensor), optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.Tensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
attention_mask (
torch.Tensorof shape(batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices after sequence compression. Returned because the sequence length may differ from the input sequence length. Mask values selected in[0, 1]:- 1 for tokens that are not masked,
- 0 for tokens that are masked.
OmniASRModel
class transformers.OmniASRModel
< source >( config: OmniASRConfig )
Parameters
- config (OmniASRConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.
The OmniASR model, which consists of a Wav2Vec2 encoder, a multi-modal projector and a LLama language model, without a language modeling head.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)
This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
forward
< source >( input_ids: typing.Optional[torch.LongTensor] = Noneinputs_embeds: typing.Optional[torch.Tensor] = Noneinput_values: typing.Optional[torch.Tensor] = Nonepadding_mask: typing.Optional[torch.Tensor] = None**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → OmniASRBaseModelOutputWithPast or tuple(torch.Tensor)
Parameters
- input_ids (
torch.LongTensorof shape(batch_size, sequence_length), optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.
- inputs_embeds (
torch.Tensorof shape(batch_size, sequence_length, hidden_size), optional) — Optionally, instead of passinginput_idsyou can choose to directly pass an embedded representation. This is useful if you want more control over how to convertinput_idsindices into associated vectors than the model’s internal embedding lookup matrix. - input_values (
torch.Tensorof shape(batch_size, sequence_length), optional) — Float values of input raw speech waveform. Values can be obtained by loading a.flacor.wavaudio file into an array of typelist[float], anumpy.ndarrayor atorch.Tensor, e.g. via the torchcodec library (pip install torchcodec) or the soundfile library (pip install soundfile). To prepare the array intoinput_values, the AutoProcessor should be used for padding and conversion into a tensor of typetorch.Tensor. SeeOmniASRProcessor.__call__()for details. - padding_mask (
torch.Tensorof shape(batch_size, num_samples), optional) — Padding mask to remove padded parts of audio.
Returns
OmniASRBaseModelOutputWithPast or tuple(torch.Tensor)
A OmniASRBaseModelOutputWithPast or a tuple of
torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (OmniASRConfig) and inputs.
The OmniASRModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
last_hidden_state (
torch.Tensorof shape(batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.If
cacheis used only the last hidden-state of the sequences of shape(batch_size, 1, hidden_size)is output.cache (
Cache, optional, returned whenuse_cache=Trueis passed or whenconfig.use_cache=True) — It is a Cache instance. For more details, see our kv cache guide.Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if
config.is_encoder_decoder=Truein the cross-attention blocks) that can be used (seecacheinput) to speed up sequential decoding.hidden_states (
tuple(torch.Tensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.Tensor(one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (
tuple(torch.Tensor), optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.Tensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
audio_features (
torch.Tensorof shape(num_audio_frames, hidden_size), optional) — Encoder output frames, before the multi-modal projector, for the non-padded audio frames.
OmniASRForConditionalGeneration
class transformers.OmniASRForConditionalGeneration
< source >( config: OmniASRConfig )
Parameters
- config (OmniASRConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.
OmniASR model, which consists of a Wav2Vec2 encoder, a multi-modal projector and a LLama language model.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)
This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
forward
< source >( input_ids: typing.Optional[torch.LongTensor] = Noneinput_values: typing.Optional[torch.Tensor] = Nonepadding_mask: typing.Optional[torch.Tensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneposition_ids: typing.Optional[torch.LongTensor] = Nonecache: transformers.cache_utils.Cache | None = Noneinputs_embeds: typing.Optional[torch.FloatTensor] = Nonelabels: typing.Optional[torch.Tensor] = Noneuse_cache: bool | None = Nonelogits_to_keep: typing.Union[int, torch.Tensor] = 0**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → CausalLMOutputWithPast or tuple(torch.Tensor)
Parameters
- input_ids (
torch.LongTensorof shape(batch_size, sequence_length), optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.
- input_values (
torch.Tensorof shape(batch_size, 1, num_samples), optional) — Float values of the raw audio waveform, scattered over the audio placeholders ofinput_ids. See OmniASRModel.forward(). - padding_mask (
torch.Tensorof shape(batch_size, num_samples), optional) — Mask to avoid running the speech encoder over padding samples ofinput_values, with 1 for samples that are not padding. Distinct fromattention_mask, which covers the decoder sequence. - attention_mask (
torch.Tensorof shape(batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in[0, 1]:- 1 for tokens that are not masked,
- 0 for tokens that are masked.
- position_ids (
torch.LongTensorof shape(batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range[0, config.n_positions - 1]. - cache (
~cache_utils.Cache, optional) — Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in thecachereturned by the model at a previous stage of decoding, whenuse_cache=Trueorconfig.use_cache=True.Only Cache instance is allowed as input, see our kv cache guide. If no
cacheare passed, DynamicCache will be initialized by default.The model will output the same cache format that is fed as input.
If
cacheare used, the user is expected to input only unprocessedinput_ids(those that don’t have their past key value states given to this model) of shape(batch_size, unprocessed_length)instead of allinput_idsof shape(batch_size, sequence_length). - inputs_embeds (
torch.FloatTensorof shape(batch_size, sequence_length, hidden_size), optional) — Optionally, instead of passinginput_idsyou can choose to directly pass an embedded representation. This is useful if you want more control over how to convertinput_idsindices into associated vectors than the model’s internal embedding lookup matrix. - labels (
torch.LongTensorof shape(batch_size, sequence_length), optional) — Labels for computing the causal language modeling loss. They are shifted internally, so they must be aligned with the decoder sequence — i.e. as long asinput_ids/inputs_embeds, with-100on the positions that should not contribute to the loss (the audio context, and padding). - use_cache (
bool, optional) — If set toTrue,cachekey value states are returned and can be used to speed up decoding (seecache). - logits_to_keep (
Union[int, torch.Tensor], optional, defaults to0) — If anint, compute logits for the lastlogits_to_keeptokens. If0, calculate logits for allinput_ids(special case). Only last token logits are needed for generation, and calculating them only for that token can save memory, which becomes pretty significant for long sequences or large vocabulary size. If atorch.Tensor, must be 1D corresponding to the indices to keep in the sequence length dimension. This is useful when using packed tensor format (single dimension for batch and sequence length).
Returns
CausalLMOutputWithPast or tuple(torch.Tensor)
A CausalLMOutputWithPast or a tuple of
torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (OmniASRConfig) and inputs.
The OmniASRForConditionalGeneration forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
loss (
torch.Tensorof shape(1,), optional, returned whenlabelsis provided) — Language modeling loss (for next-token prediction).logits (
torch.Tensorof shape(batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).cache (
Cache, optional, returned whenuse_cache=Trueis passed or whenconfig.use_cache=True) — It is a Cache instance. For more details, see our kv cache guide.Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see
cacheinput) to speed up sequential decoding.hidden_states (
tuple(torch.Tensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.Tensor(one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (
tuple(torch.Tensor), optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.Tensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
Example:
>>> from transformers import AutoProcessor, OmniASRForConditionalGeneration
>>> from datasets import load_dataset, Audio
>>> model_id = "hf-audio/omniasr-llm-300m-v2"
>>> processor = AutoProcessor.from_pretrained(model_id)
>>> model = OmniASRForConditionalGeneration.from_pretrained(model_id)
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
>>> inputs = processor.apply_transcription_request(ds[0]["audio"]["array"], language="eng_Latn")
>>> generated_ids = model.generate(**inputs, max_new_tokens=256)
>>> transcription = processor.decode(generated_ids, skip_special_tokens=True)