# VibeVoice ASR

## 概述

VibeVoice ASR 是微软推出的一款自动语音识别模型，它将声学音频分词器与语义音频分词器相结合，并配合因果语言模型，以实现稳健的语音转文本转录。该模型使用 VibeVoice 的声学分词器和语义分词器，以 24kHz 处理音频，并搭配基于 Qwen2 的语言解码器来生成转录文本。有关更多详细信息，请参阅[技术报告](https://huggingface.co/papers/2601.18184)。

模型检查点可在以下位置获取：[microsoft/VibeVoice-ASR-HF](https://huggingface.co/microsoft/VibeVoice-ASR-HF)

亮点：

- **🕒 60 分钟单次处理**：与将音频切分为短片段（通常会丢失全局上下文）的传统 ASR 模型不同，VibeVoice ASR 在 64K token 长度内可接受长达 **60 分钟** 的连续音频输入。这确保了整个一小时内说话人追踪的一致性和语义连贯性。

- **👤 自定义热词**：用户可以提供自定义热词（例如特定名称、技术术语或背景信息）来引导识别过程，从而显著提升对领域特定内容的识别准确率。

- **📝 富文本转录（谁、何时、说了什么）**：该模型同时执行 ASR、说话人分离和时间戳标注，生成结构化输出，指示*谁*在*何时*说了*什么*。
  
- **🌍 多语言与语码转换支持**：支持 50 多种语言，无需显式设置语言，并原生处理话语内部及话语之间的语码转换。

该模型由 [Eric Bezzam](https://huggingface.co/bezzam) 贡献。

## 用法

该模型支持多种自动语音识别功能。

### 带说话人时间戳的转录

VibeVoice ASR 的一个显著特性是它能够转录多说话人内容，并标注每位说话人的身份及其发言时间。

```python
from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration

model_id = "microsoft/VibeVoice-ASR-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceAsrForConditionalGeneration.from_pretrained(model_id, device_map="auto")
print(f"Model loaded on {model.device} with dtype {model.dtype}")

# Prepare inputs using `apply_transcription_request`
inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav",
).to(model.device, model.dtype)

# Apply model
output_ids = model.generate(**inputs)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
transcription = processor.decode(generated_ids)[0]
print("\n" + "=" * 60)
print("RAW OUTPUT")
print("=" * 60)
print(transcription)

transcription = processor.decode(generated_ids, return_format="parsed")[0]
print("\n" + "=" * 60)
print("TRANSCRIPTION (list of dicts)")
print("=" * 60)
for speaker_transcription in transcription:
    print(speaker_transcription)

# Remove speaker labels, only get raw transcription
transcription = processor.decode(generated_ids, return_format="transcription_only")[0]
print("\n" + "=" * 60)
print("TRANSCRIPTION ONLY")
print("=" * 60)
print(transcription)

"""
============================================================
RAW OUTPUT
============================================================
<|im_start|>assistant
[{"Start":0,"End":15.43,"Speaker":0,"Content":"Hello everyone and welcome to the Vibe Voice podcast. I'm your host, Alex, and today we're getting into one of the biggest debates in all of sports: who's the greatest basketball player of all time? I'm so excited to have Sam here to talk about it with me."},{"Start":15.43,"End":21.05,"Speaker":1,"Content":"Thanks so much for having me, Alex. And you're absolutely right. This question always brings out some seriously strong feelings."},{"Start":21.05,"End":31.66,"Speaker":0,"Content":"Okay, so let's get right into it. For me, it has to be Michael Jordan. Six trips to the finals, six championships. That kind of perfection is just incredible."},{"Start":31.66,"End":40.93,"Speaker":1,"Content":"Oh man, the first thing that always pops into my head is that shot against the Cleveland Cavaliers back in '89. Jordan just rises, hangs in the air forever, and just sinks it."}]<|im_end|>
<|endoftext|>

============================================================
TRANSCRIPTION (list of dicts)
============================================================
{'Start': 0, 'End': 15.43, 'Speaker': 0, 'Content': "Hello everyone and welcome to the Vibe Voice podcast. I'm your host, Alex, and today we're getting into one of the biggest debates in all of sports: who's the greatest basketball player of all time? I'm so excited to have Sam here to talk about it with me."}
{'Start': 15.43, 'End': 21.05, 'Speaker': 1, 'Content': "Thanks so much for having me, Alex. And you're absolutely right. This question always brings out some seriously strong feelings."}
{'Start': 21.05, 'End': 31.66, 'Speaker': 0, 'Content': "Okay, so let's get right into it. For me, it has to be Michael Jordan. Six trips to the finals, six championships. That kind of perfection is just incredible."}
{'Start': 31.66, 'End': 40.93, 'Speaker': 1, 'Content': "Oh man, the first thing that always pops into my head is that shot against the Cleveland Cavaliers back in '89. Jordan just rises, hangs in the air forever, and just sinks it."}

============================================================
TRANSCRIPTION ONLY
============================================================
Hello everyone and welcome to the Vibe Voice podcast. I'm your host, Alex, and today we're getting into one of the biggest debates in all of sports: who's the greatest basketball player of all time? I'm so excited to have Sam here to talk about it with me. Thanks so much for having me, Alex. And you're absolutely right. This question always brings out some seriously strong feelings. Okay, so let's get right into it. For me, it has to be Michael Jordan. Six trips to the finals, six championships. That kind of perfection is just incredible. Oh man, the first thing that always pops into my head is that shot against the Cleveland Cavaliers back in '89. Jordan just rises, hangs in the air forever, and just sinks it.
"""
```

VibeVoice ASR 模型经过训练，可生成类似 JSON 结构的字符串。标志 `return_format="parsed"` 会尝试将生成的输出作为字典列表返回，而 `return_format="transcription_only"` 则尝试仅提取转录的音频。如果失败，生成的输出将按原样返回。

### 提供上下文

也可以提供上下文。如果某些词语（例如专有名词）无法被正确转录，这样做会很有用。

下方我们转录了一段音频，其中说话者（带有德语口音）谈论了 VibeVoice，并对比了带有上下文 “About VibeVoice” 和不带上下文的情况。

```python
from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration

model_id = "microsoft/VibeVoice-ASR-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceAsrForConditionalGeneration.from_pretrained(model_id, device_map="auto")
print(f"Model loaded on {model.device} with dtype {model.dtype}")

# Without context
inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
).to(model.device, model.dtype)
output_ids = model.generate(**inputs)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
transcription = processor.decode(generated_ids, return_format="transcription_only")[0]
print(f"WITHOUT CONTEXT: {transcription}")

# With context
inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
    prompt="About VibeVoice",
).to(model.device, model.dtype)
output_ids = model.generate(**inputs)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
transcription = processor.decode(generated_ids, return_format="transcription_only")[0]
print(f"WITH CONTEXT   : {transcription}")

"""
WITHOUT CONTEXT: Revevoices is a novel framework designed for generating expressive, long-form, multi-speaker conversational audio.
WITH CONTEXT   : VibeVoice is this novel framework designed for generating expressive, long-form, multi-speaker, conversational audio.
"""
```

### 批量推理

通过传入音频列表以及（如果提供）提示词列表，可以进行批量推理。音频输入和提示词的数量应保持一致（对于提示词，如果某个音频不需要提示词，可以将其条目设置为 `None`）。

```python
from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration

model_id = "microsoft/VibeVoice-ASR-HF"
audio = [
    "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
    "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav"
]
prompts = ["About VibeVoice", None]

processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceAsrForConditionalGeneration.from_pretrained(model_id, device_map="auto")
print(f"Model loaded on {model.device} with dtype {model.dtype}")

inputs = processor.apply_transcription_request(audio, prompt=prompts).to(model.device, model.dtype)
output_ids = model.generate(**inputs)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
transcription = processor.decode(generated_ids, return_format="transcription_only")

print(transcription)
```

### 调整分词器块大小（例如，当出现内存不足时）

VibeVoice ASR 的一个关键特性是能够转录长达 60 分钟的连续音频。其实现方式是将音频切分为 60 秒的片段（在 24kHz 采样率下为 1440000 个采样点），并在各片段之间缓存卷积状态。

不过，如果 60 秒的片段对于您的设备来说过大，可以调整传递给 `generate` 的 `acoustic_tokenizer_chunk_size` 参数。*请注意，该值应为 hop length 的整数倍（原始 acoustic tokenizer 的 hop length 为 3200）。*

```python
from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration

acoustic_tokenizer_chunk_size = 64000    # default is 1440000 (60s @ 24kHz)
model_id = "microsoft/VibeVoice-ASR-HF"
audio = [
    "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
    "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav"
]
prompts = ["About VibeVoice", None]

processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceAsrForConditionalGeneration.from_pretrained(model_id, device_map="auto")
print(f"Model loaded on {model.device} with dtype {model.dtype}")

inputs = processor.apply_transcription_request(audio, prompt=prompts).to(model.device, model.dtype)
output_ids = model.generate(**inputs, acoustic_tokenizer_chunk_size=acoustic_tokenizer_chunk_size)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
transcription = processor.decode(generated_ids, return_format="transcription_only")
print(transcription)
```

### 聊天模板

VibeVoice ASR 也接受聊天模板输入（`apply_transcription_request` 实际上是 `apply_chat_template` 的便捷封装）：
```python
from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration

model_id = "microsoft/VibeVoice-ASR-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceAsrForConditionalGeneration.from_pretrained(model_id, device_map="auto")

chat_template = [
    [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "About VibeVoice"},
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
                },
            ],
        }
    ],
    [
        {
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav",
                },
            ],
        }
    ],
]

inputs = processor.apply_chat_template(
    chat_template,
    tokenize=True,
    return_dict=True,
).to(model.device, model.dtype)

output_ids = model.generate(**inputs)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
transcription = processor.decode(generated_ids, return_format="transcription_only")
print(transcription)
```

### 训练

VibeVoice ASR 可以使用模型输出的损失进行训练。

```python
from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration

model_id = "microsoft/VibeVoice-ASR-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceAsrForConditionalGeneration.from_pretrained(model_id, device_map="auto")
model.train()

# Prepare batch of 2
# -- NOTE: the original model is trained to output transcription, speaker ID, and timestamps in JSON-like format. Below we are only using the transcription text as the label
chat_template = [
    [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "VibeVoice is this novel framework designed for generating expressive, long-form, multi-speaker, conversational audio."},
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
                },
            ],
        }
    ],
    [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Hello everyone and welcome to the VibeVoice podcast. I'm your host, Alex, and today we're getting into one of the biggest debates in all of sports: who's the greatest basketball player of all time? I'm so excited to have Sam here to talk about it with me. Thanks so much for having me, Alex. And you're absolutely right. This question always brings out some seriously strong feelings. Okay, so let's get right into it. For me, it has to be Michael Jordan. Six trips to the finals, six championships. That kind of perfection is just incredible. Oh man, the first thing that always pops into my head is that shot against the Cleveland Cavaliers back in '89. Jordan just rises, hangs in the air forever, and just sinks it."},
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav",
                },
            ],
        }
    ],
]
inputs = processor.apply_chat_template(
    chat_template,
    tokenize=True,
    return_dict=True,
    output_labels=True,
).to(model.device, model.dtype)

loss = model(**inputs).loss
print("Loss:", loss.item())
loss.backward()
```

### Torch 编译

该模型可以编译以加快推理/训练速度。
```python
import time

import torch

from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration

model_id = "microsoft/VibeVoice-ASR-HF"

num_warmup = 5
num_runs = 20

# Load processor + model
processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceAsrForConditionalGeneration.from_pretrained(model_id, device_map="auto")

# Prepare static inputs
chat_template = [
    [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "VibeVoice is this novel framework designed for generating expressive, long-form, multi-speaker, conversational audio.",
                },
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
                },
            ],
        }
    ],
] * 4  # batch size 4
inputs = processor.apply_chat_template(
    chat_template,
    tokenize=True,
    return_dict=True,
).to(model.device, torch.bfloat16)

# Benchmark without compile
print("Warming up without compile...")
with torch.no_grad():
    for _ in range(num_warmup):
        _ = model(**inputs)

torch.accelerator.synchronize()

print("\nBenchmarking without torch.compile...")
torch.accelerator.synchronize()
start = time.time()
with torch.no_grad():
    for _ in range(num_runs):
        _ = model(**inputs)
torch.accelerator.synchronize()
no_compile_time = (time.time() - start) / num_runs
print(f"Average time without compile: {no_compile_time:.4f}s")

# Benchmark with compile
print("\nCompiling model...")
model = torch.compile(model)

print("Warming up with compile (includes graph capture)...")
with torch.no_grad():
    for _ in range(num_warmup):
        _ = model(**inputs)

torch.accelerator.synchronize()

print("\nBenchmarking with torch.compile...")
torch.accelerator.synchronize()
start = time.time()
with torch.no_grad():
    for _ in range(num_runs):
        _ = model(**inputs)
torch.accelerator.synchronize()
compile_time = (time.time() - start) / num_runs
print(f"Average time with compile: {compile_time:.4f}s")

speedup = no_compile_time / compile_time
print(f"\nSpeedup: {speedup:.2f}x")
```

### Pipeline 用法

该模型可以作为流水线使用，但您需要自行定义方法来解析原始输出。

```python
from transformers import pipeline

model_id = "microsoft/VibeVoice-ASR-HF"
pipe = pipeline("any-to-any", model=model_id, device_map="auto")
chat_template = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "About VibeVoice"},
            {
                "type": "audio",
                "path": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
            },
        ],
    }
]
outputs = pipe(text=chat_template, return_full_text=False)

print("\n" + "=" * 60)
print("RAW PIPELINE OUTPUT")
print("=" * 60)
print(outputs[0]["generated_text"])

print("\n" + "=" * 60)
print("DICT OUTPUT")
print("=" * 60)
dict_output = pipe.processor.extract_speaker_dict(outputs[0]["generated_text"])
print(dict_output)

print("\n" + "=" * 60)
print("TRANSCRIPT OUTPUT")
print("=" * 60)
transcription = pipe.processor.extract_transcription(outputs[0]["generated_text"])
print(transcription)
```

## VibeVoiceAsrConfig[[transformers.VibeVoiceAsrConfig]]

#### transformers.VibeVoiceAsrConfig[[transformers.VibeVoiceAsrConfig]]

```python
transformers.VibeVoiceAsrConfig(transformers_version: str | None = None, architectures: list[str] | None = None, output_hidden_states: bool | None = False, return_dict: bool | None = True, dtype: str | torch.dtype | None = None, chunk_size_feed_forward: int = 0, is_encoder_decoder: bool = False, id2label: dict[int, str] | dict[str, str] | None = None, label2id: dict[str, int] | dict[str, str] | None = None, problem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = None, acoustic_tokenizer_encoder_config: dict | transformers.configuration_utils.PreTrainedConfig | None = None, semantic_tokenizer_encoder_config: dict | transformers.configuration_utils.PreTrainedConfig | None = None, text_config: dict | transformers.configuration_utils.PreTrainedConfig | None = None, audio_token_id: int = 151648, audio_bos_token_id: int = 151646, audio_eos_token_id: int = 151647, acoustic_tokenizer_chunk_size: int = 1440000)
```

[来源](https://github.com/huggingface/transformers/blob/v5.23.0/src/transformers/models/vibevoice_asr/configuration_vibevoice_asr.py#L31)

**参数：**

acoustic_tokenizer_encoder_config (`Union[VibeVoiceAcousticTokenizerConfig, dict]`, *可选*) : 声学分词器的配置对象或字典。该分词器用于从音频中提取声学特征。

semantic_tokenizer_encoder_config (`Union[VibeVoiceAcousticTokenizerConfig, dict]`, *可选*) : 语义分词器的配置对象或字典。该分词器用于从音频中提取语义特征。

text_config (`Union[dict, ~configuration_utils.PreTrainedConfig]`，*可选*)：文本骨干网络的配置对象或字典。

audio_token_id (`int`，*可选*，默认为 `151648`)：用作输入音频占位符的音频 token 索引。

audio_bos_token_id (`int`, *可选*，默认为 151646)：音频序列起始（begin-of-sequence）标记的索引。

audio_eos_token_id (`int`, *可选*，默认为 151647)：音频序列结束标记的索引。

acoustic_tokenizer_chunk_size (`int`，*可选*，默认为 1440000)：对音频输入进行分词时使用的块大小（以样本数计）。默认值对应 24kHz 下的 60 秒。

这是用于存储 VibeVoiceAsrModel 配置的类。它根据指定的参数实例化 Vibevoice Asr 模型，并定义模型架构。使用默认值实例化配置将得到与 [microsoft/VibeVoice-ASR-HF](https://huggingface.co/microsoft/VibeVoice-ASR-HF) 类似的配置。

配置对象继承自 [PreTrainedConfig](/docs/transformers/v5.23.0/zh/main_classes/configuration#transformers.PreTrainedConfig)，可用于控制模型输出。有关更多信息，请参阅 [PreTrainedConfig](/docs/transformers/v5.23.0/zh/main_classes/configuration#transformers.PreTrainedConfig) 的文档。

示例：

```python
>>> from transformers import VibeVoiceAsrForConditionalGeneration, VibeVoiceAsrConfig, VibeVoiceAcousticTokenizerEncoderConfig, Qwen2Config

>>> # Initializing VibeVoice acoustic and semantic encoder configs
>>> acoustic_config = VibeVoiceAcousticTokenizerEncoderConfig()
>>> semantic_config = VibeVoiceAcousticTokenizerEncoderConfig(hidden_size=128)

>>> # Initializing a Qwen2 config
>>> text_config = Qwen2Config()

>>> # Initializing a VibeVoice ASR configuration
>>> configuration = VibeVoiceAsrConfig(acoustic_config, semantic_config, text_config)

>>> # Initializing a model from the vibevoice_asr style configuration
>>> model = VibeVoiceAsrForConditionalGeneration(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config
```

## VibeVoiceAsrProcessor[[transformers.VibeVoiceAsrProcessor]]

#### transformers.VibeVoiceAsrProcessor[[transformers.VibeVoiceAsrProcessor]]

```python
transformers.VibeVoiceAsrProcessor(feature_extractor, tokenizer, chat_template = None, audio_token = '<|box_start|>', audio_bos_token = '<|object_ref_start|>', audio_eos_token = '<|object_ref_end|>', audio_duration_token = '<|AUDIO_DURATION|>')
```

[来源](https://github.com/huggingface/transformers/blob/v5.23.0/src/transformers/models/vibevoice_asr/processing_vibevoice_asr.py#L31)

**参数：**

feature_extractor (`VibeVoiceAcousticTokenizerFeatureExtractor`)：用于音频处理的特征提取器。

tokenizer (`Qwen2TokenizerFast`)：用于文本处理的分词器。

chat_template (`str`, *可选*) : 一个 Jinja 模板，用于将聊天中的消息列表转换为可分词的字符串。

audio_token (`str`，*可选*，默认为 `"<|box_start|>"`)：在聊天模板中使用的音频标记占位符。

audio_bos_token (`str`，*可选*，默认为 `"<|object_ref_start|>"`)：在聊天模板中使用的音频序列开始标记占位符。

audio_eos_token (`str`，*可选*，默认为 `"<|object_ref_end|>"`)：在聊天模板中使用的音频序列结束标记占位符。

audio_duration_token (`str`，*可选*，默认为 `"<|AUDIO_DURATION|>"`)：在聊天模板中使用的音频时长标记占位符。

构建一个 VibeVoice ASR 处理器，它将 [VibeVoiceAcousticTokenizerFeatureExtractor](/docs/transformers/v5.23.0/zh/model_doc/vibevoice_acoustic_tokenizer#transformers.VibeVoiceAcousticTokenizerFeatureExtractor) 和 [Qwen2TokenizerFast](/docs/transformers/v5.23.0/zh/model_doc/qwen2#transformers.Qwen2Tokenizer) 封装为单个处理器，同时继承音频特征提取和分词器的功能。

有关更多信息，请参阅 [__call__()](/docs/transformers/v5.23.0/zh/model_doc/vibevoice_asr#transformers.VibeVoiceAsrProcessor.__call__)。

#### __调用__[[transformers.VibeVoiceAsrProcessor.__call__]]

```python
__call__(text: str | list[str], audio: typing.Union[numpy.ndarray, ForwardRef('torch.Tensor'), collections.abc.Sequence[numpy.ndarray], collections.abc.Sequence['torch.Tensor'], NoneType] = None, output_labels: bool | None = False, **kwargs: *<class 'transformers.processing_utils.ProcessingKwargs'>)
```

[来源](https://github.com/huggingface/transformers/blob/v5.23.0/src/transformers/models/vibevoice_asr/processing_vibevoice_asr.py#L87)

**参数：**

text (`Union[str, list[str]]`) : 要编码的序列或序列批次。每个序列可以是字符串或字符串列表（预分词字符串）。如果传入预分词输入，请设置 `is_split_into_words=True` 以避免与批量输入产生歧义。

audio (`Union[numpy.ndarray, torch.Tensor, collections.abc.Sequence[numpy.ndarray], collections.abc.Sequence[torch.Tensor]]`，*可选*)：要准备处理的音频或音频批次。每个音频可以是 NumPy 数组或 PyTorch 张量。对于 NumPy 数组/PyTorch 张量，每个音频的形状应为 (C, T)，其中 C 是通道数，T 是音频的采样长度。

output_labels (`bool`, *可选*, 默认值=False) : 是否返回用于训练的标签。

return_mm_token_type_ids (`bool`, *kwargs*) : 是否返回多模态 token 类型 ID，以指示多模态占位符 token 的位置。

return_text_replacement_offsets (`bool`, *kwargs*) : 是否返回每个多模态占位符及其替换项的字符偏移量。

return_tensors (`str` 或 [TensorType](/docs/transformers/v5.23.0/zh/internal/file_utils#transformers.TensorType)，*可选*)：如果设置，将返回特定框架的张量。可接受的值为： - `'pt'`：返回 PyTorch `torch.Tensor` 对象。 - `'np'`：返回 NumPy `np.ndarray` 对象。

- ****kwargs** ([ProcessingKwargs](/docs/transformers/v5.23.0/zh/main_classes/processors#transformers.ProcessingKwargs), *可选*) : 针对每种模态（文本、图像、视频、音频）的额外处理选项。模型特定参数已在上方列出；有关支持参数的完整列表，请参阅 TypedDict 类。

**返回：** [BatchFeature](/docs/transformers/v5.23.0/zh/main_classes/feature_extractor#transformers.BatchFeature)

一个包含分词文本（`input_ids`、`attention_mask`）和音频特征（`input_values`、`padding_mask`）的字典。

#### apply_transcription_request[[transformers.VibeVoiceAsrProcessor.apply_transcription_request]]

```python
apply_transcription_request(audio: typing.Union[str, list[str], numpy.ndarray, ForwardRef('torch.Tensor'), collections.abc.Sequence[numpy.ndarray], collections.abc.Sequence['torch.Tensor']], prompt: str | list[str] | None = None, tokenize: bool = True, **kwargs: *<class 'transformers.processing_utils.ProcessingKwargs'>)
```

[来源](https://github.com/huggingface/transformers/blob/v5.23.0/src/transformers/models/vibevoice_asr/processing_vibevoice_asr.py#L170)

**参数：**

audio (`str`、`list[str]`、`np.ndarray`、`torch.Tensor`、`list[np.ndarray]`、`list[torch.Tensor]`)：要转录的音频。字符串会被解释为本地路径或 URL，并由聊天模板加载器自动加载；NumPy 数组和 PyTorch 张量会被直接传递。

prompt (`str` 或 `list[str]`，*可选*) : 包含在用户轮次中作为额外上下文的自定义提示。列表的长度必须与批次相同。当为 `None` 时，不提供额外上下文。

tokenize (`bool`，*可选*，默认为 `True`)：是否对提示词进行分词并处理音频。如果为 `False`，则不会加载音频，提示词将以字符串形式返回，其中包含未展开的音频占位符。将它们与音频一起传递给处理器，`processor(text=prompts, audio=audio)`，将得到与 `tokenize=True` 相同的输出。

- ****kwargs** : 转发给 [apply_chat_template()](/docs/transformers/v5.23.0/zh/main_classes/processors#transformers.ProcessorMixin.apply_chat_template) 的额外关键字参数（例如 `text_kwargs`、`audio_kwargs` 等）。

**返回：** [BatchFeature](/docs/transformers/v5.23.0/zh/main_classes/feature_extractor#transformers.BatchFeature) 或 `list[str]`

当 `tokenize=False` 时，每个音频对应一个提示；否则，处理器会输出可直接传递给 [VibeVoiceAsrForConditionalGeneration.generate()](/docs/transformers/v5.23.0/zh/main_classes/text_generation#transformers.GenerationMixin.generate) 的结果。

无需手动编写聊天模板，即可为自动语音识别准备输入。

#### decode[[transformers.VibeVoiceAsrProcessor.decode]]

```python
decode(*args, return_format = 'raw', **kwargs)
```

[来源](https://github.com/huggingface/transformers/blob/v5.23.0/src/transformers/models/vibevoice_asr/processing_vibevoice_asr.py#L226)

**参数：**

return_format (`str`，*可选*，默认为 `"raw"`)：选项如下：- `"raw"`：返回来自分词器的原始解码字符串列表，不进行任何解析。- `"parsed"`：返回每个说话人话语的解析字典对象列表，包含时间戳。- `"transcription_only"`：返回提取的转录字符串列表。对于 `"parsed"` 和 `"transcription_only"`，`skip_special_tokens` 会被自动强制（硬编码）设置为 `True`。

将参数传递给 [decode()](/docs/transformers/v5.23.0/zh/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.decode)，并可选择性地解析类字典的输出。

VibeVoice ASR 以类似字典的格式输出转录结果，例如：
```
[
    'assistant
":0.0,"End":7.56,"Speaker":0,"Content":"text"}]

    'assistant
":0,"End":5.20,"Speaker":0,"Content":"text"}]

]
```

## VibeVoiceAsrModel[[transformers.VibeVoiceAsrModel]]

#### transformers.VibeVoiceAsrModel[[transformers.VibeVoiceAsrModel]]

```python
transformers.VibeVoiceAsrModel(config: VibeVoiceAsrConfig)
```

[来源](https://github.com/huggingface/transformers/blob/v5.23.0/src/transformers/models/vibevoice_asr/modeling_vibevoice_asr.py#L304)

**参数：**

config ([VibeVoiceAsrConfig](/docs/transformers/v5.23.0/zh/model_doc/vibevoice_asr#transformers.VibeVoiceAsrConfig))：模型配置类，包含模型的所有参数。通过配置文件进行初始化时，不会加载与模型关联的权重，仅加载配置。请查看 [from_pretrained()](/docs/transformers/v5.23.0/zh/main_classes/model#transformers.PreTrainedModel.from_pretrained) 方法以加载模型权重。

VibeVoice ASR 模型（声学分词器 + 语义分词器 + 多模态投影器 + 语言模型），不包含语言建模头。

该模型继承自 [PreTrainedModel](/docs/transformers/v5.23.0/zh/main_classes/model#transformers.PreTrainedModel)。请查阅父类文档，了解该库为所有模型实现的通用方法（例如下载或保存、调整输入嵌入大小、剪枝头等）。

该模型也是 PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) 的子类。请将其作为常规的 PyTorch Module 使用，有关通用用法和行为的所有事项，请参阅 PyTorch 文档。

#### forward[[transformers.VibeVoiceAsrModel.forward]]

```python
forward(input_ids: typing.Optional[torch.LongTensor] = None, attention_mask: typing.Optional[torch.Tensor] = None, past_key_values: transformers.cache_utils.Cache | None = None, inputs_embeds: typing.Optional[torch.Tensor] = None, input_values: typing.Optional[torch.Tensor] = None, padding_mask: typing.Optional[torch.BoolTensor] = None, sample: bool = True, **kwargs: *<class 'transformers.utils.generic.TransformersKwargs'>)
```

[来源](https://github.com/huggingface/transformers/blob/v5.23.0/src/transformers/models/vibevoice_asr/modeling_vibevoice_asr.py#L379)

**参数：**

input_ids (`torch.LongTensor`，形状为 `(batch_size, sequence_length)`，*可选*)：输入序列中 token 在词汇表中的索引。默认情况下，填充（padding）将被忽略。可以使用 [AutoTokenizer](/docs/transformers/v5.23.0/zh/model_doc/auto#transformers.AutoTokenizer) 获取索引。有关详细信息，请参阅 [PreTrainedTokenizer.encode()](/docs/transformers/v5.23.0/zh/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) 和 [PreTrainedTokenizer.__call__()](/docs/transformers/v5.23.0/zh/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__)。[什么是输入 ID？](../glossary#input-ids)

attention_mask (`torch.Tensor`，形状为 `(batch_size, sequence_length)`，*可选*)：用于避免对填充（padding）标记索引执行注意力机制的掩码。掩码值取自 `[0, 1]`： - 1 表示**未掩码**的标记， - 0 表示**已掩码**的标记。 [什么是注意力掩码？](../glossary#attention-mask)

past_key_values (`~cache_utils.Cache`，*可选*)：预计算的隐藏状态（自注意力块和交叉注意力块中的键和值），可用于加速顺序解码。这通常由模型在解码的先前阶段返回的 `past_key_values` 组成，前提是 `use_cache=True` 或 `config.use_cache=True`。 仅允许 [Cache](/docs/transformers/v5.23.0/zh/internal/generation_utils#transformers.Cache) 实例作为输入，请参阅我们的 [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache)。如果未传入 `past_key_values`，则默认初始化 [DynamicCache](/docs/transformers/v5.23.0/zh/internal/generation_utils#transformers.DynamicCache)。 模型将输出与输入相同的缓存格式。 如果使用 `past_key_values`，则用户预期仅输入未处理的 `input_ids`（即未向此模型提供其过去键值状态的那些），其形状为 `(batch_size, unprocessed_length)`，而不是形状为 `(batch_size, sequence_length)` 的所有 `input_ids`。

inputs_embeds (`torch.Tensor`，形状为 `(batch_size, sequence_length, hidden_size)`，*可选*)：可选地，你可以选择直接传入嵌入表示，而不是传入 `input_ids`。如果你希望比模型内部的嵌入查找矩阵更精细地控制如何将 `input_ids` 索引转换为对应的向量，这将非常有用。

input_values (`torch.Tensor`，形状为 `(batch_size, sequence_length)`，*可选*) : 输入原始语音波形的浮点值。可以通过将 `.flac` 或 `.wav` 音频文件加载为 `list[float]`、`numpy.ndarray` 或 `torch.Tensor` 类型的数组来获取这些值，*例如* 使用 torchcodec 库（`pip install torchcodec`）或 soundfile 库（`pip install soundfile`）。要将该数组准备为 `input_values`，应使用 [AutoProcessor](/docs/transformers/v5.23.0/zh/model_doc/auto#transformers.AutoProcessor) 进行填充并转换为 `torch.Tensor` 类型的张量。有关详细信息，请参阅 [VibeVoiceAsrProcessor.__call__()](/docs/transformers/v5.23.0/zh/model_doc/vibevoice_asr#transformers.VibeVoiceAsrProcessor.__call__)。

padding_mask (`torch.BoolTensor`，形状为 `(batch_size, sequence_length)`，*可选*)：用于避免对填充特征索引执行操作的掩码。

sample (`bool`, *可选*, 默认为 `True`) : 是否向声学潜变量添加高斯噪声，与原始模型一致（σ-VAE：噪声尺度按批次中的每个样本从 N(0, `vae_std`) 中抽取）。如果为 False，则直接使用编码器输出，因此输出是确定性的。

**返回：** `VibeVoiceAsrModelOutputWithPast` 或 `tuple(torch.Tensor)`

`VibeVoiceAsrModelOutputWithPast` 或 `torch.Tensor` 元组（如果传入 `return_dict=False` 或 `config.return_dict=False`），其包含的元素取决于配置（[VibeVoiceAsrConfig](/docs/transformers/v5.23.0/zh/model_doc/vibevoice_asr#transformers.VibeVoiceAsrConfig)）和输入。

[VibeVoiceAsrModel](/docs/transformers/v5.23.0/zh/model_doc/vibevoice_asr#transformers.VibeVoiceAsrModel) 的 forward 方法重写了 `__call__` 特殊方法。

尽管前向传播的实现需要在此函数中定义，但之后应调用 `Module` 实例，而不是直接调用此函数，因为前者会负责执行预处理和后处理步骤，而后者会静默地忽略这些步骤。

- **last_hidden_state**（形状为 `(batch_size, sequence_length, hidden_size)` 的 `torch.Tensor`）-- 模型最后一层输出的隐藏状态序列。

  如果使用了 `past_key_values`，则仅输出形状为 `(batch_size, 1, hidden_size)` 的序列的最后一个隐藏状态。
- **past_key_values**（`Cache`，*可选*，在传入 `use_cache=True` 或 `config.use_cache=True` 时返回）—— 它是一个 [Cache](/docs/transformers/v5.23.0/zh/internal/generation_utils#transformers.Cache) 实例。
- **hidden_states** (`tuple(torch.Tensor)`，*可选*，在传入 `output_hidden_states=True` 或 `config.output_hidden_states=True` 时返回) -- 一个 `torch.Tensor` 元组（如果模型具有嵌入层，则包含嵌入层的输出，以及每一层的输出），形状为 `(batch_size, sequence_length, hidden_size)`。

  模型在每一层输出处的隐藏状态，以及可选的初始嵌入输出。
- **attentions** (`tuple(torch.Tensor)`，*可选*，当传入 `output_attentions=True` 或 `config.output_attentions=True` 时返回) -- 一个 `torch.Tensor` 元组（每层一个），形状为 `(batch_size, num_heads, sequence_length, sequence_length)`。

  经过注意力 softmax 后的注意力权重，用于在自注意力头中计算加权平均值。
- **audio_hidden_states** (`torch.Tensor`，*可选*) -- 投影后的音频隐藏状态。

## VibeVoiceAsrForConditionalGeneration[[transformers.VibeVoiceAsrForConditionalGeneration]]

#### transformers.VibeVoiceAsrForConditionalGeneration[[transformers.VibeVoiceAsrForConditionalGeneration]]

```python
transformers.VibeVoiceAsrForConditionalGeneration(config: VibeVoiceAsrConfig)
```

[来源](https://github.com/huggingface/transformers/blob/v5.23.0/src/transformers/models/vibevoice_asr/modeling_vibevoice_asr.py#L435)

**参数：**

config ([VibeVoiceAsrConfig](/docs/transformers/v5.23.0/zh/model_doc/vibevoice_asr#transformers.VibeVoiceAsrConfig))：模型配置类，包含模型的所有参数。通过配置文件进行初始化时，不会加载与模型关联的权重，仅加载配置。请查看 [from_pretrained()](/docs/transformers/v5.23.0/zh/main_classes/model#transformers.PreTrainedModel.from_pretrained) 方法以加载模型权重。

配备预训练声学分词器和语言模型的 VibeVoice ASR 模型。

该模型继承自 [PreTrainedModel](/docs/transformers/v5.23.0/zh/main_classes/model#transformers.PreTrainedModel)。请查阅父类文档，了解该库为所有模型实现的通用方法（例如下载或保存、调整输入嵌入大小、剪枝头等）。

该模型也是 PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) 的子类。请将其作为常规的 PyTorch Module 使用，有关通用用法和行为的所有事项，请参阅 PyTorch 文档。

#### forward[[transformers.VibeVoiceAsrForConditionalGeneration.forward]]

```python
forward(input_ids: typing.Optional[torch.LongTensor] = None, attention_mask: typing.Optional[torch.Tensor] = None, past_key_values: transformers.cache_utils.Cache | None = None, inputs_embeds: typing.Optional[torch.Tensor] = None, input_values: typing.Optional[torch.Tensor] = None, padding_mask: typing.Optional[torch.BoolTensor] = None, labels: typing.Optional[torch.LongTensor] = None, logits_to_keep: typing.Union[int, torch.Tensor] = 0, sample: bool = True, **kwargs: *<class 'transformers.utils.generic.TransformersKwargs'>)
```

[来源](https://github.com/huggingface/transformers/blob/v5.23.0/src/transformers/models/vibevoice_asr/modeling_vibevoice_asr.py#L445)

**参数：**

input_ids (`torch.LongTensor`，形状为 `(batch_size, sequence_length)`，*可选*)：输入序列中 token 在词汇表中的索引。默认情况下，填充（padding）将被忽略。可以使用 [AutoTokenizer](/docs/transformers/v5.23.0/zh/model_doc/auto#transformers.AutoTokenizer) 获取索引。有关详细信息，请参阅 [PreTrainedTokenizer.encode()](/docs/transformers/v5.23.0/zh/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) 和 [PreTrainedTokenizer.__call__()](/docs/transformers/v5.23.0/zh/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__)。[什么是输入 ID？](../glossary#input-ids)

attention_mask (`torch.Tensor`，形状为 `(batch_size, sequence_length)`，*可选*)：用于避免对填充（padding）标记索引执行注意力机制的掩码。掩码值取自 `[0, 1]`： - 1 表示**未掩码**的标记， - 0 表示**已掩码**的标记。 [什么是注意力掩码？](../glossary#attention-mask)

past_key_values (`~cache_utils.Cache`，*可选*)：预计算的隐藏状态（自注意力块和交叉注意力块中的键和值），可用于加速顺序解码。这通常由模型在解码的先前阶段返回的 `past_key_values` 组成，前提是 `use_cache=True` 或 `config.use_cache=True`。 仅允许 [Cache](/docs/transformers/v5.23.0/zh/internal/generation_utils#transformers.Cache) 实例作为输入，请参阅我们的 [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache)。如果未传入 `past_key_values`，则默认初始化 [DynamicCache](/docs/transformers/v5.23.0/zh/internal/generation_utils#transformers.DynamicCache)。 模型将输出与输入相同的缓存格式。 如果使用 `past_key_values`，则用户预期仅输入未处理的 `input_ids`（即未向此模型提供其过去键值状态的那些），其形状为 `(batch_size, unprocessed_length)`，而不是形状为 `(batch_size, sequence_length)` 的所有 `input_ids`。

inputs_embeds (`torch.Tensor`，形状为 `(batch_size, sequence_length, hidden_size)`，*可选*)：可选地，你可以选择直接传入嵌入表示，而不是传入 `input_ids`。如果你希望比模型内部的嵌入查找矩阵更精细地控制如何将 `input_ids` 索引转换为对应的向量，这将非常有用。

input_values (`torch.Tensor`，形状为 `(batch_size, sequence_length)`，*可选*) : 输入原始语音波形的浮点值。可以通过将 `.flac` 或 `.wav` 音频文件加载为 `list[float]`、`numpy.ndarray` 或 `torch.Tensor` 类型的数组来获取这些值，*例如* 使用 torchcodec 库（`pip install torchcodec`）或 soundfile 库（`pip install soundfile`）。要将该数组准备为 `input_values`，应使用 [AutoProcessor](/docs/transformers/v5.23.0/zh/model_doc/auto#transformers.AutoProcessor) 进行填充并转换为 `torch.Tensor` 类型的张量。有关详细信息，请参阅 [VibeVoiceAsrProcessor.__call__()](/docs/transformers/v5.23.0/zh/model_doc/vibevoice_asr#transformers.VibeVoiceAsrProcessor.__call__)。

padding_mask (`torch.BoolTensor`，形状为 `(batch_size, sequence_length)`，*可选*)：用于避免对填充特征索引执行操作的掩码。

labels (`torch.LongTensor`，形状为 `(batch_size, sequence_length)`，*可选*)：用于计算掩码语言建模损失的标签。索引值应在 `[0, ..., config.vocab_size]` 范围内，或为 -100（参见 `input_ids` 的文档字符串）。索引值设为 `-100` 的 token 将被忽略（即被掩码），损失仅针对标签在 `[0, ..., config.vocab_size]` 范围内的 token 进行计算。

logits_to_keep (`Union[int, torch.Tensor]`，*可选*，默认为 `0`)：如果为 `int`，则计算最后 `logits_to_keep` 个 token 的 logits。如果为 `0`，则计算所有 `input_ids` 的 logits（特殊情况）。生成过程只需要最后一个 token 的 logits，仅针对该 token 进行计算可以节省内存，对于长序列或大词汇量而言，这种节省相当可观。如果为 `torch.Tensor`，则必须是一维张量，对应于序列长度维度中需要保留的索引。这在使用打包张量格式（将批次和序列长度合并为单一维度）时非常有用。

sample (`bool`, *可选*, 默认为 `True`) : 是否向声学潜变量添加高斯噪声，与原始模型一致（σ-VAE：噪声尺度按批次中的每个样本从 N(0, `vae_std`) 中抽取）。如果为 False，则直接使用编码器输出，因此输出是确定性的。

**返回：** `VibeVoiceAsrCausalLMOutputWithPast` 或 `tuple(torch.Tensor)`

`VibeVoiceAsrCausalLMOutputWithPast` 或 `torch.Tensor` 元组（如果传入 `return_dict=False` 或 `config.return_dict=False`），其包含的元素取决于配置（[VibeVoiceAsrConfig](/docs/transformers/v5.23.0/zh/model_doc/vibevoice_asr#transformers.VibeVoiceAsrConfig)）和输入。

[VibeVoiceAsrForConditionalGeneration](/docs/transformers/v5.23.0/zh/model_doc/vibevoice_asr#transformers.VibeVoiceAsrForConditionalGeneration) 的 forward 方法重写了 `__call__` 特殊方法。

尽管前向传播的实现需要在此函数中定义，但之后应调用 `Module` 实例，而不是直接调用此函数，因为前者会负责执行预处理和后处理步骤，而后者会静默地忽略这些步骤。

- **loss** (`torch.Tensor`，形状为 `(1,)`，*可选*，在提供 `labels` 时返回) -- 语言建模损失。
- **logits**（形状为 `(batch_size, sequence_length, config.vocab_size)` 的 `torch.Tensor`）-- 预测分数。
- **past_key_values**（`Cache`，*可选*）-- 缓存实例。
- **hidden_states** (`tuple[torch.Tensor]`，*可选*，在传入 `output_hidden_states=True` 或 `config.output_hidden_states=True` 时返回) -- 一个 `torch.Tensor` 元组（如果模型具有嵌入层，则包含嵌入层输出的一个张量，以及每个层输出的一个张量），形状为 `(batch_size, sequence_length, hidden_size)`。

  模型在每一层输出处的隐藏状态，以及可选的初始嵌入输出。
- **attentions** (`tuple[torch.Tensor]`，*可选*，当传入 `output_attentions=True` 或 `config.output_attentions=True` 时返回) -- 一个 `torch.Tensor` 元组（每层一个），形状为 `(batch_size, num_heads, sequence_length, sequence_length)`。

  经过注意力 softmax 后的注意力权重，用于在自注意力头中计算加权平均值。
- **audio_hidden_states** (`torch.Tensor`，*可选*) -- 投影后的音频隐藏状态。

示例：

```python
>>> from transformers import VibeVoiceAsrForConditionalGeneration, AutoProcessor

>>> model_id = "microsoft/VibeVoice-ASR-HF"
>>> processor = AutoProcessor.from_pretrained(model_id)
>>> model = VibeVoiceAsrForConditionalGeneration.from_pretrained(model_id, dtype="auto", device_map="auto")
```

#### get_audio_features[[transformers.VibeVoiceAsrForConditionalGeneration.get_audio_features]]

```python
get_audio_features(*args, **kwargs)
```

[来源](https://github.com/huggingface/transformers/blob/v5.23.0/src/transformers/models/vibevoice_asr/modeling_vibevoice_asr.py#L442)

