Hugging Face's logo

Transformers documentation

零样本目标检测

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.2).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

零样本目标检测

传统上，用于目标检测的模型需要带标签的图像数据集进行训练，并且只能检测训练数据中定义的类别集合。

零样本目标检测是一种计算机视觉任务，旨在检测图像中的对象及其类别，且无需任何先验训练或类别知识。零样本目标检测模型以图像和候选类别列表作为输入，输出检测到的对象的边界框和标签。

Hugging Face 托管了许多此类开放词汇零样本目标检测器。

在本指南中，您将学习如何使用此类模型：

  • 根据文本提示检测对象
  • 用于批量目标检测
  • 用于图像引导的目标检测

在开始之前，请确保已安装所有必要的库：

pip install -q transformers

零样本目标检测流水线

尝试使用模型进行推理的最简单方法是将其用于 pipeline() 中。从 Hugging Face Hub 上的检查点 实例化一个用于零样本目标检测的 pipeline：

>>> from transformers import pipeline

>>> # Use any checkpoint from the hf.co/models?pipeline_tag=zero-shot-object-detection
>>> checkpoint = "iSEE-Laboratory/llmdet_large"
>>> detector = pipeline(model=checkpoint, task="zero-shot-object-detection")

接下来，选择一张您想要进行目标检测的图像。这里我们将使用 NASA Great Images 数据集中宇航员 Eileen Collins 的图像。

>>> from transformers.image_utils import load_image

>>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_1.png"
>>> image = load_image(url)
>>> image
Astronaut Eileen Collins

将图像和要查找的候选对象标签传入 pipeline。 这里我们直接传入图像；其他合适的选项包括图像的本地路径或图像 URL。我们还传入了所有希望查询图像中是否包含的物品的文本描述。

>>> predictions = detector(
...     image,
...     candidate_labels=["human face", "rocket", "nasa badge", "star-spangled banner"],
...     threshold=0.45,
... )
>>> predictions
[{'score': 0.8409242033958435,
  'label': 'human face',
  'box': {'xmin': 179, 'ymin': 74, 'xmax': 272, 'ymax': 179}},
 {'score': 0.7380027770996094,
  'label': 'rocket',
  'box': {'xmin': 353, 'ymin': 0, 'xmax': 466, 'ymax': 284}},
 {'score': 0.5850900411605835,
  'label': 'star-spangled banner',
  'box': {'xmin': 0, 'ymin': 0, 'xmax': 96, 'ymax': 511}},
 {'score': 0.5697067975997925,
  'label': 'human face',
  'box': {'xmin': 18, 'ymin': 15, 'xmax': 366, 'ymax': 511}},
 {'score': 0.47813931107521057,
  'label': 'star-spangled banner',
  'box': {'xmin': 353, 'ymin': 0, 'xmax': 459, 'ymax': 274}},
 {'score': 0.46597740054130554,
  'label': 'nasa badge',
  'box': {'xmin': 353, 'ymin': 0, 'xmax': 462, 'ymax': 279}},
 {'score': 0.4585932493209839,
  'label': 'nasa badge',
  'box': {'xmin': 132, 'ymin': 348, 'xmax': 208, 'ymax': 423}}]

让我们可视化预测结果：

>>> from PIL import ImageDraw

>>> draw = ImageDraw.Draw(image)

>>> for prediction in predictions:
...     box = prediction["box"]
...     label = prediction["label"]
...     score = prediction["score"]

...     xmin, ymin, xmax, ymax = box.values()
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="red", width=1)
...     draw.text((xmin, ymin), f"{label}: {round(score,2)}", fill="white")

>>> image
Visualized predictions on NASA image

手动实现基于文本提示的零样本目标检测

现在你已经了解了如何使用零样本目标检测流水线，让我们手动复现相同的结果。

首先，从Hugging Face Hub 上的检查点加载模型及其关联的处理器。 这里我们将使用与之前相同的检查点：

>>> from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection

>>> model = AutoModelForZeroShotObjectDetection.from_pretrained(checkpoint, device_map="auto")
>>> processor = AutoProcessor.from_pretrained(checkpoint)

让我们换一张不同的图片来变换一下花样。

>>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_3.png"
>>> image = load_image(url)
>>> image
Beach photo

使用处理器为模型准备输入。

>>> text_labels = ["hat", "book", "sunglasses", "camera"]
>>> inputs = processor(text=text_labels, images=image, return_tensors="pt").to(model.device)

将输入数据传入模型，进行后处理，并可视化结果。由于图像处理器在将图像送入模型之前对其进行了缩放，因此您需要使用 post_process_object_detection 方法，以确保预测的边界框相对于原始图像具有正确的坐标：

>>> import torch

>>> with torch.inference_mode():
...     outputs = model(**inputs)

>>> results = processor.post_process_grounded_object_detection(
...    outputs, threshold=0.50, target_sizes=[(image.height, image.width)], text_labels=text_labels,
... )[0]

>>> draw = ImageDraw.Draw(image)

>>> scores = results["scores"]
>>> text_labels = results["text_labels"]
>>> boxes = results["boxes"]

>>> for box, score, text_label in zip(boxes, scores, text_labels):
...     xmin, ymin, xmax, ymax = box
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="red", width=1)
...     draw.text((xmin, ymin), f"{text_label}: {round(score.item(),2)}", fill="white")

>>> image
Beach photo with detected objects

批处理

您可以传入多组图像和文本查询，以在若干图像中搜索不同的（或相同的）对象。 让我们同时使用一张宇航员图像和一张海滩图像。 对于批处理，您应该将文本查询作为嵌套列表传入处理器，并将图像作为 PIL 图像列表、PyTorch 张量或 NumPy 数组传入。

>>> url1 = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_1.png"
>>> url2 = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_3.png"
>>> images = [load_image(url1), load_image(url2)]
>>> text_queries = [
...     ["human face", "rocket", "nasa badge", "star-spangled banner"],
...     ["hat", "book", "sunglasses", "camera", "can"],
... ]
>>> inputs = processor(text=text_queries, images=images, return_tensors="pt", padding=True)

之前在后处理中，你传入的是单张图像的尺寸张量，但也可以传入元组；对于多张图像的情况，则可以传入元组列表。下面我们来为这两个示例创建预测结果，并可视化第二个示例（image_idx = 1）。

>>> with torch.no_grad():
>>>     outputs = model(**inputs)

>>> target_sizes = [(image.height, image.width) for image in images]
>>> results = processor.post_process_grounded_object_detection(
...     outputs, threshold=0.3, target_sizes=target_sizes, text_labels=text_labels,
... )

让我们可视化结果：

>>> image_idx = 1
>>> draw = ImageDraw.Draw(images[image_idx])

>>> scores = results[image_idx]["scores"].tolist()
>>> text_labels = results[image_idx]["text_labels"]
>>> boxes = results[image_idx]["boxes"].tolist()

>>> for box, score, text_label in zip(boxes, scores, text_labels):
>>>     xmin, ymin, xmax, ymax = box
>>>     draw.rectangle((xmin, ymin, xmax, ymax), outline="red", width=1)
>>>     draw.text((xmin, ymin), f"{text_label}: {round(score,2)}", fill="white")

>>> images[image_idx]
Beach photo with detected objects

图像引导的目标检测

除了使用文本查询进行零样本目标检测外，OWL-ViT 和 OWLv2 等模型还支持图像引导的目标检测。这意味着你可以使用图像查询在目标图像中查找相似的对象。

>>> from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection

>>> checkpoint = "google/owlv2-base-patch16-ensemble"
>>> model = AutoModelForZeroShotObjectDetection.from_pretrained(checkpoint, device_map="auto")
>>> processor = AutoProcessor.from_pretrained(checkpoint)

与文本查询不同，仅允许使用单张示例图像。

让我们以一张沙发上两只猫的图片作为目标图像，以一张单只猫的图片作为查询图像：

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image_target = Image.open(requests.get(url, stream=True).raw)

>>> query_url = "http://images.cocodataset.org/val2017/000000524280.jpg"
>>> query_image = Image.open(requests.get(query_url, stream=True).raw)

让我们快速看一下这些图像：

>>> import matplotlib.pyplot as plt

>>> fig, ax = plt.subplots(1, 2)
>>> ax[0].imshow(image_target)
>>> ax[1].imshow(query_image)
>>> fig.show()
Cats

在预处理步骤中，现在需要使用 query_images 来替代文本查询：

>>> inputs = processor(images=image_target, query_images=query_image, return_tensors="pt")

对于预测，请将输入传递给 image_guided_detection()，而不是传递给模型。像之前一样绘制预测结果，但现在没有标签。

>>> with torch.no_grad():
...     outputs = model.image_guided_detection(**inputs)
...     target_sizes = torch.tensor([image_target.size[::-1]])
...     results = processor.post_process_image_guided_detection(outputs=outputs, target_sizes=target_sizes)[0]

>>> draw = ImageDraw.Draw(image_target)

>>> scores = results["scores"].tolist()
>>> boxes = results["boxes"].tolist()

>>> for box, score in zip(boxes, scores):
...     xmin, ymin, xmax, ymax = box
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="white", width=4)

>>> image_target
Cats with bounding boxes
Update on GitHub

←实例分割 零样本图像分类→