Llama-Adapter

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

Llama-Adapter is a PEFT method specifically designed for turning Llama into an instruction-following model. The Llama model is frozen and only a set of adaptation prompts prefixed to the input instruction tokens are learned. Since randomly initialized modules inserted into the model can cause the model to lose some of its existing knowledge, Llama-Adapter uses zero-initialized attention with zero gating to progressively add the instructional prompts to the model.

The abstract from the paper is:

We present LLaMA-Adapter, a lightweight adaption method to efficiently fine-tune LLaMA into an instruction-following model. Using 52K self-instruct demonstrations, LLaMA-Adapter only introduces 1.2M learnable parameters upon the frozen LLaMA 7B model, and costs less than one hour for fine-tuning on 8 A100 GPUs. Specifically, we adopt a set of learnable adaption prompts, and prepend them to the input text tokens at higher transformer layers. Then, a zero-init attention mechanism with zero gating is proposed, which adaptively injects the new instructional cues into LLaMA, while effectively preserves its pre-trained knowledge. With efficient training, LLaMA-Adapter generates high-quality responses, comparable to Alpaca with fully fine-tuned 7B parameters. Furthermore, our approach can be simply extended to multi-modal input, e.g., images, for image-conditioned LLaMA, which achieves superior reasoning capacity on ScienceQA. We release our code at https://github.com/ZrrSkywalker/LLaMA-Adapter.

When to use Llama-Adapter

Llama-Adapter is a good choice when:

You want a very small checkpoint. Only the adaption prompts and their gates are trained and saved, which amounts to adapter_layers * (adapter_len * hidden_size + 1) parameters.
You are instruction-tuning a supported model (Llama, Mistral, or GPT-2), the use case the paper focuses on.
You want to keep several task-specific adapters for the same base model and switch between them with set_adapter.

When not to use Llama-Adapter

Llama-Adapter may not be the best choice when:

Your model is not a Llama, Mistral, or GPT-2 model. Other architectures raise a ValueError. Consider LoRA instead.
You need zero inference overhead. The adapter cannot be merged into the base weights (merge_and_unload() is not supported), so each adapted layer attends over adapter_len additional tokens at inference time.
You want several adapters to be active at the same time. Only one adapter can be active, and adaption prompts cannot be stacked on the same attention module.

Benchmark overview

API

AdaptionPromptConfig

class peft.AdaptionPromptConfig

< source >

( task_type: Optional[Union[str, TaskType]] = Nonepeft_type: Optional[Union[str, PeftType]] = Noneauto_mapping: Optional[dict] = Nonepeft_version: Optional[str] = Nonebase_model_name_or_path: Optional[str] = Nonerevision: Optional[str] = Noneinference_mode: bool = Falsetarget_modules: str = Noneadapter_len: int = Noneadapter_layers: int = None )

Stores the configuration of an AdaptionPromptModel.

AdaptionPromptModel

class peft.AdaptionPromptModel

< source >

( modelconfigs: dictadapter_name: str )

Implements adaption prompts as described in https://huggingface.co/papers/2303.16199.

The top L attention modules are replaced with AdaptedAttention modules that wrap the original ones, but insert trainable prompts with gates (for zero init).

Notes on the multi-adapter pattern:

We store the states of different adapters by keeping a dictionary of AdaptedAttention modules indexed by adapter name.
Every time we switch adapters, we remove the modules of the currently active adapter from the model, store them in the dictionary, and replace them with the modules of the new adapter.
To avoid duplicated and potentially inconsistent state, the currently active adapter is always removed from the dictionary.
Disabling the adapter would also result in the modules being removed from the model.