This article <ahref="https://www.leewayhertz.com/parameter-efficient-fine-tuning/">AGuid to Parameter-efficient Fine-tuning (PEFT)</a> made a very goodsummary with nice drawings. There are some differences between itsexplanation with the original paper but the basic architecture is allgood.

<h2 id="prompt-tuning">Prompt Tuning</h2>Prefix tuning, prompt tuning, and p-tuning all prepend some vectorsas prefixes / soft prompts to the vector inputs to transformers. Theirgoal is to find a context that steers the language model towardgenerating text that solves a particular task.<h2 id="adapter"><ahref=”https://arxiv.org/pdf/1902.00751.pdf”>Adapter</a></h2><blockquote>Before Adapter, when performing vanilla fine-tuning, a modificationis made to the top layer of the network because the label spaces andlosses for the upstream and downstream tasks differ. Now, Adaptermodules perform more general architectural modifications to re-purpose apretrained network for a downstream task: it injects new layers into theoriginal network.In standard fine-tuning, the new top-layer and the original weightsare co-trained. In contrast, in Adapter tuning, the parameters of theoriginal network are frozen and therefore may be shared by manytasks.</blockquote>

<blockquote>Left: We add the adapter module twice to each Transformer layer:after the projection following multiheaded attention and after the twofeed-forward layers.Right: The adapter consists of a bottleneck which contains fewparameters relative to the attention and feedforward layers in theoriginal model. The adapter also contains a skip-connection. Duringadapter tuning, the green layers are trained on the downstream data,this includes the adapter, the layer normalization parameters, and thefinal classification layer</blockquote>Therefore, we can denote the Adapter layer as:<spanclass=”math display”>y = B(σ(Ax)) + xDefine bottleneck dimension <spanclass=”math inline”>r (to be consistent with LoRA, inthe original Adapter paper, this was <spanclass=”math inline”>m), so $A\in \R^{r \times d}, B \in \R^{d \times r}$. Including biases, weadd a total <spanclass=”math inline”>2dr + d + rparameters with <spanclass=”math inline”>r ≪ d.In initialization, we initialize the adapters to a near-identityfunction, so original network is unaffected when training starts.Adapter achieves similar results with only 1% needed parameters ascompared to full fine-tuning.<h2 id="lora-low-rank-adaptation-of-llm"><ahref=”https://arxiv.org/abs/2106.09685”>LoRA: Low-Rank Adaptation ofLLM</a></h2>Before LoRA, SOTA techniques have some drawbacks:<ul><li>Adapter Layers Introduce Inference Latency: Adapter layers have fewparameters, but “large neural networks rely on hardwareparallelism to keep the latency low, and adapter layershave to be processed sequentially. This makes adifference in the online inference setting where the batch size istypically as small as one.” I actually don’t understand how toparallelize an LLM inference even if without Adapter</li><li>Directly Optimizing the Prompt is Hard: Prompt tuning and prefixtuning both require adding a prefix (to either the input vector or tothe hidden vector in middle). In this way, it “reduces thesequence length available to process a downstream task, whichwe suspect makes tuning the prompt less performant compared to othermethods.” Its performance changes non-monotonically in trainableparameters, too. So it’s hard to optimize.</li></ul>LoRA’s architecture is simply a matrix multiplication - Adapterwithout non-linearity or skip connection. So instead of <spanclass=”math inline”>y = B(σ(Ax)) + x,we do <spanclass=”math inline”>Δy = BAx.There is one fundamental difference though: Adapter is an extra layeradded into the original network, but LoRA is a layeradded along side with the original network. That’s whyI have a delta in LoRA’s formula.In fact, LoRA was specifically designed for low-rank matrixmultiplication decomposition, note for any pretrained weight <spanclass=”math inline”>$W_0 \in \R^{d \times k}$ with <spanclass=”math inline”>y = W₀xand we want to update this weight matrix in fine-tuning: <spanclass=”math inline”>W = W₀ + ΔW,we define <spanclass=”math inline”>ΔW = BA,so <spanclass=”math inline”>W = W₀ + BA.This simple design yields a unimaginable good result: note matrix <spanclass=”math inline”>BA is also of dimension$\R^{d \times k}$. Therefore, we candirectly add this result matrix to the original matrix and inferencingwith LoRA, when we add this matrix in, gives us no additionalinference latency.Similar to Adapter, LoRA uses a random Gaussian initialization for Aand zero for B, so <spanclass=”math inline”>ΔWx = BAxis zero at the beginning of training and <spanclass=”math inline”>W = W₀ + ΔWyields the same result (identity) as before.Thanks to the matrix decomposition nature, we can apply LoRA to anymatrix multiplication in principle. However, we found that intransformers, imagine if we have 8 ranks to distribute, when <spanclass=”math inline”>r(W_q) = r(W_v) = 4or <spanclass=”math inline”>r(W_q) = r(W_k) = r(W_v) = r(W_o) = 2gives best result. The author also found that the fine-tune matrixactually has a very low rank, so in practice even if we set <spanclass=”math inline”>r = 1 can give good enough results.Go to Section 7 for more interesting experiments they conducted.In diffusion models, we use LoRA on the stable diffusion’s crossattention layer.