The paper Decoupled WeightDecay Regularization mainly introduces AdamW, which is the SOTAoptimizer since then. It investigates why Adam with L2 regularizationsometimes performs worse than SGD with L2 regularization. Itdemonstrates weight decay and L2 regularization, two things peopleusually draw an equal sign, are not the same. And it shows weight decayis the ultimate go-to choice.

<p>Weight decay and L2 regularization are equivalent in SGD when set L2regularizer $\lambda’ = \frac \lambda\alpha$, which is our common practice. The situation is morecomplicated with adaptive gradient algorithms like Adam. Adam performsmuch better with weight decay and the authors propose the new SOTAoptimizer AdamW (Adam with decoupled weight decay). All the conclusionsand main finding can be found in the first 2 pages of the paper andmostly in the Introduction section. I did not read the math.</p><p><ahref=”https://www.fast.ai/posts/2018-07-02-adam-weight-decay.html”>Thisblogpost from Fast.ai</a> demonstrates how the two methods are differentin code, a bit easier to understand than the paper which doesn’t providea comparision.</p><h2 id="weight-decay-in-transformers">Weight Decay in Transformers</h2><p>AdamW is the go-to optimizer for LLM these days. Researchers chose itbecause LLMs are hard to train and rarely overfit, and Adam is the bestchoice when convergence speed is considered (<ahref=”https://www.zhihu.com/question/519307910/answer/2384626354”>reference</a>).People have also found AdamW usually performs best with big weight decaycoefficient like 0.05 or 0.1 (<ahref=”https://www.zhihu.com/question/536185388”>zhihu question</a>, <ahref=”https://arxiv.org/abs/2010.11929”>ViT paper: Training &Fine-tuning section</a>)</p><p>When we apply weight decay in transformers, we apply it to all layersexcept LayerNorm and bias layers.</p><p>In <ahref=”https://github.com/karpathy/nanoGPT/blob/325be85d9be8c81b436728a420e85796c57dba7e/model.py#L268-L271”>nanoGPT</a>,Karpathy filtered them out using: