From Fine-Tuning to Prompting: A Review of Adaptation Strategies for Large Language Models

Large language models have changed artificial intelligence by showing surprising skills - like learning from context. But their size, sometimes with hundreds of billions of parts, makes them tough to use for specific tasks. Training every part isn't practical - it takes too much power and space. So people look for smarter ways to adapt these systems. This article breaks down main methods: training on labeled data or instruction-following examples to shape behavior; using prompts instead of changing weights; and tweaking just a few parameters to save resources. We compare how each handles memory use, efficiency, downsides like forgetting past knowledge, and picking up bad shortcuts. We also look into newer methods - like RetICL, GOP, or how PEFT works in shifting setups such as Fed LLM and PECFT. In the end, we highlight key open challenges while pointing toward what's next, especially auto-building PEFT systems and creating stable, checkable logic paths.

Keywords:

Large Language Models (LLM) , Parameter-Efficient Fine-Tuning (PEFT) , Instruction Tuning (IT) , In-Context Learning (ICL) , Prompt Engineering , Adaptation Strategies , Catastrophic Forgetting , Shortcut Learning