Tag: Parameter-Efficient Fine-Tuning
All the articles with the tag "Parameter-Efficient Fine-Tuning".
-
Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization
Published: at 11:08 AM
86.94 🤔This paper introduces a fine-tuning strategy for LLMs that leverages the unequal importance of attention matrices and customized learning rates to enhance efficiency, demonstrating through theoretical analysis and experiments on GLUE benchmarks that fine-tuning only Wq and Wv with higher learning rates for Wv can match or exceed full fine-tuning performance with fewer parameters.
-
R-LoRA: Randomized Multi-Head LoRA for Efficient Multi-Task Learning
Published: at 11:25 AM 86.91 🤔R-LoRA通过多头随机化（包括多头Dropout和随机初始化）增强了LoRA在多任务学习中的性能，有效提升了任务特定知识的捕获能力，同时降低了GPU内存使用和训练时间。
-
ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Models
Published: at 11:21 AM 86.90 🤔本文提出 ALPS 算法，通过基于权重分布的参数对齐分布分数（sPAD）定位任务敏感注意力头并剪枝，仅更新 10% 的注意力参数即在通用、数学和代码任务上实现性能提升，同时展现头部可转移性和知识遗忘缓解效果。
-
Why Do More Experts Fail? A Theoretical Analysis of Model Merging
Published: at 11:34 AM 86.88 🤔本文通过理论分析揭示了模型融合性能随专家模型数量增加而饱和的原因，并提出Reparameterized Heavy-Tailed方法扩展参数空间覆盖范围，在多个基准任务上验证了其有效性。
-
Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models
Published: at 11:02 AM 86.87 🤔This paper introduces a systematic approach to enhance large reasoning models by aligning them with deduction, induction, and abduction meta-abilities through a three-stage pipeline of individual training, parameter merging, and domain-specific RL, achieving up to 4% performance gains over instruction-tuned baselines across math, coding, and science benchmarks.