Leveraging Self-Attention for Input-Dependent Soft Prompting in LLMs

The performance of large language models in domain-specific tasksnecessitates fine-tuning, which is computationally expensive and technicallychallenging. This paper focuses on parameter-efficient fine-tuning using softprompting, a promising approach that adapts pre-trained models to downstreamtasks by learning a small set of parameters. We propose a novel Input DependentSoft Prompting technique with a self-Attention Mechanism (ID-SPAM) thatgenerates soft prompts based on the input tokens and attends different tokenswith varying importance. Our method is simple and efficient, keeping the numberof trainable parameters small. We show the merits of the proposed approachcompared to state-of-the-art techniques on various tasks and show the improvedzero shot domain transfer capability.