DAPE Position Encoding Method

DAPE stands for Data-Adaptive Positional Encoding, a new positional encoding method proposed by Zheng Chuanyang and others from the Chinese University of Hong Kong. The research team also includes researchers from the National University of Singapore, Noah Lab, the University of Hong Kong, and Hong Kong Baptist University. This research was accepted by NeurIPS 2024, and the paper results are "DAPE: Data-Adaptive Positional Encoding for Length Extrapolation"

DAPE significantly improves the performance of the model when processing long texts by dynamically adjusting the position encoding to adapt to the input context and the learned fixed priors. It combines semantic information and position information, allowing the position encoding to be adaptively adjusted according to the input data, overcoming the limitations of traditional position encoding methods (such as absolute position encoding APE and relative position encoding RPE).

The core idea of DAPE is to use a two-layer neural network to parameterize the positional encoding so that it can be dynamically adjusted according to the input context. This architecture allows the positional encoding to be both adaptive and dependent on the input data. In natural language tasks, DAPE is designed to capture the complex relationship between tokens. By introducing the combination of semantic and positional information, DAPE greatly improves the performance of the Transformer model in long text processing.