Rotary Positional Embeddings (RoPE)
RoPE was proposed by Su Jianlin and others. It is an absolute position encoding. Its basic idea is to enable the model to pay attention to relative positions through absolute position encoding.
RoPE was proposed by Su Jianlin and others. It is an absolute position encoding. Its basic idea is to enable the model to pay attention to relative positions through absolute position encoding.