Introduction

Triton is a language and compiler for parallel programming, designed to provide a Python-based programming environment for efficiently writing custom DNN computation kernels that can run at maximum throughput on GPU hardware.

This project is a complete Triton learning tutorial, covering all aspects from basic to advanced, including vector operations, matrix operations, layer normalization, attention mechanisms, and FP8 matrix multiplication.

Table of contents

1. Basic Operation Tutorial

1.1 Vector Addition

• 01-vector-add.cn.ipynb – An introductory tutorial to vector addition, introducing the basic Triton programming model.

2. Core Operator Tutorial

2.1 Fused Softmax

• 02-fused-softmax.cn.ipynb – Integrate Softmax operations to learn kernel fusion and reduction operations.

2.2 Matrix Multiplication

• 03-matrix-multiplication.cn.ipynb High-performance matrix multiplication implementation

2.3 Layer Normalization

• 05-layer-norm.cn.ipynb – Layer normalization operator implementation

3. Advanced Features Tutorial

3.1 Low-Memory Dropout

• 04-low-memory-dropout.cn.ipynb – Memory-optimized Dropout implementation

3.2 Fused Attention

• 06-fused-attention.cn.ipynb – Implementation of Transformer attention mechanism

3.3 Libdevice External Functions

• 07-extern-functions.cn.ipynb – Using the tl_extra.libdevice external library

3.4 Grouped GEMM

• 08-grouped-gemm.cn.ipynb – Grouped General Matrix Multiplication Implementation

3.5 Continuous FP8 Matrix Multiplication

• 09-persistent-matmul.cn.ipynb – Optimization of matrix multiplication with FP8 precision

3.6 Block Scaling Matrix Multiplication

• 10-block-scaled-matmul.cn.ipynb – Block scaling matrix multiplication implementation

Reference Resources

• Triton Official Documentation
• Triton GitHub repository
• Triton Official Tutorial