Repository navigation
Releases
Releases · Dao-AILab/flash-attention
fa4-v4.0.0.beta39
What's Changed
- [mla] pad all dense 2CTA MQA head counts in-kernel by @liangel-02 in #3140
- [FA2] Fix flash_attn_with_kvcache with cache_leftpad and no cache_seqlens by @andrewliu2001 in #3092
- [CuTe, SM100] Preserve the 15-bit sm_107 SMEM descriptor address by @drisspg in #3029
- [CuTe, SM100] Backward: explicit work-tile and head coordinates by @Johnsonms in #3141
- [CuTe, SM100] Compile autotune candidates for sm_107a by @drisspg in #3121
- [CuTe, SM100] Make the dynamic varlen forward scheduler bind tiles late and self-reset by @drisspg in #3086
- [CuTe, SM100] Route padded varlen forward calls to the dynamic persistent scheduler by @drisspg in #3087
- [CuTe, SM100] Give BF16 2CTA hd128 softmax 192 registers by @drisspg in #3016
- [CuTe, SM100] Release softmax statistics per stage by default by @drisspg in #3017
- [CuTe, SM100] Group FP8 descales in a typed scales argument by @drisspg in #3027
- [CuTe, SM100] Unify hd256 backward (6/7): pack 2 query heads per tile and loop the KV head's remaining heads inside the CTA by @Johnsonms in #3059
- [FA2] Keep split-KV block steps and accumulator offsets in 64 bits by @0z5a in #3101
- [CuTe, Bwd, SM100] Fold the hdim64 LSE / dPsum subtractions into the S / dP MMA by @drisspg in #3098
- [FA2] Check cache_batch_idx length in mha_fwd_kvcache by @bhargavchintam in #3102
- [CuTe, SM100] hd256 keeps PackGQA when 2CTA is not possible by @Johnsonms in #3148
- [CuTe, SM100] Unify hd256 backward (7/7): make it the default and delete the dedicated kernels by @Johnsonms in #2968
- [CuTe, SM100] Add MXFP8 quantization and scale-layout helpers by @drisspg in #3018
- [CuTe, SM100] Add MXFP8 QK forward with BF16 PV by @drisspg in #3019
- [CuTe, SM100] hd256 backward follow-ups from #2968 by @Johnsonms in #3155
New Contributors
- @andrewliu2001 made their first contribution in #3092
- @0z5a made their first contribution in #3101
- @bhargavchintam made their first contribution in #3102
Full Changelog: fa4-v4.0.0.beta38...fa4-v4.0.0.beta39
fa4-v4.0.0.beta38
What's Changed
- [CuTe, SM100] Unify hd256 backward (5/7): cheaper deterministic dQ semaphore (overlapped release on the M=64 serial schedule, early acquire, relaxed release) by @Johnsonms in #3058
- [CuTe] Route forward through typed configs by @drisspg in #2733
- [CuTe, SM100] Add swap-AB split-KV decode by @ighoshsubho in #2973
- [CuTe, SM100] Run causal forward on CLC by default by @drisspg in #3062
- [CuTe] Route backward through typed configs by @drisspg in #3044
- [CuTe] Route MLA forward through typed configs by @drisspg in #3054
- [CuTe, SM100] Put the forward shared-memory layout in the config by @drisspg in #3056
- [CuTe] Add explicit forward config benchmark by @drisspg in #2735
- [CuTe] Tune GB300 forward scheduling by @drisspg in #2738
- [CuTe] Add explicit forward config autotuning by @drisspg in #2743
- [mla] expose kernel cta choice for autotuning by @liangel-02 in #3091
- [mla] pad dense 2CTA MQA heads to the 128-row tile for 65-127 heads by @liangel-02 in #3118
New Contributors
- @ighoshsubho made their first contribution in #2973
Full Changelog: fa4-v4.0.0.beta37...fa4-v4.0.0.beta38
fa4-v4.0.0.beta37
What's Changed
- [CuTe, Fwd, SM100] Prevent NaN contamination from unused V cache entries by @fusheng-ji in #2983
- [CuTe] Format config.py with ruff so the Lint workflow passes again by @WeiYedi in #3067
- [CuTe, Bwd, SM100] Bound the K/V TMA loads by seqlen_k so unused rows cannot poison dQ (packed varlen and dense seqused_k) by @yunweili3 in #3012
- [CuTe, SM90] Bound the K/V TMA loads by seqlen_k (fwd + bwd, packed varlen and dense seqused_k) by @yunweili3 in #3013
- [CuTe] Bound ragged K/V TMA loads with per-CTA descriptor updates instead of a wrap-around descriptor by @yunweili3 in #3071
- [CuTe, SM100] Unify hd256 backward (4/7): pack the query heads of a KV head into the Q tile (GQA) by @Johnsonms in #2967
- [CuTe, SM100] Fix 2CTA CLC tile coords and make forward work-tile coordinates logical by @drisspg in #3034
New Contributors
- @fusheng-ji made their first contribution in #2983
- @WeiYedi made their first contribution in #3067
Full Changelog: fa4-v4.0.0.beta36...fa4-v4.0.0.beta37
fa4-v4.0.0.beta36
What's Changed
- Fix paged KV-cache CUDA graph capture by @pkuwangh in #2984
- [CuTe, SM100] Unify hd256 backward (2/5): 64-row KV tile per CTA (M=64 TMEM layout) with GQA, varlen and deterministic support by @Johnsonms in #2965
- [Tests] CUDA graph capture/replay coverage for FA4 (CuTe) and FA2 entry points by @yunweili3 in #3007
- [CuTe, SM100] Unify hd256 backward (3/5): hdim 256 on FlashAttentionBackwardSm100 (opt-in) by @Johnsonms in #2966
- [CuTe] Add typed forward configurations by @drisspg in #2732
New Contributors
Full Changelog: fa4-v4.0.0.beta35...fa4-v4.0.0.beta36
v2.8.3.post2
Maintenance release of FlashAttention 2 on top of v2.8.3.post1. No kernel feature changes.
- Backport #2622:
torch.load(..., weights_only=True)inflash_attn/utils/distributed.pyandflash_attn/models(CVE-2026-31253 / GHSA-7g5w-pq96-8c5w). #2974 - Backport #2702: safe
tarfileextraction inhopper/setup.py(CVE-2026-62239 / GHSA-w9r7-4gwr-j959). #2974 - Backport #2260: fix int32 overflow in the FA2 backward
dq_accumoffsets, which caused an illegal memory access or NaN gradients whenbatch * round128(seqlen_q) * nheads * headdimexceeded 2^31 (#2987). #2994 setup.py: request cu13 prebuilt wheels for CUDA 13 torch.
fa4-v4.0.0.beta35
What's Changed
- bump version to 3.0.1 by @liangel-02 in #2992
- [CuTe, SM120] Support automatic num_splits by @bvolpato in #2758
New Contributors
Full Changelog: fa4-v4.0.0.beta34...fa4-v4.0.0.beta35
fa4-v4.0.0.beta34
What's Changed
- Fix Windows builds with CUDA 13 and PyTorch 2.15 by @tuananhlfc in #2872
- [CuTe] Allow apache-tvm-ffi 0.1.11 by @felipemello1 in #2948
- [CuTe, SM100] S/P ping-pong to overlap QK with softmax in decode by @drisspg in #2928
- [CuTe, SM100] Unify hd256 forward and recover the remaining stack by @drisspg in #2962
- [CuTe, SM100] Fix hd128 decode crash on SM103: cap other-WG registers at the launch count by @felipemello1 in #2950
- [Cute,Fwd,Sm100] hd256: S ping-pong on SM103, 2CTA for varlen Q and decode by @felipemello1 in #2951
- [CuTe, SM100] Allow deterministic=True in the hd256 backward by @felipemello1 in #2952
- [Cute, Sm100] bwd postprocess: load 2-CTA dQaccum directly into registers by @pchen7e2 in #2960
- [CuTe, SM100] Unify hd256 backward (1/5): tests first, named 2CTA schedules and TMEM widths by @Johnsonms in #2964
- [CuTe, SM100] Key MLA forward compilation on the KV head count by @drisspg in #2978
New Contributors
- @tuananhlfc made their first contribution in #2872
- @felipemello1 made their first contribution in #2948
- @pchen7e2 made their first contribution in #2960
Full Changelog: fa4-v4.0.0.beta33...fa4-v4.0.0.beta34
fa4-v4.0.0.beta33
What's Changed
- [CuTe, SM100] Sparse MLA bwd: keep P normalization and dS product in fp32 by @abcdabcd987 in #2893
- [CuTe, SM100] Sparse MLA training fwd: exact running softmax max by @abcdabcd987 in #2907
- [CuTe, SM100] Sparse MLA training: dpsum from an O residual by @abcdabcd987 in #2908
- [CuTe, SM100] Sparse MLA recompute-P: allow 64 Q heads by @abcdabcd987 in #2911
- [CuTe, SM100] Sparse MLA bwd preprocess: stop the lse_log2 store at the sequence's rows by @abcdabcd987 in #2912
- [CuTe, SM100] Write real zeros for block-sparse empty tiles by @pashu-cohere in #2906
- [CuTe, SM90] Fit forward tile to the block-sparse block size by @LiRunGuo in #2903
- [CuTe, SM90] Skip the causal/local mask on unmasked KV blocks in forward by @LiRunGuo in #2904
- [ROCm/CK] Fix num_splits heuristic, which could never return more than 1 by @Johnsonms in #2901
- Support seqused_q/seqused_k in SM100 HD256 backward by @Mellonta in #2891
- [CuTe, SM100] hd256 fwd: derive lengths and KV ranges from BlockInfo/SeqlenInfoQK by @drisspg in #2917
- [Cute, Sm100] 1CTA MLA forward (dense + sparse top-k), MLA dispatch heuristic, sparse MLA backward at any head count by @jayhshah in #2938
- Fix lint failure by @jayhshah in #2940
New Contributors
Full Changelog: fa4-v4.0.0.beta32...fa4-v4.0.0.beta33
fa4-v4.0.0.beta32
What's Changed
- [CuTe, SM100] Sparse MLA bwd: in-kernel recompute-P + token-chunked backward by @abcdabcd987 in #2816
- Build CUDA extension with C++20 on PyTorch 2.13+ by @zhang-keliang in #2879
- ci: add PyTorch 2.14 to the wheel build matrix by @Johnsonms in #2892
- ci: registry-free GPU job (runner-local SIF) + verify the provisioned overlay from a fresh session by @Johnsonms in #2894
- Compile with c++20 for pytorch 2.13+ by @cih9088 in #2899
- [ROCM] add FLASH_ATTENTION_USE_SYSTEM_AITER flag and commit bump by @micmelesse in #2900
- [CuTe, SM100] Support 1..128 Q heads in sparse MLA via in-kernel TMA padding by @drisspg in #2883
New Contributors
- @zhang-keliang made their first contribution in #2879
- @cih9088 made their first contribution in #2899
Full Changelog: fa4-v4.0.0.beta31...fa4-v4.0.0.beta32
fa4-v4.0.0.beta31
What's Changed
- Add an Attention Sink to Flash MLA for SM100 by @floatingtrees in #2768
- [CuTe, SM100] Stop the non-causal hdim 128 softmax warps spilling by @dawnop in #2869
- [CuTe, SM100] hd256 2-CTA: support seqused_q/seqused_k and the decoder paged-KV shape by @kzos in #2810
- [CuTe, SM100] Preserve hd256 output alignment for vectorized stores by @drisspg in #2880
- One more deprecation; by @drisspg in #2867
- [CuTe, SM100, HD256] optimize SM100 HD256 varlen scheduling and epilogues by @catwinee in #2807
- [CuTe, SM90] Fix swapped backward MMA layout and use 2-stage hdim256 config by @SuperGoodGame in #2888
- [CuTe, Bwd] Return zero gradients for empty Q/K workloads by @guoriyue in #2776
- [CuTe, Fwd] Fix PackGQA predication for padded head dimensions by @guoriyue in #2775
New Contributors
- @floatingtrees made their first contribution in #2768
- @dawnop made their first contribution in #2869
- @kzos made their first contribution in #2810
- @catwinee made their first contribution in #2807
- @SuperGoodGame made their first contribution in #2888
Full Changelog: fa4-v4.0.0.beta30...fa4-v4.0.0.beta31