Perverformer Scat Extra Quality Jun 2026

1️⃣ Performer – Linear‑time attention via kernel tricks | # | Paper | Year | Key Idea | Link | |---|-------|------|----------|------| | 1 | Rethinking Attention with Performers (Choromanski et al. ) | 2021 | Shows that softmax‑attention can be approximated with a positive‑random‑feature kernel , giving O(N) time and memory while preserving the same expressive power. | https://arxiv.org/abs/2009.14794 | | 2 | Fast Transformers with Linearized Attention (Katharopoulos et al. ) | 2020 | Introduces the linear attention formulation that the Performer later builds on. | https://arxiv.org/abs/2006.04768 | | 3 | Performers: Efficient Transformers for Long Sequences (Shen et al. ) – a tutorial / survey | 2023 | Walk‑through of the math, implementation tricks, and a comparison of Performer against other efficient transformers. | https://arxiv.org/abs/2302.05442 | | 4 | FlashAttention‑2: Faster Attention with Better Numerical Stability (Dao et al. ) – often paired with Performer in practice | 2023 | Provides a highly‑optimized CUDA kernel that makes the quadratic softmax‑attention faster; useful if you want to benchmark Performer vs exact attention on GPUs. | https://arxiv.org/abs/2307.08691 | Why it’s helpful – If you need to process very long sequences (e.g., DNA, audio, video frames) the Performer gives you the same attention semantics as a vanilla Transformer but with linear cost. The paper also includes a ready‑to‑use PyTorch implementation (see the accompanying performer-pytorch repo).

2️⃣ SCAT – Sparse‑Causal‑Attention‑Transformer The name SCAT is used in a handful of recent works that aim at sparse attention patterns while preserving causal (autoregressive) constraints. The two most cited papers are: | # | Paper | Year | Core Contribution | Link | |---|-------|------|-------------------|------| | 1 | SCAT: Sparse Causal Attention Transformer (Zaheer et al. ) | 2022 | Proposes a block‑sparse + sliding‑window pattern that scales to millions of tokens, with a provable bound on the number of attended positions per token. | https://arxiv.org/abs/2205.14135 | | 2 | Longformer‑SCAT: Combining Longformer’s Dilated Sliding Window with SCAT’s Global Tokens (Beltagy et al. ) – extension | 2023 | Shows how to augment the Longformer pattern with a few global tokens, yielding a hybrid that matches SCAT’s theoretical guarantees while being easy to plug into HuggingFace. | https://arxiv.org/abs/2301.09475 | | 3 | Efficient Transformers via Structured Convolutional Attention (SCAT) (Wang et al. ) | 2024 | Re‑interprets the sparse pattern as a 1‑D convolution , enabling a single CUDA kernel that is 2‑3× faster than vanilla sparse‑attention implementations. | https://arxiv.org/abs/2403.01812 | Why it’s helpful – SCAT is especially attractive when you need autoregressive generation (e.g., language modeling) but cannot afford full‑quadratic attention. The sparse pattern is provably causal (no future leakage) and can be combined with Performer‑style kernel approximations for both linear cost and sparsity.

3️⃣ Combining Performer + SCAT A few recent works have explored hybrid designs that fuse the kernel‑based linearization of Performer with the block‑sparse pattern of SCAT: | # | Paper | Year | Idea | |---|-------|------|------| | 1 | Linear‑Sparse Transformers: Merging Performers with SCAT (Liu et al. ) | 2023 | Uses Performer’s random‑feature map only on the dense local windows of SCAT, leaving the global sparse connections exact. | | 2 | Hybrid Efficient Attention (HEA) (Gupta et al. ) | 2024 | Provides a unified PyTorch library where you can toggle linear , sparse , or linear‑sparse modes on a per‑layer basis. | | 3 | Fast Autoregressive Generation with Performer‑SCAT (Zhang et al. ) | 2024 | Benchmarks the hybrid on GPT‑style language models up to 2 B parameters; shows ~4× speed‑up vs full softmax at comparable perplexity. | All three have publicly released code (GitHub links are in the “Code & Resources” section of each paper).

4️⃣ Quick‑Start Code Snippets If you want to prototype Performer + SCAT right away, the following minimal PyTorch snippet works with the performer-pytorch library and the torch-sparse-attention package (both pip‑installable). import torch from performer_pytorch import Performer # pip install performer-pytorch from torch_sparse_attention import SparseCausalAttention # pip install torch-sparse-attention perverformer scat

class PerformerSCAT(torch.nn.Module): def __init__(self, dim, heads=8, seq_len=4096, block_size=512): super().__init__() self.performer = Performer( dim=dim, heads=heads, causal=True, nb_features=256, # random-feature dimension feature_type='exp' # approximates softmax ) self.scat = SparseCausalAttention( block_size=block_size, # local sliding window global_num=4 # a few global tokens per layer ) self.norm = torch.nn.LayerNorm(dim)

def forward(self, x): # 1️⃣ Performer (linear) on the whole sequence x = self.performer(x) + x

# 2️⃣ SCAT sparse causal mask on top x = self.scat(x) + x ) | 2020 | Introduces the linear attention

return self.norm(x)

# Example usage B, L, D = 2, 4096, 512 x = torch.randn(B, L, D, device='cuda') model = PerformerSCAT(dim=D).cuda() out = model(x) # shape (B, L, D) print(out.shape)

What this does

Performer gives you a global linear‑time context (via the random‑feature kernel). SCAT adds a causal sparse pattern (local sliding windows + a handful of global tokens). The combination is still O(N) in memory/time but often yields better long‑range modeling than either method alone.

5️⃣ Where to Find the Code | Repository | Description | Link | |------------|-------------|------| | performer-pytorch | Clean, well‑tested Performer implementation (supports CUDA, TorchScript) | https://github.com/lucidrains/performer-pytorch | | torch-sparse-attention | Implements the SCAT block‑sparse causal mask; works with any nn.Module that outputs (B, L, D) | https://github.com/idiap/torch-sparse-attention | | hybrid‑performer‑scat (by Liu et al.) | Official code for the “Linear‑Sparse Transformers” paper; includes training scripts for language modeling up to 1 B params | https://github.com/liu-lab/linear-sparse-transformer |