An Efficient Alternative to Self-Attention
This paper presents FFTNet, a framework that replaces costly self-attention with an adaptive spectral filtering technique based on the Fast Fourier Transform (FFT). Key components:
Ask this paper
Global token mixing via FFT – Instead of pairwise token attention, FFTNet uses frequency-domain transforms, cutting complexity from O(n²) to O(n log n) while preserving global context.
Adaptive spectral filtering – A learnable filter dynamically reweights Fourier coefficients, letting the model emphasize important frequency bands similarly to attention weights.
Complex-domain nonlinearity – A modReLU activation on the real and imaginary parts enriches representation, capturing higher-order interactions beyond linear transforms. Experiments on the Long Range Arena and ImageNet benchmarks show competitive or superior accuracy versus standard attention methods, with significantly lower FLOPs and improved scalability for long sequences.