SliceGPT
Free while signed in. Answers cite the passages they came from.
Microsoft's SliceGPT is a post-training LLM compression technique that literally slices rows and columns out of weight matrices while preserving zero-shot quality.
Structured sparsification: Unlike unstructured pruning that zeros weights, SliceGPT replaces each weight matrix with a smaller dense matrix by removing whole rows and columns, giving actual speedups on standard hardware.
Embedding dimension reduction: The approach reduces the effective embedding dimension of the network, which shrinks activations and optimizer state along with weights.
20% parameter removal: Up to 20% of parameters can be removed from Llama 2 70B and Phi-2 while retaining most zero-shot performance on standard benchmarks.
Dense and fast: Because the resulting model is still dense and smaller, it runs faster on GPU without requiring specialized sparse kernels, unlike many competing pruning methods.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack