SliceGPT
Microsoft's SliceGPT is a post-training LLM compression technique that literally slices rows and columns out of weight matrices while preserving zero-shot quality.
Ask this paper
Structured sparsification: Unlike unstructured pruning that zeros weights, SliceGPT replaces each weight matrix with a smaller dense matrix by removing whole rows and columns, giving actual speedups on standard hardware.
Embedding dimension reduction: The approach reduces the effective embedding dimension of the network, which shrinks activations and optimizer state along with weights.
20% parameter removal: Up to 20% of parameters can be removed from Llama 2 70B and Phi-2 while retaining most zero-shot performance on standard benchmarks.
Dense and fast: Because the resulting model is still dense and smaller, it runs faster on GPU without requiring specialized sparse kernels, unlike many competing pruning methods.