S-LoRA
First page
Paper summary
S-LoRA enables serving thousands of LoRA adapters concurrently on a single GPU through memory-paging and custom CUDA kernels.
Ask this paper
01
Main-memory adapter pool: Stores all adapters in main memory and loads adapters for currently running queries into GPU memory on demand, dramatically increasing the adapter pool size.
02
Novel tensor parallelism: Introduces a tensor-parallelism strategy tailored for heterogeneous LoRA batches, where each query might use a different adapter.
03
4x throughput: Improves throughput by 4x compared to prior adapter-serving solutions at comparable latency.
04
Adapter scale: Enables serving several orders of magnitude more adapters on the same hardware - important for multi-tenant LoRA deployments and personalized fine-tuning services.