S-LoRA
Free while signed in. Answers cite the passages they came from.
S-LoRA enables serving thousands of LoRA adapters concurrently on a single GPU through memory-paging and custom CUDA kernels.
Main-memory adapter pool: Stores all adapters in main memory and loads adapters for currently running queries into GPU memory on demand, dramatically increasing the adapter pool size.
Novel tensor parallelism: Introduces a tensor-parallelism strategy tailored for heterogeneous LoRA batches, where each query might use a different adapter.
4x throughput: Improves throughput by 4x compared to prior adapter-serving solutions at comparable latency.
Adapter scale: Enables serving several orders of magnitude more adapters on the same hardware - important for multi-tenant LoRA deployments and personalized fine-tuning services.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack