LLM LMDeploy vs vLLM: The Complete Playbook for Production Inference

Deploying a large language model at scale boils down to one question: which framework gives you the most tokens per second on your hardware, at the lowest cost, without sacrificing quality?

LMDeploy (by InternLM/ModelScope) and vLLM (by UC Berkeley) are the two most popular open-source inference engines for serving LLMs in production today. Both support quantization, both handle multi-GPU scaling, and both integrate with Hugging Face models. Yet they make fundamentally different architectural trade-offs—and picking the wrong one can cost you significant throughput, VRAM, or engineering effort.

This playbook cuts through the noise with benchmarks, decision criteria, and implementation checklists so you can ship the right stack for your workload.

Key Takeaways

• vLLM is the throughput champion for general-purpose serving—its PagedAttention system and continuous batching deliver industry-leading GPU utilization, especially under high concurrent load.
• LMDeploy dominates on NVIDIA + Ascend heterogeneity and supports FlashDecoding+ and weight caching out of the box, making it stronger for multi-node and heterogeneous clusters.
• Quantization: LMDeploy leads in depth, vLLM in ecosystem maturity. LMDeploy ships built-in AWQ/GPTQ/INT4/INT8/FP8 with zero-code switches; vLLM offers PTQ and QLoRA but requires more manual configuration for some pipelines.
• Multi-GPU and multi-node: LMDeploy wins on flexibility—it supports pipeline, tensor, and data parallelism natively and runs on both NVIDIA and Huawei Ascend GPUs.
• Latency-sensitive chat applications should start with vLLM; high-throughput batch inference and heterogeneous GPU farms should evaluate LMDeploy first.
• Both frameworks integrate with Ray, Triton, and OpenAI-compatible APIs—the choice often comes down to your existing infrastructure and model family.

Head-to-Head Comparison

When to Choose vLLM

vLLM is the right choice when your primary goal is maximum throughput on standard NVIDIA GPU clusters. It is the default recommendation for most production LLM services in North America and Europe.

Ideal vLLM Use Cases

• Chat/bot applications requiring low latency and high concurrency
• API services (OpenAI-compatible) on NVIDIA A100/H100/H200 clusters
• Budget-conscious projects that need to run the largest models (70B+) on the fewest GPUs
• Teams that want the widest third-party integrations (LangChain, LlamaIndex, Ray Serve, FastAPI)
• Multilingual or English-first model families (Llama, Mistral, Phi)

Quick vLLM Setup Checklist

• [ ] Install vLLM: pip install vllm
• [ ] Verify GPU availability: nvidia-smi and python -c "import vllm; print(vllm.__version__)"
• [ ] Run a serving container: vllm serve meta-llama/Llama-3.1-8B --dtype auto --tensor-parallel-size 2
• [ ] Test with the OpenAI-compatible client
• [ ] Set GPU_MEMORY_UTILIZATION=0.9 to maximize KV cache allocation
• [ ] Enable --enable-prefix-caching for repeated prompt prefixes
• [ ] Benchmark with a