后端人工智能模型推理服务集群管理可观测性【免费下载链接】gpustackA GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.项目地址https://gitcode.com/gh_mirrors/gp/gpustack点击查看免费下载本篇技术指南基于 GPUStack 性能实验Performance Lab中 GLM-4.5-Air 在 NVIDIA A100 上的完整调优报告系统讲解在高并发请求场景下如何通过推理引擎选型、并行策略TP/EP、Attention 后端选择与 CUDA Graph 调优等手段将模型总吞吐量Total TPS提升最高 107.5% 的全过程。读者将掌握一套可复现的基准测试流程vLLM Bench CLI、逐项对比的实验方法以及 GPUStack 中 SGLang 后端的参数注入机制与内置 Benchmark 功能能够在自己的 A100 集群上直接落地验证。结论速览在 4 × NVIDIA A100 SXM 80GB 单机环境下将 GLM-4.5-Air 的吞吐量最大化推荐采用如下 SGLang 启动配置python3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air \ --tp-size 4 --ep-size 4 --tool-call-parser glm --reasoning-parser glm45以 vLLM 默认配置vllm serve zai-org/GLM-4.5-Air -tp 4无任何优化为基线在同等 GPU 数量下对比优化前后的基准结果Benchmark Casebaseline (vLLM without any optimizations)Optimized (scaled to same GPU count)ShareGPTTotal TPS: 3903.44Mean TPOT(ms): 104.59Total TPS: 6245.77(60.0%)Mean TPOT(ms): 309.31Short PromptTotal TPS: 6271.24Mean TPOT(ms): 153.00Total TPS: 13012.20(107.5%)Mean TPOT(ms): 1246.57Medium PromptTotal TPS: 8643.75Mean TPOT(ms): 204.31Total TPS: 10956.49(26.8%)Mean TPOT(ms): 165.34Long PromptTotal TPS: 6975.10Mean TPOT(ms): 267.38Total TPS: 9100.71(30.5%)Mean TPOT(ms): 124.68Very Long PromptTotal TPS: 6435.97Mean TPOT(ms): 305.47Total TPS: 7945.66(23.5%)Mean TPOT(ms): 276.41关于该结论需要注意以下几点边界本次基准测试并未覆盖所有可能的优化组合。例如实验选择默认配置下表现最优的推理引擎作为进一步调优的起点这种剪枝策略得到的是局部最优解未必是全局最优。还存在依赖具体用户场景的优化手段包括 max batch size、调度配置scheduling config、扩展 KV Cacheextended KV cache、CUDA Graph 等。本文结论可作为进一步定向优化的起点。所有测试均在特定的硬件与软件组合下完成随着推理引擎的版本演进结论可能被更新。优化目标与实验设计优化目标本次实验的核心目标非常明确在高并发请求场景下实现高吞吐。因此所有调优决策都以「总 Token 吞吐量Total Token throughputTPS」为主要衡量指标同时记录 TTFT首 Token 延迟、TPOT每个输出 Token 的耗时等辅助指标用于观察代价。实验环境模型zai-org/GLM-4.5-Air硬件单节点 4 × NVIDIA A100 SXM 80GB引擎版本vLLM: v0.11.0SGLang: v0.5.4.post2TensorRT-LLM: v1.2.0rc1基准测试数据集实验使用两类数据集覆盖真实会话流量与不同序列长度压力ShareGPT真实对话数据用于模拟常见的多轮会话场景。随机数据集固定随机种子保证可复现按输入/输出 token 长度分为四档Very long prompt32000 输入 tokens100 输出 tokensLong prompt4000 输入 tokens200 输出 tokensMedium prompt2000 输入 tokens100 输出 tokensShort prompt128 输入 tokens4 输出 tokens基准测试脚本实验统一使用vLLM bench CLI工具即vllm bench serve对模型进行压测命令如下# Prepare the ShareGPT dataset wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json # Benchmark on ShareGPT dataset vllm bench serve --model zai-org/GLM-4.5-Air --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000 # Benchmark on random dataset (fixed seed for reproducibility) vllm bench serve --model zai-org/GLM-4.5-Air --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42其中--num-prompts控制并发请求规模本次实验中并发峰值即为该值--seed 42固定随机序列长度采样保证多轮对比可复现。这也解释了为何在结果中可以看到Peak concurrent requests等于 1000 或 500。补充说明GPUStack 自身也内置了 Benchmarking 能力可以在模型实例运行健康后通过Benchmarks页面创建、查看、导出基准测试并由 worker 在专用的 benchmark 容器镜像中执行。这意味着在 GPUStack 平台上你可以用本文的配置先部署模型实例再直接通过平台内置基准验证吞吐表现而不必完全依赖 vLLM bench CLI。实验一推理引擎选型结论SGLang(6160.72 tok/s) vLLM (3903.44 tok/s) TensorRT-LLM (Not supported)在未做任何额外优化仅启用张量并行 TP4的前提下三种引擎在 ShareGPT 数据集上的表现差异显著。TensorRT-LLM 在该场景下不支持此模型直接排除SGLang 相比 vLLM 有约 57.8% 的总吞吐量优势因此被选为后续调优的起点引擎。vLLM 基线Serving script:vllm serve zai-org/GLM-4.5-Air -tp 4 Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 106.15 Total input tokens: 214465 Total generated tokens: 199904 Request throughput (req/s): 9.42 Output token throughput (tok/s): 1883.14 Peak output token throughput (tok/s): 3380.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 3903.44 ---------------Time to First Token---------------- Mean TTFT (ms): 38971.22 Median TTFT (ms): 37502.20 P99 TTFT (ms): 76635.16 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 104.59 Median TPOT (ms): 101.18 P99 TPOT (ms): 210.38 ---------------Inter-token Latency---------------- Mean ITL (ms): 91.67 Median ITL (ms): 87.56 P99 ITL (ms): 178.94 SGLangServing script:python3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air --tp-size 4 Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 67.25 Total input tokens: 214465 Total generated tokens: 199873 Request throughput (req/s): 14.87 Output token throughput (tok/s): 2971.88 Peak output token throughput (tok/s): 11709.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 6160.72 ---------------Time to First Token---------------- Mean TTFT (ms): 14772.52 Median TTFT (ms): 14625.90 P99 TTFT (ms): 22760.64 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 328.21 Median TPOT (ms): 129.31 P99 TPOT (ms): 2061.59 ---------------Inter-token Latency---------------- Mean ITL (ms): 109.79 Median ITL (ms): 67.14 P99 ITL (ms): 411.08 从源码层面看GPUStack 的 SGLang 后端实现 中--tp-size与--pp-size是自动推断注入的核心并行参数当用户未显式提供时GPUStack 根据所选 GPU 数量推断 TP 大小见 built-in-inference-backends.md 中 SGLang 一节。因此本文的手动启动命令与 GPUStack 平台内的参数注入机制是等价的--tp-size 4对应单节点 4 卡张量并行GPUStack 在多节点场景下还会自动组合--tp-size/--pp-size并注入--nnodes、--node-rank、--dist-init-addr等通信参数。实验二量化FP8结论不适用于 A100启动即崩溃。python3 -m sglang.launch_server --model zai-org/GLM-4.5-Air-FP8 --tp-size 4# Crash. FP8 not supported on A100FP8 需要硬件原生支持如 Hopper 架构的 H100 等而 A100Ampere 架构不具备 FP8 计算能力。对照仓库中同系列的 H100 实验文档 可以看到在同一模型下 H100 上启用 FP8 量化可带来约 27.7% 的吞吐收益——这恰恰印证了 overview 中的观察结论量化是最大化吞吐量的关键技术但高度依赖硬件与引擎的特定组合见 Inference Performance Tuning Overview。实验三并行策略PP / TPEP在 A100 上测试了两种并行策略。PPPipeline Parallelism——不支持python3 -m sglang.launch_server --model zai-org/GLM-4.5-Air --pp-size 8# Crash. PP not supported for this setup.在本次软硬件组合下流水线并行PP无法启动。从 GPUStack 源码看SGLang 后端 在单节点场景默认使用--tp-size承载并行度--pp-size更多用于多节点部署的组合推断单节点内 PP 的适用性取决于引擎与模型结构的支持情况。TP4 EP4Tensor Parallelism Expert Parallelism——小幅提升python3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air --tp-size 4 --ep-size 4 Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 66.34 Total input tokens: 214465 Total generated tokens: 199904 Request throughput (req/s): 15.07 Output token throughput (tok/s): 3013.15 Peak output token throughput (tok/s): 9933.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 6245.77 ---------------Time to First Token---------------- Mean TTFT (ms): 8445.43 Median TTFT (ms): 8321.65 P99 TTFT (ms): 15245.84 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 309.31 Median TPOT (ms): 130.15 P99 TPOT (ms): 1969.56 ---------------Inter-token Latency---------------- Mean ITL (ms): 112.44 Median ITL (ms): 75.24 P99 ITL (ms): 484.22 在纯 TP4 的 SGLang 起点6160.72 tok/s之上叠加 EP4专家并行使 ShareGPT 总吞吐量提升到 6245.77 tok/s1.4%。虽然绝对增益有限但 EP 将 MoE 专家层切分到多卡显著降低了 TTFT均值从 14772.52ms 降至 8445.43ms对首 Token 延迟友好的场景仍有价值。实验四Attention 后端选择SGLang 在该配置下的默认 Attention 后端为flashinfer。实验中对比了另外两种后端tritonpython3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air --tp-size 4 --ep-size 4 --attention-backend triton Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 68.84 Total input tokens: 214465 Total generated tokens: 199904 Request throughput (req/s): 14.53 Output token throughput (tok/s): 2904.08 Peak output token throughput (tok/s): 9443.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 6019.69 ---------------Time to First Token---------------- Mean TTFT (ms): 9348.81 Median TTFT (ms): 9050.60 P99 TTFT (ms): 17294.26 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 373.94 Median TPOT (ms): 141.96 P99 TPOT (ms): 2701.08 ---------------Inter-token Latency---------------- Mean ITL (ms): 120.58 Median ITL (ms): 74.19 P99 ITL (ms): 413.30 fa3FlashAttention 3python3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air --tp-size 4 --ep-size 4 --attention-backend fa3 Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 67.66 Total input tokens: 214465 Total generated tokens: 199904 Request throughput (req/s): 14.78 Output token throughput (tok/s): 2954.39 Peak output token throughput (tok/s): 8239.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 6123.98 ---------------Time to First Token---------------- Mean TTFT (ms): 8260.21 Median TTFT (ms): 8222.20 P99 TTFT (ms): 15115.68 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 310.40 Median TPOT (ms): 133.38 P99 TPOT (ms): 2082.45 ---------------Inter-token Latency---------------- Mean ITL (ms): 114.41 Median ITL (ms): 78.46 P99 ITL (ms): 397.40 结论默认的 flashinfer 后端在该场景下仍是最优选择6245.77 vs 6019.69 vs 6123.98。这与 Inference Performance Tuning Overview 中的观察一致vLLM/SGLang 通常基于硬件环境提供合理的默认内核选择绝大多数场景下默认 Attention 后端就是最合适的。值得留意的是GPUStack 的 SGLang 后端在特定条件下会注入--attention-backend参数例如多模态场景注入--mm-attention-backend以及根据模型类别选择torch_sdpa等见 sglang.py普通文本模型默认交由引擎自选这与本实验的结论方向一致。实验五CUDA Graph 调优python3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air --tp-size 4 --ep-size 4 --cuda-graph-max-bs 512 Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 66.38 Total input tokens: 214465 Total generated tokens: 199904 Request throughput (req/s): 15.06 Output token throughput (tok/s): 3011.43 Peak output token throughput (tok/s): 10146.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 6242.21 ---------------Time to First Token---------------- Mean TTFT (ms): 8458.72 Median TTFT (ms): 8006.49 P99 TTFT (ms): 15322.87 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 311.99 Median TPOT (ms): 131.91 P99 TPOT (ms): 1946.11 ---------------Inter-token Latency---------------- Mean ITL (ms): 112.67 Median ITL (ms): 73.66 P99 ITL (ms): 516.14 将 CUDA Graph 的最大 batch size 调至 512 后ShareGPT 总吞吐量为 6242.21 tok/s与默认值基本持平6245.77。CUDA Graph 属于与并发度强相关的调优项见 overview 的「Deeper Tuning」清单在高并发解码场景中通常可以降低内核启动开销但本次测试中未带来显著差异可以推断默认取值已接近该负载下的合理区间。各优化项贡献汇总Optimization OptionThroughput ImprovementEngine Selection57.8%Quantization-Parallelism1.4%Attention Backend-CUDA Graph Tuning-可以看到引擎选型vLLM → SGLang贡献了绝大部分收益57.8%其余维度在本场景下增益有限甚至无增益——这再次印证推理调优往往高度依赖「特定模型 × 特定量化 × 特定 GPU」的组合不存在放之四海而皆准的最优引擎。泛化验证其他基准场景为进一步验证优化配置在不同负载下的泛化能力实验将「优化配置」与「vLLM 基线」在四种随机序列长度场景32K / 4K / 2K / 128 input下进行了对比。基线vllm serve zai-org/GLM-4.5-Air -tp 4# random 32K input Serving Benchmark Result Successful requests: 100 Benchmark duration (s): 498.76 Total input tokens: 3200000 Total generated tokens: 9987 Request throughput (req/s): 0.20 Output token throughput (tok/s): 20.02 Peak output token throughput (tok/s): 218.00 Peak concurrent requests: 100.00 Total Token throughput (tok/s): 6435.97 ---------------Time to First Token---------------- Mean TTFT (ms): 248013.29 Median TTFT (ms): 248015.97 P99 TTFT (ms): 491884.32 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 305.47 Median TPOT (ms): 317.93 P99 TPOT (ms): 319.14 ---------------Inter-token Latency---------------- Mean ITL (ms): 303.40 Median ITL (ms): 309.84 P99 ITL (ms): 406.26 # random 4K input Serving Benchmark Result Successful requests: 500 Benchmark duration (s): 300.98 Total input tokens: 1999787 Total generated tokens: 99547 Request throughput (req/s): 1.66 Output token throughput (tok/s): 330.75 Peak output token throughput (tok/s): 1743.00 Peak concurrent requests: 500.00 Total Token throughput (tok/s): 6975.10 ---------------Time to First Token---------------- Mean TTFT (ms): 142573.31 Median TTFT (ms): 141422.49 P99 TTFT (ms): 289825.31 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 267.38 Median TPOT (ms): 295.72 P99 TPOT (ms): 300.72 ---------------Inter-token Latency---------------- Mean ITL (ms): 266.48 Median ITL (ms): 295.37 P99 ITL (ms): 307.18 # random 2K input Serving Benchmark Result Successful requests: 500 Benchmark duration (s): 121.41 Total input tokens: 999429 Total generated tokens: 50000 Request throughput (req/s): 4.12 Output token throughput (tok/s): 411.83 Peak output token throughput (tok/s): 1796.00 Peak concurrent requests: 500.00 Total Token throughput (tok/s): 8643.75 ---------------Time to First Token---------------- Mean TTFT (ms): 60300.00 Median TTFT (ms): 60039.46 P99 TTFT (ms): 116422.28 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 204.31 Median TPOT (ms): 224.99 P99 TPOT (ms): 225.56 ---------------Inter-token Latency---------------- Mean ITL (ms): 202.57 Median ITL (ms): 224.72 P99 ITL (ms): 230.94 # random 128 input Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 21.03 Total input tokens: 127881 Total generated tokens: 4000 Request throughput (req/s): 47.55 Output token throughput (tok/s): 190.21 Peak output token throughput (tok/s): 952.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 6271.24 ---------------Time to First Token---------------- Mean TTFT (ms): 16583.77 Median TTFT (ms): 16009.09 P99 TTFT (ms): 20726.61 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 153.00 Median TPOT (ms): 153.83 P99 TPOT (ms): 173.39 ---------------Inter-token Latency---------------- Mean ITL (ms): 114.74 Median ITL (ms): 153.44 P99 ITL (ms): 192.15 优化配置python3 -m sglang.launch_server --model GLM-4.5-Air --tp-size 4 --ep-size 4 --tool-call-parser glm --reasoning-parser glm45# random 32K input Serving Benchmark Result Successful requests: 100 Benchmark duration (s): 403.99 Total input tokens: 3200000 Total generated tokens: 9931 Request throughput (req/s): 0.25 Output token throughput (tok/s): 24.58 Peak output token throughput (tok/s): 332.00 Peak concurrent requests: 100.00 Total Token throughput (tok/s): 7945.66 ---------------Time to First Token---------------- Mean TTFT (ms): 200446.56 Median TTFT (ms): 199894.57 P99 TTFT (ms): 397275.29 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 276.41 Median TPOT (ms): 295.02 P99 TPOT (ms): 410.95 ---------------Inter-token Latency---------------- Mean ITL (ms): 271.13 Median ITL (ms): 32.39 P99 ITL (ms): 3555.22 # random 4K input Serving Benchmark Result Successful requests: 500 Benchmark duration (s): 230.67 Total input tokens: 1999787 Total generated tokens: 99437 Request throughput (req/s): 2.17 Output token throughput (tok/s): 431.09 Peak output token throughput (tok/s): 1680.00 Peak concurrent requests: 500.00 Total Token throughput (tok/s): 9100.71 ---------------Time to First Token---------------- Mean TTFT (ms): 109316.41 Median TTFT (ms): 113614.39 P99 TTFT (ms): 222368.09 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 124.68 Median TPOT (ms): 124.45 P99 TPOT (ms): 179.33 ---------------Inter-token Latency---------------- Mean ITL (ms): 123.34 Median ITL (ms): 59.70 P99 ITL (ms): 294.40 # random 2K input Serving Benchmark Result Successful requests: 500 Benchmark duration (s): 95.78 Total input tokens: 999429 Total generated tokens: 49964 Request throughput (req/s): 5.22 Output token throughput (tok/s): 521.66 Peak output token throughput (tok/s): 3160.00 Peak concurrent requests: 500.00 Total Token throughput (tok/s): 10956.49 ---------------Time to First Token---------------- Mean TTFT (ms): 45224.78 Median TTFT (ms): 44492.00 P99 TTFT (ms): 91909.36 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 165.34 Median TPOT (ms): 163.74 P99 TPOT (ms): 281.83 ---------------Inter-token Latency---------------- Mean ITL (ms): 162.06 Median ITL (ms): 63.35 P99 ITL (ms): 166.19 # random 128 input Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 10.14 Total input tokens: 127881 Total generated tokens: 4000 Request throughput (req/s): 98.67 Output token throughput (tok/s): 394.66 Peak output token throughput (tok/s): 2166.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 13012.20 ---------------Time to First Token---------------- Mean TTFT (ms): 5882.75 Median TTFT (ms): 5678.55 P99 TTFT (ms): 9321.13 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 1246.57 Median TPOT (ms): 1420.02 P99 TPOT (ms): 2379.65 ---------------Inter-token Latency---------------- Mean ITL (ms): 747.94 Median ITL (ms): 59.22 P99 ITL (ms): 6782.02 泛化结论场景基线 TPS优化后 TPS提升Very Long Prompt32K in / 100 out6435.977945.6623.5%Long Prompt4K in / 200 out6975.109100.7130.5%Medium Prompt2K in / 100 out8643.7510956.4926.8%Short Prompt128 in / 4 out6271.2413012.20107.5%优化配置在所有序列长度场景下均保持正向提升但提升比例差异显著23.5% ~ 107.5%——这符合 overview 中「性能良好的配置通常在不同 ISL/OSL 下都呈现正收益但提升比例可能差异很大」的通用观察。短 Prompt 场景128 in / 4 out收益最大且在此场景下优化配置的 TTFT 从 16583.77ms 大幅降至 5882.75ms而长 Prompt 场景由于 prefill 占比高吞吐提升相对温和。如何在 GPUStack 中落地这套配置本文推荐的调优配置可以直接在 GPUStack 平台内落地两条路径任选通过模型部署表单在 GPUStack 的模型部署页面选择 SGLang 作为推理后端并在backend_parameters中填入--tp-size 4 --ep-size 4 --tool-call-parser glm --reasoning-parser glm45。GPUStack 的 SGLang 后端会解析并注入这些参数参数注入逻辑见 gpustack/worker/backends/sglang.py支持的参数说明见 built-in-inference-backends.md 中 SGLang 一节。对于大上下文模型GPUStack 会自动设置--context-length为 8192 以适配资源受限环境你可按需覆盖。通过平台内置基准验证模型实例运行健康后进入Benchmarks页面创建基准任务选择目标实例并配置压测参数即可复现本文的吞吐对比使用方法见 Benchmarking。需要注意的落地边界本文结论基于 A100Ampere的硬件特性FP8 量化路径在 A100 上不可用若部署到 H100 集群应参考同系列的 H100 优化文档——那里推荐启用 FP8 模型zai-org/GLM-4.5-Air-FP8收益结构与本报告完全不同。总结与进一步调优建议本次实验的核心可迁移经验可以归纳为三点引擎选型是吞吐优化的第一杠杆同一模型、同一 GPU不同引擎的默认性能差距可达 50% 以上优先用目标流量在自己设备上做基准验证而不是依赖通用印象。量化收益高度依赖硬件FP8 在 HopperH100上是吞吐利器在 AmpereA100上直接不可用量化收益的评估必须绑定目标 GPU。默认配置通常是好起点Attention 后端、CUDA Graph 等维度的默认取值在本场景已接近最优小幅调整收益有限更值得探索的是与业务负载强相关的参数——max batch size、调度配置、扩展 KV Cache、Torch compile 等详见 Inference Performance Tuning Overview 的 Deeper Tuning 清单它们需要结合真实的 ISL/OSL 分布、前缀复用率与并发模型进行针对性调优。如果你想在其他模型或 GPU 组合上重复这套方法论仓库的hack/perf/目录提供了基准数据提取、吞吐/延迟对比图生成等配套脚本可以帮助你快速产出可读的对比报告。赞分享后端人工智能模型推理服务集群管理可观测性【免费下载链接】gpustackA GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.项目地址https://gitcode.com/gh_mirrors/gp/gpustack点击查看免费下载相关推荐ADK Data Science Agent 的 SQL 后处理管道SQLite 转 BigQuery 的转译与纠错实战ADK Data Science Agent 的 SQL 后处理管道SQLite 转 BigQuery 的转译与纠错实战 本篇文章聚焦 Agent Devel后端人工智能模型推理服务集群管理可观测性GPUStack 性能实验室在 NVIDIA A100 上优化 GPT-OSS-20B 吞吐量的完整实战指南GPUStack 性能实验室在 NVIDIA A100 上优化 GPT OSS 20B 吞吐量的完整实战指南 导读 本文是 GPUStack 性能实验室 性后端人工智能模型推理服务集群管理可观测性GPUStack 性能实验室GLM-4.x 在 NVIDIA H200 上的吞吐量优化实战指南GPUStack 性能实验室GLM 4.x 在 NVIDIA H200 上的吞吐量优化实战指南 本文基于 GPUStack 性能实验室 docs/perfo后端人工智能模型推理服务集群管理可观测性上一篇7天搞定数据结构从链表到堆的完整学习指南 | GitHub加速计划下一篇HookLib² trampolines全解析三种跳转指令的实现与选择逻辑创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 
长河君 · 2026/10/4 10:38:31 · 阅读 1.2k GPUStack 性能实验：GLM-4.5-Air 在 NVIDIA A100 上的吞吐量优化实践
后端人工智能模型推理服务集群管理可观测性【免费下载链接】gpustackA GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.项目地址https://gitcode.com/gh_mirrors/gp/gpustack点击查看免费下载本篇技术指南基于 GPUStack 性能实验Performance Lab中 GLM-4.5-Air 在 NVIDIA A100 上的完整调优报告系统讲解在高并发请求场景下如何通过推理引擎选型、并行策略TP/EP、Attention 后端选择与 CUDA Graph 调优等手段将模型总吞吐量Total TPS提升最高 107.5% 的全过程。读者将掌握一套可复现的基准测试流程vLLM Bench CLI、逐项对比的实验方法以及 GPUStack 中 SGLang 后端的参数注入机制与内置 Benchmark 功能能够在自己的 A100 集群上直接落地验证。结论速览在 4 × NVIDIA A100 SXM 80GB 单机环境下将 GLM-4.5-Air 的吞吐量最大化推荐采用如下 SGLang 启动配置python3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air \ --tp-size 4 --ep-size 4 --tool-call-parser glm --reasoning-parser glm45以 vLLM 默认配置vllm serve zai-org/GLM-4.5-Air -tp 4无任何优化为基线在同等 GPU 数量下对比优化前后的基准结果Benchmark Casebaseline (vLLM without any optimizations)Optimized (scaled to same GPU count)ShareGPTTotal TPS: 3903.44Mean TPOT(ms): 104.59Total TPS: 6245.77(60.0%)Mean TPOT(ms): 309.31Short PromptTotal TPS: 6271.24Mean TPOT(ms): 153.00Total TPS: 13012.20(107.5%)Mean TPOT(ms): 1246.57Medium PromptTotal TPS: 8643.75Mean TPOT(ms): 204.31Total TPS: 10956.49(26.8%)Mean TPOT(ms): 165.34Long PromptTotal TPS: 6975.10Mean TPOT(ms): 267.38Total TPS: 9100.71(30.5%)Mean TPOT(ms): 124.68Very Long PromptTotal TPS: 6435.97Mean TPOT(ms): 305.47Total TPS: 7945.66(23.5%)Mean TPOT(ms): 276.41关于该结论需要注意以下几点边界本次基准测试并未覆盖所有可能的优化组合。例如实验选择默认配置下表现最优的推理引擎作为进一步调优的起点这种剪枝策略得到的是局部最优解未必是全局最优。还存在依赖具体用户场景的优化手段包括 max batch size、调度配置scheduling config、扩展 KV Cacheextended KV cache、CUDA Graph 等。本文结论可作为进一步定向优化的起点。所有测试均在特定的硬件与软件组合下完成随着推理引擎的版本演进结论可能被更新。优化目标与实验设计优化目标本次实验的核心目标非常明确在高并发请求场景下实现高吞吐。因此所有调优决策都以「总 Token 吞吐量Total Token throughputTPS」为主要衡量指标同时记录 TTFT首 Token 延迟、TPOT每个输出 Token 的耗时等辅助指标用于观察代价。实验环境模型zai-org/GLM-4.5-Air硬件单节点 4 × NVIDIA A100 SXM 80GB引擎版本vLLM: v0.11.0SGLang: v0.5.4.post2TensorRT-LLM: v1.2.0rc1基准测试数据集实验使用两类数据集覆盖真实会话流量与不同序列长度压力ShareGPT真实对话数据用于模拟常见的多轮会话场景。随机数据集固定随机种子保证可复现按输入/输出 token 长度分为四档Very long prompt32000 输入 tokens100 输出 tokensLong prompt4000 输入 tokens200 输出 tokensMedium prompt2000 输入 tokens100 输出 tokensShort prompt128 输入 tokens4 输出 tokens基准测试脚本实验统一使用vLLM bench CLI工具即vllm bench serve对模型进行压测命令如下# Prepare the ShareGPT dataset wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json # Benchmark on ShareGPT dataset vllm bench serve --model zai-org/GLM-4.5-Air --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000 # Benchmark on random dataset (fixed seed for reproducibility) vllm bench serve --model zai-org/GLM-4.5-Air --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42其中--num-prompts控制并发请求规模本次实验中并发峰值即为该值--seed 42固定随机序列长度采样保证多轮对比可复现。这也解释了为何在结果中可以看到Peak concurrent requests等于 1000 或 500。补充说明GPUStack 自身也内置了 Benchmarking 能力可以在模型实例运行健康后通过Benchmarks页面创建、查看、导出基准测试并由 worker 在专用的 benchmark 容器镜像中执行。这意味着在 GPUStack 平台上你可以用本文的配置先部署模型实例再直接通过平台内置基准验证吞吐表现而不必完全依赖 vLLM bench CLI。实验一推理引擎选型结论SGLang(6160.72 tok/s) vLLM (3903.44 tok/s) TensorRT-LLM (Not supported)在未做任何额外优化仅启用张量并行 TP4的前提下三种引擎在 ShareGPT 数据集上的表现差异显著。TensorRT-LLM 在该场景下不支持此模型直接排除SGLang 相比 vLLM 有约 57.8% 的总吞吐量优势因此被选为后续调优的起点引擎。vLLM 基线Serving script:vllm serve zai-org/GLM-4.5-Air -tp 4 Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 106.15 Total input tokens: 214465 Total generated tokens: 199904 Request throughput (req/s): 9.42 Output token throughput (tok/s): 1883.14 Peak output token throughput (tok/s): 3380.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 3903.44 ---------------Time to First Token---------------- Mean TTFT (ms): 38971.22 Median TTFT (ms): 37502.20 P99 TTFT (ms): 76635.16 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 104.59 Median TPOT (ms): 101.18 P99 TPOT (ms): 210.38 ---------------Inter-token Latency---------------- Mean ITL (ms): 91.67 Median ITL (ms): 87.56 P99 ITL (ms): 178.94 SGLangServing script:python3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air --tp-size 4 Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 67.25 Total input tokens: 214465 Total generated tokens: 199873 Request throughput (req/s): 14.87 Output token throughput (tok/s): 2971.88 Peak output token throughput (tok/s): 11709.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 6160.72 ---------------Time to First Token---------------- Mean TTFT (ms): 14772.52 Median TTFT (ms): 14625.90 P99 TTFT (ms): 22760.64 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 328.21 Median TPOT (ms): 129.31 P99 TPOT (ms): 2061.59 ---------------Inter-token Latency---------------- Mean ITL (ms): 109.79 Median ITL (ms): 67.14 P99 ITL (ms): 411.08 从源码层面看GPUStack 的 SGLang 后端实现 中--tp-size与--pp-size是自动推断注入的核心并行参数当用户未显式提供时GPUStack 根据所选 GPU 数量推断 TP 大小见 built-in-inference-backends.md 中 SGLang 一节。因此本文的手动启动命令与 GPUStack 平台内的参数注入机制是等价的--tp-size 4对应单节点 4 卡张量并行GPUStack 在多节点场景下还会自动组合--tp-size/--pp-size并注入--nnodes、--node-rank、--dist-init-addr等通信参数。实验二量化FP8结论不适用于 A100启动即崩溃。python3 -m sglang.launch_server --model zai-org/GLM-4.5-Air-FP8 --tp-size 4# Crash. FP8 not supported on A100FP8 需要硬件原生支持如 Hopper 架构的 H100 等而 A100Ampere 架构不具备 FP8 计算能力。对照仓库中同系列的 H100 实验文档 可以看到在同一模型下 H100 上启用 FP8 量化可带来约 27.7% 的吞吐收益——这恰恰印证了 overview 中的观察结论量化是最大化吞吐量的关键技术但高度依赖硬件与引擎的特定组合见 Inference Performance Tuning Overview。实验三并行策略PP / TPEP在 A100 上测试了两种并行策略。PPPipeline Parallelism——不支持python3 -m sglang.launch_server --model zai-org/GLM-4.5-Air --pp-size 8# Crash. PP not supported for this setup.在本次软硬件组合下流水线并行PP无法启动。从 GPUStack 源码看SGLang 后端 在单节点场景默认使用--tp-size承载并行度--pp-size更多用于多节点部署的组合推断单节点内 PP 的适用性取决于引擎与模型结构的支持情况。TP4 EP4Tensor Parallelism Expert Parallelism——小幅提升python3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air --tp-size 4 --ep-size 4 Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 66.34 Total input tokens: 214465 Total generated tokens: 199904 Request throughput (req/s): 15.07 Output token throughput (tok/s): 3013.15 Peak output token throughput (tok/s): 9933.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 6245.77 ---------------Time to First Token---------------- Mean TTFT (ms): 8445.43 Median TTFT (ms): 8321.65 P99 TTFT (ms): 15245.84 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 309.31 Median TPOT (ms): 130.15 P99 TPOT (ms): 1969.56 ---------------Inter-token Latency---------------- Mean ITL (ms): 112.44 Median ITL (ms): 75.24 P99 ITL (ms): 484.22 在纯 TP4 的 SGLang 起点6160.72 tok/s之上叠加 EP4专家并行使 ShareGPT 总吞吐量提升到 6245.77 tok/s1.4%。虽然绝对增益有限但 EP 将 MoE 专家层切分到多卡显著降低了 TTFT均值从 14772.52ms 降至 8445.43ms对首 Token 延迟友好的场景仍有价值。实验四Attention 后端选择SGLang 在该配置下的默认 Attention 后端为flashinfer。实验中对比了另外两种后端tritonpython3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air --tp-size 4 --ep-size 4 --attention-backend triton Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 68.84 Total input tokens: 214465 Total generated tokens: 199904 Request throughput (req/s): 14.53 Output token throughput (tok/s): 2904.08 Peak output token throughput (tok/s): 9443.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 6019.69 ---------------Time to First Token---------------- Mean TTFT (ms): 9348.81 Median TTFT (ms): 9050.60 P99 TTFT (ms): 17294.26 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 373.94 Median TPOT (ms): 141.96 P99 TPOT (ms): 2701.08 ---------------Inter-token Latency---------------- Mean ITL (ms): 120.58 Median ITL (ms): 74.19 P99 ITL (ms): 413.30 fa3FlashAttention 3python3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air --tp-size 4 --ep-size 4 --attention-backend fa3 Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 67.66 Total input tokens: 214465 Total generated tokens: 199904 Request throughput (req/s): 14.78 Output token throughput (tok/s): 2954.39 Peak output token throughput (tok/s): 8239.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 6123.98 ---------------Time to First Token---------------- Mean TTFT (ms): 8260.21 Median TTFT (ms): 8222.20 P99 TTFT (ms): 15115.68 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 310.40 Median TPOT (ms): 133.38 P99 TPOT (ms): 2082.45 ---------------Inter-token Latency---------------- Mean ITL (ms): 114.41 Median ITL (ms): 78.46 P99 ITL (ms): 397.40 结论默认的 flashinfer 后端在该场景下仍是最优选择6245.77 vs 6019.69 vs 6123.98。这与 Inference Performance Tuning Overview 中的观察一致vLLM/SGLang 通常基于硬件环境提供合理的默认内核选择绝大多数场景下默认 Attention 后端就是最合适的。值得留意的是GPUStack 的 SGLang 后端在特定条件下会注入--attention-backend参数例如多模态场景注入--mm-attention-backend以及根据模型类别选择torch_sdpa等见 sglang.py普通文本模型默认交由引擎自选这与本实验的结论方向一致。实验五CUDA Graph 调优python3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air --tp-size 4 --ep-size 4 --cuda-graph-max-bs 512 Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 66.38 Total input tokens: 214465 Total generated tokens: 199904 Request throughput (req/s): 15.06 Output token throughput (tok/s): 3011.43 Peak output token throughput (tok/s): 10146.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 6242.21 ---------------Time to First Token---------------- Mean TTFT (ms): 8458.72 Median TTFT (ms): 8006.49 P99 TTFT (ms): 15322.87 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 311.99 Median TPOT (ms): 131.91 P99 TPOT (ms): 1946.11 ---------------Inter-token Latency---------------- Mean ITL (ms): 112.67 Median ITL (ms): 73.66 P99 ITL (ms): 516.14 将 CUDA Graph 的最大 batch size 调至 512 后ShareGPT 总吞吐量为 6242.21 tok/s与默认值基本持平6245.77。CUDA Graph 属于与并发度强相关的调优项见 overview 的「Deeper Tuning」清单在高并发解码场景中通常可以降低内核启动开销但本次测试中未带来显著差异可以推断默认取值已接近该负载下的合理区间。各优化项贡献汇总Optimization OptionThroughput ImprovementEngine Selection57.8%Quantization-Parallelism1.4%Attention Backend-CUDA Graph Tuning-可以看到引擎选型vLLM → SGLang贡献了绝大部分收益57.8%其余维度在本场景下增益有限甚至无增益——这再次印证推理调优往往高度依赖「特定模型 × 特定量化 × 特定 GPU」的组合不存在放之四海而皆准的最优引擎。泛化验证其他基准场景为进一步验证优化配置在不同负载下的泛化能力实验将「优化配置」与「vLLM 基线」在四种随机序列长度场景32K / 4K / 2K / 128 input下进行了对比。基线vllm serve zai-org/GLM-4.5-Air -tp 4# random 32K input Serving Benchmark Result Successful requests: 100 Benchmark duration (s): 498.76 Total input tokens: 3200000 Total generated tokens: 9987 Request throughput (req/s): 0.20 Output token throughput (tok/s): 20.02 Peak output token throughput (tok/s): 218.00 Peak concurrent requests: 100.00 Total Token throughput (tok/s): 6435.97 ---------------Time to First Token---------------- Mean TTFT (ms): 248013.29 Median TTFT (ms): 248015.97 P99 TTFT (ms): 491884.32 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 305.47 Median TPOT (ms): 317.93 P99 TPOT (ms): 319.14 ---------------Inter-token Latency---------------- Mean ITL (ms): 303.40 Median ITL (ms): 309.84 P99 ITL (ms): 406.26 # random 4K input Serving Benchmark Result Successful requests: 500 Benchmark duration (s): 300.98 Total input tokens: 1999787 Total generated tokens: 99547 Request throughput (req/s): 1.66 Output token throughput (tok/s): 330.75 Peak output token throughput (tok/s): 1743.00 Peak concurrent requests: 500.00 Total Token throughput (tok/s): 6975.10 ---------------Time to First Token---------------- Mean TTFT (ms): 142573.31 Median TTFT (ms): 141422.49 P99 TTFT (ms): 289825.31 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 267.38 Median TPOT (ms): 295.72 P99 TPOT (ms): 300.72 ---------------Inter-token Latency---------------- Mean ITL (ms): 266.48 Median ITL (ms): 295.37 P99 ITL (ms): 307.18 # random 2K input Serving Benchmark Result Successful requests: 500 Benchmark duration (s): 121.41 Total input tokens: 999429 Total generated tokens: 50000 Request throughput (req/s): 4.12 Output token throughput (tok/s): 411.83 Peak output token throughput (tok/s): 1796.00 Peak concurrent requests: 500.00 Total Token throughput (tok/s): 8643.75 ---------------Time to First Token---------------- Mean TTFT (ms): 60300.00 Median TTFT (ms): 60039.46 P99 TTFT (ms): 116422.28 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 204.31 Median TPOT (ms): 224.99 P99 TPOT (ms): 225.56 ---------------Inter-token Latency---------------- Mean ITL (ms): 202.57 Median ITL (ms): 224.72 P99 ITL (ms): 230.94 # random 128 input Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 21.03 Total input tokens: 127881 Total generated tokens: 4000 Request throughput (req/s): 47.55 Output token throughput (tok/s): 190.21 Peak output token throughput (tok/s): 952.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 6271.24 ---------------Time to First Token---------------- Mean TTFT (ms): 16583.77 Median TTFT (ms): 16009.09 P99 TTFT (ms): 20726.61 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 153.00 Median TPOT (ms): 153.83 P99 TPOT (ms): 173.39 ---------------Inter-token Latency---------------- Mean ITL (ms): 114.74 Median ITL (ms): 153.44 P99 ITL (ms): 192.15 优化配置python3 -m sglang.launch_server --model GLM-4.5-Air --tp-size 4 --ep-size 4 --tool-call-parser glm --reasoning-parser glm45# random 32K input Serving Benchmark Result Successful requests: 100 Benchmark duration (s): 403.99 Total input tokens: 3200000 Total generated tokens: 9931 Request throughput (req/s): 0.25 Output token throughput (tok/s): 24.58 Peak output token throughput (tok/s): 332.00 Peak concurrent requests: 100.00 Total Token throughput (tok/s): 7945.66 ---------------Time to First Token---------------- Mean TTFT (ms): 200446.56 Median TTFT (ms): 199894.57 P99 TTFT (ms): 397275.29 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 276.41 Median TPOT (ms): 295.02 P99 TPOT (ms): 410.95 ---------------Inter-token Latency---------------- Mean ITL (ms): 271.13 Median ITL (ms): 32.39 P99 ITL (ms): 3555.22 # random 4K input Serving Benchmark Result Successful requests: 500 Benchmark duration (s): 230.67 Total input tokens: 1999787 Total generated tokens: 99437 Request throughput (req/s): 2.17 Output token throughput (tok/s): 431.09 Peak output token throughput (tok/s): 1680.00 Peak concurrent requests: 500.00 Total Token throughput (tok/s): 9100.71 ---------------Time to First Token---------------- Mean TTFT (ms): 109316.41 Median TTFT (ms): 113614.39 P99 TTFT (ms): 222368.09 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 124.68 Median TPOT (ms): 124.45 P99 TPOT (ms): 179.33 ---------------Inter-token Latency---------------- Mean ITL (ms): 123.34 Median ITL (ms): 59.70 P99 ITL (ms): 294.40 # random 2K input Serving Benchmark Result Successful requests: 500 Benchmark duration (s): 95.78 Total input tokens: 999429 Total generated tokens: 49964 Request throughput (req/s): 5.22 Output token throughput (tok/s): 521.66 Peak output token throughput (tok/s): 3160.00 Peak concurrent requests: 500.00 Total Token throughput (tok/s): 10956.49 ---------------Time to First Token---------------- Mean TTFT (ms): 45224.78 Median TTFT (ms): 44492.00 P99 TTFT (ms): 91909.36 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 165.34 Median TPOT (ms): 163.74 P99 TPOT (ms): 281.83 ---------------Inter-token Latency---------------- Mean ITL (ms): 162.06 Median ITL (ms): 63.35 P99 ITL (ms): 166.19 # random 128 input Serving Benchmark Result Successful requests: 1000 Benchmark duration (s): 10.14 Total input tokens: 127881 Total generated tokens: 4000 Request throughput (req/s): 98.67 Output token throughput (tok/s): 394.66 Peak output token throughput (tok/s): 2166.00 Peak concurrent requests: 1000.00 Total Token throughput (tok/s): 13012.20 ---------------Time to First Token---------------- Mean TTFT (ms): 5882.75 Median TTFT (ms): 5678.55 P99 TTFT (ms): 9321.13 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 1246.57 Median TPOT (ms): 1420.02 P99 TPOT (ms): 2379.65 ---------------Inter-token Latency---------------- Mean ITL (ms): 747.94 Median ITL (ms): 59.22 P99 ITL (ms): 6782.02 泛化结论场景基线 TPS优化后 TPS提升Very Long Prompt32K in / 100 out6435.977945.6623.5%Long Prompt4K in / 200 out6975.109100.7130.5%Medium Prompt2K in / 100 out8643.7510956.4926.8%Short Prompt128 in / 4 out6271.2413012.20107.5%优化配置在所有序列长度场景下均保持正向提升但提升比例差异显著23.5% ~ 107.5%——这符合 overview 中「性能良好的配置通常在不同 ISL/OSL 下都呈现正收益但提升比例可能差异很大」的通用观察。短 Prompt 场景128 in / 4 out收益最大且在此场景下优化配置的 TTFT 从 16583.77ms 大幅降至 5882.75ms而长 Prompt 场景由于 prefill 占比高吞吐提升相对温和。如何在 GPUStack 中落地这套配置本文推荐的调优配置可以直接在 GPUStack 平台内落地两条路径任选通过模型部署表单在 GPUStack 的模型部署页面选择 SGLang 作为推理后端并在backend_parameters中填入--tp-size 4 --ep-size 4 --tool-call-parser glm --reasoning-parser glm45。GPUStack 的 SGLang 后端会解析并注入这些参数参数注入逻辑见 gpustack/worker/backends/sglang.py支持的参数说明见 built-in-inference-backends.md 中 SGLang 一节。对于大上下文模型GPUStack 会自动设置--context-length为 8192 以适配资源受限环境你可按需覆盖。通过平台内置基准验证模型实例运行健康后进入Benchmarks页面创建基准任务选择目标实例并配置压测参数即可复现本文的吞吐对比使用方法见 Benchmarking。需要注意的落地边界本文结论基于 A100Ampere的硬件特性FP8 量化路径在 A100 上不可用若部署到 H100 集群应参考同系列的 H100 优化文档——那里推荐启用 FP8 模型zai-org/GLM-4.5-Air-FP8收益结构与本报告完全不同。总结与进一步调优建议本次实验的核心可迁移经验可以归纳为三点引擎选型是吞吐优化的第一杠杆同一模型、同一 GPU不同引擎的默认性能差距可达 50% 以上优先用目标流量在自己设备上做基准验证而不是依赖通用印象。量化收益高度依赖硬件FP8 在 HopperH100上是吞吐利器在 AmpereA100上直接不可用量化收益的评估必须绑定目标 GPU。默认配置通常是好起点Attention 后端、CUDA Graph 等维度的默认取值在本场景已接近最优小幅调整收益有限更值得探索的是与业务负载强相关的参数——max batch size、调度配置、扩展 KV Cache、Torch compile 等详见 Inference Performance Tuning Overview 的 Deeper Tuning 清单它们需要结合真实的 ISL/OSL 分布、前缀复用率与并发模型进行针对性调优。如果你想在其他模型或 GPU 组合上重复这套方法论仓库的hack/perf/目录提供了基准数据提取、吞吐/延迟对比图生成等配套脚本可以帮助你快速产出可读的对比报告。赞分享后端人工智能模型推理服务集群管理可观测性【免费下载链接】gpustackA GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.项目地址https://gitcode.com/gh_mirrors/gp/gpustack点击查看免费下载相关推荐ADK Data Science Agent 的 SQL 后处理管道SQLite 转 BigQuery 的转译与纠错实战ADK Data Science Agent 的 SQL 后处理管道SQLite 转 BigQuery 的转译与纠错实战 本篇文章聚焦 Agent Devel后端人工智能模型推理服务集群管理可观测性GPUStack 性能实验室在 NVIDIA A100 上优化 GPT-OSS-20B 吞吐量的完整实战指南GPUStack 性能实验室在 NVIDIA A100 上优化 GPT OSS 20B 吞吐量的完整实战指南 导读 本文是 GPUStack 性能实验室 性后端人工智能模型推理服务集群管理可观测性GPUStack 性能实验室GLM-4.x 在 NVIDIA H200 上的吞吐量优化实战指南GPUStack 性能实验室GLM 4.x 在 NVIDIA H200 上的吞吐量优化实战指南 本文基于 GPUStack 性能实验室 docs/perfo后端人工智能模型推理服务集群管理可观测性上一篇7天搞定数据结构从链表到堆的完整学习指南 | GitHub加速计划下一篇HookLib² trampolines全解析三种跳转指令的实现与选择逻辑创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考