diff --git a/docs/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation.mdx b/docs/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation.mdx index f2126dfa7a..939eb8ae23 100644 --- a/docs/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation.mdx @@ -26,10 +26,7 @@ First, launch the SGLang environment using the provided container image: ```shell Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily uses the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -export IMAGE=quay.io/ascend/sglang:main-cann9.1.0-a3 +export IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ diff --git a/docs/docs/hardware-platforms/ascend-npus/faq.mdx b/docs/docs/hardware-platforms/ascend-npus/faq.mdx index 9c7bc30f6f..165744d34e 100644 --- a/docs/docs/hardware-platforms/ascend-npus/faq.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/faq.mdx @@ -297,10 +297,13 @@ Replace model weights, e.g., switch Qwen/Qwen3.5-27B-FP8 to Eco-Tech/Qwen3.5-27B Docker images for Ascend NPU are available in two types: -{/* TODO: CANN 9.1.0 has no stable image yet, so the stable release entry below has no example tag for now. Once the stable image is published, add it back. */} - - **Stable release** — validated version with a specific tag. Recommended for production deployments. - **Daily build** — includes the latest development changes, e.g., `quay.io/ascend/sglang:main-cann9.1.0-a3`. Use this if you need the latest features or bug fixes that have not yet been included in a stable release. +{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and update the example image to the stable release once it is published. */} +CANN 9.1.0 has no stable release yet, so use the latest validated build instead, e.g., +`swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105`. These +`cann9.1.0--B` tags are validated builds, not stable releases. + If you encounter issues with a stable release, try switching to a daily build to see if the issue has been resolved in the latest development version. diff --git a/docs/docs/hardware-platforms/ascend-npus/getting-started/installation.mdx b/docs/docs/hardware-platforms/ascend-npus/getting-started/installation.mdx index 21e94bb0e7..3722679381 100644 --- a/docs/docs/hardware-platforms/ascend-npus/getting-started/installation.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/getting-started/installation.mdx @@ -295,31 +295,25 @@ Ensure sufficient disk space before pulling images. Each Docker image requires a 1. Download SGLang image -We publish both **stable releases** and **daily builds**. Choose a stable release tag if you prefer a validated version, or a daily build tag (e.g., `main-cann9.1.0-a3`, `main-cann9.1.0-950`) if you need the latest development changes. CANN 9.1.0 has no stable image yet, so use the daily build tag for now. +We publish both **stable releases** and **daily builds**. Choose a stable release tag if you prefer a validated version, or a daily build tag (e.g., `main-cann9.1.0-a3`, `main-cann9.1.0-950`) if you need the latest development changes. -Validated release builds are also published as `cann9.1.0--B` tags (e.g., `cann9.1.0-950-B080`). The exact component versions inside each release image are listed in the [image dependency manifest](https://ascend.github.io/sglang/) — see for example the `cann9.1.0-950-B100` entry for the 950PR/DT Series. +CANN 9.1.0 has no stable release yet, so use a validated build for now. The `cann9.1.0--B` tags (e.g., `cann9.1.0-a3-B105`, `cann9.1.0-950-B105`) are validated builds rather than stable releases; they stand in until a stable release is published. The exact component versions inside each build are listed in the [image dependency manifest](https://ascend.github.io/sglang/) — see for example the `cann9.1.0-950-B105` entry for the 950PR/DT Series. -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tags `main-cann9.1.0-a3` / `main-cann9.1.0-950`. Once the stable image is published, switch back to the stable release tag. */} +{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tags below to the stable release once it is published. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 ``` ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-950 +docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-950-B105 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/getting-started/quick_start.mdx b/docs/docs/hardware-platforms/ascend-npus/getting-started/quick_start.mdx index 8d54560ec5..4a24d7f148 100644 --- a/docs/docs/hardware-platforms/ascend-npus/getting-started/quick_start.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/getting-started/quick_start.mdx @@ -40,10 +40,7 @@ at least **30GB** of free space. If you need to download model weights, check th We publish both **stable releases** and **daily builds**. Choose a stable release tag if you prefer a validated version, -or a daily build tag (e.g., `main-cann9.1.0-a3`) if you need the latest development changes. CANN 9.1.0 has no stable -image yet, so use the daily build tag for now. - -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +or a daily build tag (e.g., `main-cann9.1.0-a3`) if you need the latest development changes. If you have already downloaded model weights to a local path (e.g., `/path/to/model`), mount the path into the container by @@ -53,10 +50,7 @@ adding `--volume /path/to/model:/path/to/model` to the `docker run` command belo ```shell Command -# TODO: CANN 9.1.0 has no stable image yet, so only the daily build tag is -# available for now. Once the stable image is published, restore the stable -# release option here. -export IMAGE=quay.io/ascend/sglang:main-cann9.1.0-a3 +export IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1.mdx index 1767d1fbd0..3cc12b037e 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1.mdx @@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for DeepSeek-R1 On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. + +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} +The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes. + + ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | @@ -125,7 +130,7 @@ do --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ - --attn-dp-size 8 \ + --enable-dp-attention --dp-size 8 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --reasoning-parser deepseek-r1 \ @@ -159,7 +164,7 @@ do --port 8001 \ --nnodes 1 \ --tp-size 16 \ - --attn-dp-size 16 \ + --enable-dp-attention --dp-size 16 \ --mem-fraction-static 0.9 \ --max-running-requests 416 \ --quantization modelslim \ @@ -300,7 +305,7 @@ python3 -m sglang.launch_server \ --max-prefill-tokens 3000 \ --moe-a2a-backend deepep \ --deepep-mode auto \ - --attn-dp-size 16 \ + --enable-dp-attention --dp-size 16 \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ @@ -427,7 +432,7 @@ do --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ - --attn-dp-size 2 \ + --enable-dp-attention --dp-size 2 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ @@ -464,7 +469,7 @@ do --node-rank $i \ --nnodes 2 \ --tp-size 32 \ - --attn-dp-size 16 \ + --enable-dp-attention --dp-size 16 \ --mem-fraction-static 0.75 \ --max-running-requests 32 \ --quantization modelslim \ @@ -631,7 +636,7 @@ do --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ - --attn-dp-size 4 \ + --enable-dp-attention --dp-size 4 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ @@ -669,7 +674,7 @@ do --node-rank $i \ --nnodes 2 \ --tp-size 32 \ - --attn-dp-size 32 \ + --enable-dp-attention --dp-size 32 \ --mem-fraction-static 0.82 \ --max-running-requests 1024 \ --quantization modelslim \ @@ -831,7 +836,7 @@ do --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ - --attn-dp-size 2 \ + --enable-dp-attention --dp-size 2 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ @@ -868,7 +873,7 @@ do --node-rank $i \ --nnodes 2 \ --tp-size 32 \ - --attn-dp-size 16 \ + --enable-dp-attention --dp-size 16 \ --mem-fraction-static 0.75 \ --max-running-requests 32 \ --quantization modelslim \ @@ -1031,7 +1036,7 @@ do --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ - --attn-dp-size 2 \ + --enable-dp-attention --dp-size 2 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ @@ -1068,7 +1073,7 @@ do --node-rank $i \ --nnodes 2 \ --tp-size 32 \ - --attn-dp-size 16 \ + --enable-dp-attention --dp-size 16 \ --mem-fraction-static 0.75 \ --max-running-requests 32 \ --quantization modelslim \ @@ -1230,7 +1235,7 @@ do --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ - --attn-dp-size 2 \ + --enable-dp-attention --dp-size 2 \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ @@ -1267,7 +1272,7 @@ do --node-rank $i \ --nnodes 2 \ --tp-size 32 \ - --attn-dp-size 8 \ + --enable-dp-attention --dp-size 8 \ --mem-fraction-static 0.75 \ --max-running-requests 32 \ --quantization modelslim \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2.mdx index b8a4cdb65a..d714999e74 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2.mdx @@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for DeepSeek-V3 On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. + +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} +The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes. + + ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | @@ -159,7 +164,7 @@ do --node-rank $i \ --nnodes 2 \ --tp 32 \ - --attn-dp-size 8 \ + --enable-dp-attention --dp-size 8 \ --ep 32 \ --moe-dense-tp-size 1 \ --enable-dp-lm-head \ @@ -355,7 +360,7 @@ do --node-rank $i \ --nnodes 2 \ --tp 32 \ - --attn-dp-size 8 \ + --enable-dp-attention --dp-size 8 \ --ep 32 \ --moe-dense-tp-size 1 \ --enable-dp-lm-head \ @@ -550,7 +555,7 @@ do --node-rank $i \ --nnodes 2 \ --tp 32 \ - --attn-dp-size 8 \ + --enable-dp-attention --dp-size 8 \ --ep 32 \ --moe-dense-tp-size 1 \ --enable-dp-lm-head \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6.mdx index 9356e08d79..3f6a928f52 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6.mdx @@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for Kimi-K2.6 o On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. + +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} +The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes. + + ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | @@ -119,7 +124,7 @@ do --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ - --attn-dp-size 32 \ + --enable-dp-attention --dp-size 32 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 \ @@ -876,7 +881,7 @@ do --tp-size 16 \ --mem-fraction-static 0.82 \ --max-running-requests 2 \ - --attn-dp-size 2 \ + --enable-dp-attention --dp-size 2 \ --enable-dp-lm-head \ --disable-radix-cache \ --enable-multimodal \ @@ -1011,7 +1016,7 @@ python3 -m sglang.launch_server \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ - --attn-dp-size 16 \ + --enable-dp-attention --dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 4 8 9 10 11 \ @@ -1120,7 +1125,7 @@ python3 -m sglang.launch_server \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ - --attn-dp-size 16 \ + --enable-dp-attention --dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 3 4 \ @@ -1234,7 +1239,7 @@ python3 -m sglang.launch_server \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ - --attn-dp-size 16 \ + --enable-dp-attention --dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 3 4 5 \ @@ -1262,7 +1267,6 @@ python -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ - --max-attempts 5 \ --max-concurrency 64 \ --num-prompts 256 \ --random-input-len 3500 \ @@ -1346,7 +1350,7 @@ python3 -m sglang.launch_server \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ - --attn-dp-size 16 \ + --enable-dp-attention --dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 4 8 11 12 13 \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_5_pro.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_5_pro.mdx index d51aa3e808..5eeb3afbbe 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_5_pro.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_5_pro.mdx @@ -10,6 +10,11 @@ This page focuses on the deployment of MiMo-V2.5-Pro (FP4) with DFlash speculati On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. + +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} +The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes. + + ### Model Deployment MiMo-V2.5-Pro-FP4-DFlash is an MXFP4-quantized checkpoint with a built-in DFlash draft model (located in the `dflash/` subdirectory of the weights). The following example deploys it in 1P1D mode (1 prefill node + 1 decode node, TP8 + DP2 per node). @@ -82,7 +87,7 @@ python3 -m sglang.launch_server \ --disaggregation-mode prefill --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8996 \ --cuda-graph-backend-prefill disabled \ - --attn-dp-size 2 --enable-dp-lm-head \ + --enable-dp-attention --dp-size 2 --enable-dp-lm-head \ --moe-a2a-backend deepep --deepep-mode normal ``` @@ -110,7 +115,7 @@ python3 -m sglang.launch_server \ --disaggregation-mode decode --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8996 \ --moe-a2a-backend deepep --deepep-mode low_latency \ - --attn-dp-size 2 --enable-dp-lm-head \ + --enable-dp-attention --dp-size 2 --enable-dp-lm-head \ --speculative-algorithm DFLASH \ --speculative-num-draft-tokens 8 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash.mdx index 377e538991..ce75d4c8da 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash.mdx @@ -10,31 +10,36 @@ This page focuses on optimal configuration and benchmark results for MiMo-V2-Fla On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. + +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} +The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes. + + ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 16k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1k-tpot-20ms) | -| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 32k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1k-tpot-20ms) | +| MiMo-V2-Flash | Ascend A3 Series Products | 16 | PD Disaggregation | 16k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-16p-in16k-out1k-tpot-20ms) | +| MiMo-V2-Flash | Ascend A3 Series Products | 16 | PD Disaggregation | 32k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-16p-in32k-out1k-tpot-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 16k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1-ttft-5s) | -| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 32k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1-ttft-5s) | +| MiMo-V2-Flash | Ascend A3 Series Products | 16 | PD Disaggregation | 16k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-16p-in16k-out1-ttft-5s) | +| MiMo-V2-Flash | Ascend A3 Series Products | 16 | PD Disaggregation | 32k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-16p-in32k-out1-ttft-5s) | ## Optimal Configuration -### MiMo-V2-Flash 1P1D 12P IN16K OUT1 TTFT 5s +### MiMo-V2-Flash 1P1D 16P IN16K OUT1 TTFT 5s **Model**: MiMo-V2-Flash **Hardware**: Ascend A3 Series Products -**Cards**: 12 +**Cards**: 16 **Deploy Mode**: PD Disaggregation @@ -166,7 +171,7 @@ do --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ --disable-radix-cache \ - --attn-dp-size 2 \ + --enable-dp-attention --dp-size 2 \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode low_latency @@ -211,13 +216,13 @@ python -m sglang.bench_serving \ --seed 1 ``` -### MiMo-V2-Flash 1P1D 12P IN16K OUT1K TPOT 20ms +### MiMo-V2-Flash 1P1D 16P IN16K OUT1K TPOT 20ms **Model**: MiMo-V2-Flash **Hardware**: Ascend A3 Series Products -**Cards**: 12 +**Cards**: 16 **Deploy Mode**: PD Disaggregation @@ -349,7 +354,7 @@ do --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ --disable-radix-cache \ - --attn-dp-size 2 \ + --enable-dp-attention --dp-size 2 \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode low_latency @@ -394,13 +399,13 @@ python -m sglang.bench_serving \ --seed 1 ``` -### MiMo-V2-Flash 1P1D 12P IN32K OUT1 TTFT 5s +### MiMo-V2-Flash 1P1D 16P IN32K OUT1 TTFT 5s **Model**: MiMo-V2-Flash **Hardware**: Ascend A3 Series Products -**Cards**: 12 +**Cards**: 16 **Deploy Mode**: PD Disaggregation @@ -532,7 +537,7 @@ do --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ --disable-radix-cache \ - --attn-dp-size 2 \ + --enable-dp-attention --dp-size 2 \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode low_latency @@ -577,13 +582,13 @@ python -m sglang.bench_serving \ --seed 1 ``` -### MiMo-V2-Flash 1P1D 12P IN32K OUT1K TPOT 20ms +### MiMo-V2-Flash 1P1D 16P IN32K OUT1K TPOT 20ms **Model**: MiMo-V2-Flash **Hardware**: Ascend A3 Series Products -**Cards**: 12 +**Cards**: 16 **Deploy Mode**: PD Disaggregation @@ -715,7 +720,7 @@ do --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ --disable-radix-cache \ - --attn-dp-size 2 \ + --enable-dp-attention --dp-size 2 \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode low_latency diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5.mdx index e63a327eac..042ac00ff1 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5.mdx @@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for MiniMax-M2. On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. + +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} +The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes. + + ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | @@ -312,7 +317,7 @@ python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 16 \ - --attn-dp-size 2 \ + --enable-dp-attention --dp-size 2 \ --mem-fraction-static 0.65 \ --max-running-requests 4 \ --reasoning-parser minimax-append-think \ @@ -421,7 +426,7 @@ python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 16 \ - --attn-dp-size 16 \ + --enable-dp-attention --dp-size 16 \ --mem-fraction-static 0.53 \ --max-running-requests 96 \ --disable-radix-cache \ @@ -525,7 +530,7 @@ python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 16 \ - --attn-dp-size 16 \ + --enable-dp-attention --dp-size 16 \ --mem-fraction-static 0.75 \ --max-running-requests 320 \ --disable-radix-cache \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b.mdx index c7941e71e1..1cda819603 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b.mdx @@ -11,6 +11,7 @@ On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes. diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b.mdx index 6a821b99ae..103a2efff1 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b.mdx @@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for Qwen3-30B-A On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. + +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} +The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes. + + ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | @@ -105,7 +110,7 @@ python3 -m sglang.launch_server \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 8 \ --tp-size 2 \ - --attn-dp-size 2 \ + --enable-dp-attention --dp-size 2 \ --mem-fraction-static 0.85 \ --cuda-graph-bs-decode 1 2 4 8 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 \ --dtype bfloat16 \ @@ -128,8 +133,7 @@ python -m sglang.bench_serving \ --random-input-len 1000 \ --random-output-len 100 \ --random-range-ratio 1 \ - --seed 1 \ - --max-attempts 4 + --seed 1 ``` ### Qwen3-30B-A3B W8A8 1P IN3K5 OUT1K5 10ms @@ -423,7 +427,6 @@ python -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ - --max-attempts 5 \ --max-concurrency 16 \ --num-prompts 16 \ --random-input-len 6144 \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b.mdx index 2a42f7d8ee..c8d18b3913 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b.mdx @@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for Qwen3-32B o On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. + +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} +The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes. + + ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b.mdx index bdc4284435..7f76c71134 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b.mdx @@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for Qwen3-8B on On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. + +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} +The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes. + + ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | @@ -213,7 +218,6 @@ python -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ - --max-attempts 5 \ --max-concurrency 1 \ --num-prompts 4 \ --random-input-len 3500 \ @@ -310,7 +314,6 @@ python -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ - --max-attempts 5 \ --max-concurrency 16 \ --num-prompts 16 \ --random-input-len 6144 \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct.mdx index f875fe1181..a359a33849 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct.mdx @@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for Qwen3-Next- On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. + +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} +The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes. + + ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | @@ -228,7 +233,7 @@ python3 -m sglang.launch_server \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ - --attn-dp-size 2 \ + --enable-dp-attention --dp-size 2 \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode auto \ @@ -336,7 +341,7 @@ python3 -m sglang.launch_server \ --max-prefill-tokens 28672 \ --context-length 81920 \ --max-total-tokens 122304 \ - --attn-dp-size 2 \ + --enable-dp-attention --dp-size 2 \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ @@ -362,7 +367,6 @@ python -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ - --max-attempts 5 \ --max-concurrency 16 \ --num-prompts 16 \ --random-input-len 6144 \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1.mdx index f0c6153721..1b41dad264 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1.mdx @@ -14,9 +14,9 @@ reinforcement learning-based training. This document demonstrates the deployment of DeepSeek-R1 on Ascend NPUs using SGLang, including single-node PD mixed mode, multi-node PD disaggregation mode, feature configuration, and performance optimization. -This document is validated and written based on **SGLang v0.5.16**. The current model (DeepSeek-R1) is fully supported in -this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use -v0.5.16 or a later version. +This document is validated and written based on **SGLang v0.5.18**. The current model (DeepSeek-R1) is fully supported +in this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not +covered by this document and may introduce breaking changes. ## Supported features @@ -90,17 +90,12 @@ The Docker image requires at least **30GB** of free space. Ensure sufficient dis The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -128,7 +123,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2.mdx index 187b581832..fb393d0c22 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2.mdx @@ -15,9 +15,9 @@ improvements in long-context scenarios. This document demonstrates the deployment of DeepSeek-V3.2 on Ascend NPUs using SGLang, including multi-node PD disaggregation mode, feature configuration, and performance optimization. -This document is validated and written based on **SGLang v0.5.16**. The current model (DeepSeek-V3.2) is fully supported in -this version. To use the latest features (e.g., PD disaggregation, speculative decoding, DSA context parallel), it is -recommended to use v0.5.16 or a later version. +This document is validated and written based on **SGLang v0.5.18**. The current model (DeepSeek-V3.2) is fully supported +in this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not +covered by this document and may introduce breaking changes. ## Supported features @@ -85,17 +85,12 @@ Ensure sufficient disk space before pulling images. The Docker image requires at The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -123,7 +118,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash.mdx index b1358b9a1f..3d806a2864 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash.mdx @@ -84,17 +84,13 @@ Ensure sufficient disk space before pulling images. The Docker image requires at The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable release yet, so the following +command uses the validated build tag `cann9.1.0-a3-B105`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -122,7 +118,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1.mdx index 7978fc8e6e..c32ca07158 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1.mdx @@ -90,17 +90,13 @@ Ensure sufficient disk space before pulling images. The Docker image requires at The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable release yet, so the following +command uses the validated build tag `cann9.1.0-a3-B105`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -128,7 +124,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2.mdx index c02453780d..dbec2e0356 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2.mdx @@ -81,17 +81,13 @@ Ensure sufficient disk space before pulling images. The Docker image requires at The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable release yet, so the following +command uses the validated build tag `cann9.1.0-a3-B105`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -119,7 +115,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3.mdx index b9b9590c37..e15daf0b73 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3.mdx @@ -78,19 +78,15 @@ The dependencies required for the NPU runtime environment have been integrated i online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable release yet, so the following +command uses the validated build tag `cann9.1.0-a3-B105`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 docker run -itd --shm-size=64g --name ${NAME} \ --privileged=true --net=host \ @@ -118,7 +114,7 @@ docker run -itd --shm-size=64g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6.mdx index a50190779b..33a755d326 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6.mdx @@ -16,9 +16,9 @@ This document demonstrates the deployment of Kimi-K2.6 on Ascend NPUs using SGLa mode, multi-node PD mixed mode, multi-node PD disaggregation mode, feature configuration, and performance optimization. -This document is validated and written based on **SGLang v0.5.16**. The current model (Kimi-K2.6) is fully supported in -this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to use -v0.5.16 or a later version. +This document is validated and written based on **SGLang v0.5.18**. The current model (Kimi-K2.6) is fully supported in +this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not +covered by this document and may introduce breaking changes. ## Supported features @@ -91,17 +91,12 @@ Ensure sufficient disk space before pulling images. The Docker image requires at The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -129,7 +124,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 ``` @@ -186,7 +181,8 @@ curl http://${HOST}:${PORT}/generate \ Expected result: an HTTP 200 response with the generated text containing "Paris". -For multimodal requests (text + image): +For multimodal requests (text + image). The example image URL is fetched by the server at request time, so the server +needs internet access; otherwise, replace it with a local image path or any URL reachable from the server: ```shell # ============================================================ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash.mdx index 68d0e71d49..193ba215c3 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash.mdx @@ -13,9 +13,9 @@ latency scenarios through PD disaggregation deployment. This document demonstrates the deployment of MiMo-V2-Flash on Ascend NPUs using SGLang, including multi-node PD disaggregation mode, feature configuration, and performance optimization. -This document is validated and written based on **SGLang v0.5.16**. The current model (MiMo-V2-Flash) is fully supported in -this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use -v0.5.16 or a later version. +This document is validated and written based on **SGLang v0.5.18**. The current model (MiMo-V2-Flash) is fully supported +in this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not +covered by this document and may introduce breaking changes. ## Supported features @@ -83,17 +83,12 @@ The Docker image requires at least **30GB** of free space. Ensure sufficient dis The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -121,7 +116,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5.mdx index 19fba32c3a..d74714e54b 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5.mdx @@ -8,16 +8,16 @@ metatags: MiniMax-M2.5 is a Mixture-of-Experts (MoE) large language model developed by MiniMax, featuring a sparse MoE architecture with approximately 230B total parameters and 10B active parameters. It supports native long-context -processing up to 200k tokens. The model supports EAGLE3 speculative decoding with a +processing up to 192k tokens. The model supports EAGLE3 speculative decoding with a custom eagle model for accelerated inference, and excels at general language understanding, reasoning, and long-context tasks. This document demonstrates the deployment of MiniMax-M2.5 on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. -This document is validated and written based on **SGLang v0.5.16**. The current model (MiniMax-M2.5) is fully supported in -this version. To use the latest features (e.g., speculative decoding), it is recommended to use -v0.5.16 or a later version. +This document is validated and written based on **SGLang v0.5.18**. The current model (MiniMax-M2.5) is fully supported +in this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not +covered by this document and may introduce breaking changes. ## Supported features @@ -83,17 +83,12 @@ Ensure sufficient disk space before pulling images. The Docker image requires at The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -121,7 +116,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b.mdx index 0cab9a8c93..0d8e3a2b05 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b.mdx @@ -34,7 +34,7 @@ by this document and may introduce breaking changes. | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | -| Context Parallelism | `--enable-prefill-cp \`
`--attn-cp-size 2 \`
`--moe-dp-size 2` | +| Context Parallelism | `--enable-prefill-cp \`
`--cp-strategy zigzag \`
`--attn-cp-size 2 \`
`--moe-dp-size 2` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment @@ -94,6 +94,7 @@ online platform. You can directly pull it. The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} ```bash Command docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 @@ -266,6 +267,7 @@ PD disaggregation is required. The following command is based on the **W8A8** qu **Constraints:** +- `--cp-strategy` is required when `--enable-prefill-cp` is enabled (e.g., `--cp-strategy zigzag`) - Prefill side must set `--max-running-requests 1` (PCP only supports batch_size=1) - `--attn-cp-size` must evenly divide `--tp-size`; each CP rank occupies `tp_size / cp_size` NPUs @@ -298,6 +300,7 @@ python3 -m sglang.launch_server \ --base-gpu-id 0 \ --tp-size 16 \ --enable-prefill-cp \ + --cp-strategy zigzag \ --attn-cp-size 2 \ --moe-dp-size 2 \ --max-running-requests 1 \ @@ -313,6 +316,7 @@ Key parameters for PCP: | Parameter | Value | Description | |-----------|-------|-------------| | `--enable-prefill-cp` | flag | Enable PCP feature | +| `--cp-strategy` | zigzag | Context sharding layout (required with `--enable-prefill-cp`) | | `--attn-cp-size` | 2 | Split context across 2 CP ranks (each rank handles half the sequence) | | `--moe-dp-size` | 2 | MoE DP size, should match `--attn-cp-size` | | `--max-running-requests` | 1 | Required by PCP (batch_size=1 constraint) | @@ -353,9 +357,29 @@ python3 -m sglang.launch_server \ --dist-init-addr :6688 ``` +3. Launch the SGLang Router (on any reachable node): + +```bash Router +#============================================================ +# Before running, update the following variables: +# PREFILL_HOST_IP: prefill node IP address +# DECODE_HOST_IP: decode node IP address +# ROUTER_HOST_IP: router node IP address +#============================================================ + +python3 -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy cache_aware \ + --prefill http://:8000 8995 \ + --decode http://:8001 \ + --host \ + --port 6689 +``` + `ASCEND_MF_STORE_URL` on both nodes must point to the same KV store (typically the prefill node IP). -`ASCEND_USE_FIA=True` enables fast interconnect aggregation for KV transfer. +`ASCEND_USE_FIA=True` switches the Ascend attention backend to the FIA path, which PCP prefill +attention requires (the non-FIA CP path is not implemented). PCP is a prefill-only feature; the decode side needs no CP-related flags. diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b.mdx index e8029965aa..d3679e8ef9 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b.mdx @@ -15,9 +15,9 @@ and thinking/reasoning-enhanced editions. This document demonstrates the deployment of Qwen3-30B-A3B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. -This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-30B-A3B) is fully supported in -this version. To use the latest features (e.g., speculative decoding), it is recommended to use -v0.5.16 or a later version. +This document is validated and written based on **SGLang v0.5.18**. The current model (Qwen3-30B-A3B) is fully supported +in this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not +covered by this document and may introduce breaking changes. ## Supported features @@ -82,17 +82,12 @@ The Docker image requires at least **30GB** of free space. Ensure sufficient dis The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -120,7 +115,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b.mdx index 41fa01407e..4d213ccaf9 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b.mdx @@ -14,9 +14,9 @@ decoding for accelerated inference and supports both standard and thinking/reaso This document demonstrates the deployment of Qwen3-32B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. -This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-32B) is fully supported in -this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later -version. +This document is validated and written based on **SGLang v0.5.18**. The current model (Qwen3-32B) is fully supported in +this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not +covered by this document and may introduce breaking changes. ## Supported features @@ -82,17 +82,12 @@ Ensure sufficient disk space before pulling images. The Docker image requires at The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -120,7 +115,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b.mdx index 2d43eacf0f..e34012b41a 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b.mdx @@ -83,17 +83,13 @@ Ensure sufficient disk space before pulling images. The Docker image requires at The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable release yet, so the following +command uses the validated build tag `cann9.1.0-a3-B105`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -121,7 +117,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 ``` @@ -163,7 +159,8 @@ curl http://${HOST}:${PORT}/generate \ Expected result: an HTTP 200 response with the generated text containing "Paris". -For multimodal requests (text + image): +For multimodal requests (text + image). The example image URL is fetched by the server at request time, so the server +needs internet access; otherwise, replace it with a local image path or any URL reachable from the server: ```shell # ============================================================ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b.mdx index e34a4f16f3..4df88ecc97 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b.mdx @@ -80,17 +80,13 @@ Ensure sufficient disk space before pulling images. The Docker image requires at The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable release yet, so the following +command uses the validated build tag `cann9.1.0-a3-B105`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -118,7 +114,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 ``` @@ -160,7 +156,8 @@ curl http://${HOST}:${PORT}/generate \ Expected result: an HTTP 200 response with the generated text containing "Paris". -For multimodal requests (text + image): +For multimodal requests (text + image). The example image URL is fetched by the server at request time, so the server +needs internet access; otherwise, replace it with a local image path or any URL reachable from the server: ```shell # ============================================================ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b.mdx index b99e24aaf7..3d746b9354 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b.mdx @@ -80,17 +80,13 @@ Ensure sufficient disk space before pulling images. The Docker image requires at The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable release yet, so the following +command uses the validated build tag `cann9.1.0-a3-B105`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -118,7 +114,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 ``` @@ -160,7 +156,8 @@ curl http://${HOST}:${PORT}/generate \ Expected result: an HTTP 200 response with the generated text containing "Paris". -For multimodal requests (text + image): +For multimodal requests (text + image). The example image URL is fetched by the server at request time, so the server +needs internet access; otherwise, replace it with a local image path or any URL reachable from the server: ```shell # ============================================================ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8_max.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8_max.mdx index a13cb97c39..25fd66c99c 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8_max.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8_max.mdx @@ -83,15 +83,13 @@ Ensure sufficient disk space before pulling images. The Docker image requires at The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -The following command is based on the daily build tag, which contains the latest SGLang main branch changes. +CANN 9.1.0 has no stable release yet, so the following command is based on the validated build +tag `cann9.1.0-a3-B105`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -119,7 +117,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b.mdx index 0aa3d2a414..f63aa05171 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b.mdx @@ -15,9 +15,9 @@ editions. This document demonstrates the deployment of Qwen3-8B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. -This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-8B) is fully supported in -this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later -version. +This document is validated and written based on **SGLang v0.5.18**. The current model (Qwen3-8B) is fully supported in +this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not +covered by this document and may introduce breaking changes. ## Supported features @@ -83,17 +83,12 @@ Ensure sufficient disk space before pulling images. The Docker image requires at The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -121,7 +116,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct.mdx index c4b653e546..4edbcf5368 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct.mdx @@ -15,9 +15,9 @@ following, reasoning, and text generation tasks. This document demonstrates the deployment of Qwen3-Next-80B-A3B-Instruct on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. -This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-Next-80B-A3B-Instruct) is -fully supported in this version. To use the latest features (e.g., speculative decoding), it is recommended to use -v0.5.16 or a later version. +This document is validated and written based on **SGLang v0.5.18**. The current model (Qwen3-Next-80B-A3B-Instruct) is +fully supported in this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later +versions are not covered by this document and may introduce breaking changes. ## Supported features @@ -85,17 +85,12 @@ Ensure sufficient disk space before pulling images. The Docker image requires at The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following -command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable -release tag. +The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). -{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */} +{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */} ```bash Command -# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the -# daily build tag. Once the stable image is published, switch back to the -# stable release tag. -docker pull quay.io/ascend/sglang:main-cann9.1.0-a3 +docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ @@ -123,7 +118,7 @@ docker run -itd --shm-size=16g --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:main-cann9.1.0-a3 +quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/reference/glossary.mdx b/docs/docs/hardware-platforms/ascend-npus/reference/glossary.mdx index fa85e73de6..91f8085e24 100644 --- a/docs/docs/hardware-platforms/ascend-npus/reference/glossary.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/reference/glossary.mdx @@ -20,9 +20,8 @@ SGLang supports the following Ascend inference hardware: Throughout these docs, **A3 Series** and **950PR/DT Series** are used as shorthand for the hardware above. -Docker image tags use `a3` for the A3 Series. For example, `main-cann9.1.0-a3`. -{/* TODO: CANN 9.1.0 has no stable image yet, so the example temporarily uses the daily build tag. Once the stable image is published, switch it back to a stable release tag. */} -Tags for the 950PR/DT Series use `950`. For example, `main-cann9.1.0-950`. +Docker image tags use `a3` for the A3 Series. For example, `cann9.1.0-a3-B105`. +Tags for the 950PR/DT Series use `950`. For example, `cann9.1.0-950-B105`. On the A3 Series, each card has 2 dies, giving 16 devices per node. Benchmark pages use "Cards" to refer to physical cards, so `Cards: 4` on the A3 Series means `--tp-size 8`.