mirrors/sglang
0
0 Fork 0
mirror of https://gitee.com/trusted-list/sglang.git synced 2026-10-28 08:34:56 +08:00
Code Activity

[Doc] [NPU] docs: update npu docs and fix some issues (#46736)

Browse Source
This commit is contained in:
amote-i authored and GitHub committed 2026-10-27 19:52:56 +08:00
1 parent 7e5cd57028
commit 82b90efc84
34 files changed
+251 -266
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation.mdx
+1 -4
View File
@@ -26,10 +26,7 @@ First, launch the SGLang environment using the provided container image:
<Tab title="Ascend A3 Series">
```shell Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily uses the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
export IMAGE=quay.io/ascend/sglang:main-cann9.1.0-a3
export IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
docs/docs/hardware-platforms/ascend-npus/faq.mdx
+5 -2
View File
@@ -297,10 +297,13 @@ Replace model weights, e.g., switch Qwen/Qwen3.5-27B-FP8 to Eco-Tech/Qwen3.5-27B
Docker images for Ascend NPU are available in two types:
{/* TODO: CANN 9.1.0 has no stable image yet, so the stable release entry below has no example tag for now. Once the stable image is published, add it back. */}
- **Stable release** — validated version with a specific tag. Recommended for production deployments.
- **Daily build** — includes the latest development changes, e.g., `quay.io/ascend/sglang:main-cann9.1.0-a3`. Use this
if you need the latest features or bug fixes that have not yet been included in a stable release.
{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and update the example image to the stable release once it is published. */}
CANN 9.1.0 has no stable release yet, so use the latest validated build instead, e.g.,
`swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105`. These
`cann9.1.0-<device>-B<NN>` tags are validated builds, not stable releases.
If you encounter issues with a stable release, try switching to a daily build to see if the issue has been resolved in the latest development version.
docs/docs/hardware-platforms/ascend-npus/getting-started/installation.mdx
+5 -11
View File
@@ -295,31 +295,25 @@ Ensure sufficient disk space before pulling images. Each Docker image requires a
1. Download SGLang image
<Tip>
We publish both **stable releases** and **daily builds**. Choose a stable release tag if you prefer a validated version, or a daily build tag (e.g., `main-cann9.1.0-a3`, `main-cann9.1.0-950`) if you need the latest development changes. CANN 9.1.0 has no stable image yet, so use the daily build tag for now.
We publish both **stable releases** and **daily builds**. Choose a stable release tag if you prefer a validated version, or a daily build tag (e.g., `main-cann9.1.0-a3`, `main-cann9.1.0-950`) if you need the latest development changes.
Validated release builds are also published as `cann9.1.0-<device>-B<NN>` tags (e.g., `cann9.1.0-950-B080`). The exact component versions inside each release image are listed in the [image dependency manifest](https://ascend.github.io/sglang/) — see for example the `cann9.1.0-950-B100` entry for the 950PR/DT Series.
CANN 9.1.0 has no stable release yet, so use a validated build for now. The `cann9.1.0-<device>-B<NN>` tags (e.g., `cann9.1.0-a3-B105`, `cann9.1.0-950-B105`) are validated builds rather than stable releases; they stand in until a stable release is published. The exact component versions inside each build are listed in the [image dependency manifest](https://ascend.github.io/sglang/) — see for example the `cann9.1.0-950-B105` entry for the 950PR/DT Series.
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tags `main-cann9.1.0-a3` / `main-cann9.1.0-950`. Once the stable image is published, switch back to the stable release tag. */}
{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tags below to the stable release once it is published. */}
</Tip>
<Tabs>
<Tab title="Ascend A3 Series">
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
```
</Tab>
<Tab title="Ascend 950PR/DT Series">
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-950
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-950-B105
```
</Tab>
docs/docs/hardware-platforms/ascend-npus/getting-started/quick_start.mdx
+2 -8
View File
@@ -40,10 +40,7 @@ at least **30GB** of free space. If you need to download model weights, check th
<Tip>
We publish both **stable releases** and **daily builds**. Choose a stable release tag if you prefer a validated version,
or a daily build tag (e.g., `main-cann9.1.0-a3`) if you need the latest development changes. CANN 9.1.0 has no stable
image yet, so use the daily build tag for now.
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
or a daily build tag (e.g., `main-cann9.1.0-a3`) if you need the latest development changes.
</Tip>
If you have already downloaded model weights to a local path (e.g., `/path/to/model`), mount the path into the container by
@@ -53,10 +50,7 @@ adding `--volume /path/to/model:/path/to/model` to the `docker run` command belo
<Tab title="Ascend A3 Series">
```shell Command
# TODO: CANN 9.1.0 has no stable image yet, so only the daily build tag is
# available for now. Once the stable image is published, restore the stable
# release option here.
export IMAGE=quay.io/ascend/sglang:main-cann9.1.0-a3
export IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1.mdx
+18 -13
View File
@@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for DeepSeek-R1
On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.
</Note>
<Warning>
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes.
</Warning>
### Low Latency
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
@@ -125,7 +130,7 @@ do
--speculative-num-steps 1 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--attn-dp-size 8 \
--enable-dp-attention --dp-size 8 \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--reasoning-parser deepseek-r1 \
@@ -159,7 +164,7 @@ do
--port 8001 \
--nnodes 1 \
--tp-size 16 \
--attn-dp-size 16 \
--enable-dp-attention --dp-size 16 \
--mem-fraction-static 0.9 \
--max-running-requests 416 \
--quantization modelslim \
@@ -300,7 +305,7 @@ python3 -m sglang.launch_server \
--max-prefill-tokens 3000 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--attn-dp-size 16 \
--enable-dp-attention --dp-size 16 \
--enable-dp-lm-head \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
@@ -427,7 +432,7 @@ do
--speculative-num-steps 1 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--attn-dp-size 2 \
--enable-dp-attention --dp-size 2 \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--enable-attn-tp-input-scattered \
@@ -464,7 +469,7 @@ do
--node-rank $i \
--nnodes 2 \
--tp-size 32 \
--attn-dp-size 16 \
--enable-dp-attention --dp-size 16 \
--mem-fraction-static 0.75 \
--max-running-requests 32 \
--quantization modelslim \
@@ -631,7 +636,7 @@ do
--speculative-num-steps 1 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--attn-dp-size 4 \
--enable-dp-attention --dp-size 4 \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--enable-attn-tp-input-scattered \
@@ -669,7 +674,7 @@ do
--node-rank $i \
--nnodes 2 \
--tp-size 32 \
--attn-dp-size 32 \
--enable-dp-attention --dp-size 32 \
--mem-fraction-static 0.82 \
--max-running-requests 1024 \
--quantization modelslim \
@@ -831,7 +836,7 @@ do
--speculative-num-steps 1 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--attn-dp-size 2 \
--enable-dp-attention --dp-size 2 \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--enable-attn-tp-input-scattered \
@@ -868,7 +873,7 @@ do
--node-rank $i \
--nnodes 2 \
--tp-size 32 \
--attn-dp-size 16 \
--enable-dp-attention --dp-size 16 \
--mem-fraction-static 0.75 \
--max-running-requests 32 \
--quantization modelslim \
@@ -1031,7 +1036,7 @@ do
--speculative-num-steps 1 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--attn-dp-size 2 \
--enable-dp-attention --dp-size 2 \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--enable-attn-tp-input-scattered \
@@ -1068,7 +1073,7 @@ do
--node-rank $i \
--nnodes 2 \
--tp-size 32 \
--attn-dp-size 16 \
--enable-dp-attention --dp-size 16 \
--mem-fraction-static 0.75 \
--max-running-requests 32 \
--quantization modelslim \
@@ -1230,7 +1235,7 @@ do
--speculative-num-steps 1 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--attn-dp-size 2 \
--enable-dp-attention --dp-size 2 \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--enable-attn-tp-input-scattered \
@@ -1267,7 +1272,7 @@ do
--node-rank $i \
--nnodes 2 \
--tp-size 32 \
--attn-dp-size 8 \
--enable-dp-attention --dp-size 8 \
--mem-fraction-static 0.75 \
--max-running-requests 32 \
--quantization modelslim \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2.mdx
+8 -3
View File
@@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for DeepSeek-V3
On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.
</Note>
<Warning>
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes.
</Warning>
### Low Latency
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
@@ -159,7 +164,7 @@ do
--node-rank $i \
--nnodes 2 \
--tp 32 \
--attn-dp-size 8 \
--enable-dp-attention --dp-size 8 \
--ep 32 \
--moe-dense-tp-size 1 \
--enable-dp-lm-head \
@@ -355,7 +360,7 @@ do
--node-rank $i \
--nnodes 2 \
--tp 32 \
--attn-dp-size 8 \
--enable-dp-attention --dp-size 8 \
--ep 32 \
--moe-dense-tp-size 1 \
--enable-dp-lm-head \
@@ -550,7 +555,7 @@ do
--node-rank $i \
--nnodes 2 \
--tp 32 \
--attn-dp-size 8 \
--enable-dp-attention --dp-size 8 \
--ep 32 \
--moe-dense-tp-size 1 \
--enable-dp-lm-head \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6.mdx
+11 -7
View File
@@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for Kimi-K2.6 o
On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.
</Note>
<Warning>
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes.
</Warning>
### Low Latency
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration |
@@ -119,7 +124,7 @@ do
--enable-multimodal \
--mm-attention-backend ascend_attn \
--sampling-backend ascend \
--attn-dp-size 32 \
--enable-dp-attention --dp-size 32 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs-decode 1 \
@@ -876,7 +881,7 @@ do
--tp-size 16 \
--mem-fraction-static 0.82 \
--max-running-requests 2 \
--attn-dp-size 2 \
--enable-dp-attention --dp-size 2 \
--enable-dp-lm-head \
--disable-radix-cache \
--enable-multimodal \
@@ -1011,7 +1016,7 @@ python3 -m sglang.launch_server \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--sampling-backend ascend \
--attn-dp-size 16 \
--enable-dp-attention --dp-size 16 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs-decode 1 2 4 8 9 10 11 \
@@ -1120,7 +1125,7 @@ python3 -m sglang.launch_server \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--sampling-backend ascend \
--attn-dp-size 16 \
--enable-dp-attention --dp-size 16 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs-decode 1 2 3 4 \
@@ -1234,7 +1239,7 @@ python3 -m sglang.launch_server \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--sampling-backend ascend \
--attn-dp-size 16 \
--enable-dp-attention --dp-size 16 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs-decode 1 2 3 4 5 \
@@ -1262,7 +1267,6 @@ python -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 6688 \
--max-attempts 5 \
--max-concurrency 64 \
--num-prompts 256 \
--random-input-len 3500 \
@@ -1346,7 +1350,7 @@ python3 -m sglang.launch_server \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--sampling-backend ascend \
--attn-dp-size 16 \
--enable-dp-attention --dp-size 16 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs-decode 1 2 4 8 11 12 13 \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_5_pro.mdx
+7 -2
View File
@@ -10,6 +10,11 @@ This page focuses on the deployment of MiMo-V2.5-Pro (FP4) with DFlash speculati
On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.
</Note>
<Warning>
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes.
</Warning>
### Model Deployment
MiMo-V2.5-Pro-FP4-DFlash is an MXFP4-quantized checkpoint with a built-in DFlash draft model (located in the `dflash/` subdirectory of the weights). The following example deploys it in 1P1D mode (1 prefill node + 1 decode node, TP8 + DP2 per node).
@@ -82,7 +87,7 @@ python3 -m sglang.launch_server \
--disaggregation-mode prefill --disaggregation-transfer-backend ascend \
--disaggregation-bootstrap-port 8996 \
--cuda-graph-backend-prefill disabled \
--attn-dp-size 2 --enable-dp-lm-head \
--enable-dp-attention --dp-size 2 --enable-dp-lm-head \
--moe-a2a-backend deepep --deepep-mode normal
```
@@ -110,7 +115,7 @@ python3 -m sglang.launch_server \
--disaggregation-mode decode --disaggregation-transfer-backend ascend \
--disaggregation-bootstrap-port 8996 \
--moe-a2a-backend deepep --deepep-mode low_latency \
--attn-dp-size 2 --enable-dp-lm-head \
--enable-dp-attention --dp-size 2 --enable-dp-lm-head \
--speculative-algorithm DFLASH \
--speculative-num-draft-tokens 8
```
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash.mdx
+21 -16
View File
@@ -10,31 +10,36 @@ This page focuses on optimal configuration and benchmark results for MiMo-V2-Fla
On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.
</Note>
<Warning>
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes.
</Warning>
### Low Latency
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 16k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1k-tpot-20ms) |
| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 32k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1k-tpot-20ms) |
| MiMo-V2-Flash | Ascend A3 Series Products | 16 | PD Disaggregation | 16k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-16p-in16k-out1k-tpot-20ms) |
| MiMo-V2-Flash | Ascend A3 Series Products | 16 | PD Disaggregation | 32k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-16p-in32k-out1k-tpot-20ms) |
### High Throughput
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 16k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1-ttft-5s) |
| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 32k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1-ttft-5s) |
| MiMo-V2-Flash | Ascend A3 Series Products | 16 | PD Disaggregation | 16k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-16p-in16k-out1-ttft-5s) |
| MiMo-V2-Flash | Ascend A3 Series Products | 16 | PD Disaggregation | 32k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-16p-in32k-out1-ttft-5s) |
## Optimal Configuration
<a id="pd-disaggregation" title="Referenced by external docs. Verify before removing."></a>
### MiMo-V2-Flash 1P1D 12P IN16K OUT1 TTFT 5s
### MiMo-V2-Flash 1P1D 16P IN16K OUT1 TTFT 5s
**Model**: MiMo-V2-Flash
**Hardware**: Ascend A3 Series Products
**Cards**: 12
**Cards**: 16
**Deploy Mode**: PD Disaggregation
@@ -166,7 +171,7 @@ do
--speculative-num-draft-tokens 4 \
--enable-multi-layer-eagle \
--disable-radix-cache \
--attn-dp-size 2 \
--enable-dp-attention --dp-size 2 \
--enable-dp-lm-head \
--moe-a2a-backend deepep \
--deepep-mode low_latency
@@ -211,13 +216,13 @@ python -m sglang.bench_serving \
--seed 1
```
### MiMo-V2-Flash 1P1D 12P IN16K OUT1K TPOT 20ms
### MiMo-V2-Flash 1P1D 16P IN16K OUT1K TPOT 20ms
**Model**: MiMo-V2-Flash
**Hardware**: Ascend A3 Series Products
**Cards**: 12
**Cards**: 16
**Deploy Mode**: PD Disaggregation
@@ -349,7 +354,7 @@ do
--speculative-num-draft-tokens 4 \
--enable-multi-layer-eagle \
--disable-radix-cache \
--attn-dp-size 2 \
--enable-dp-attention --dp-size 2 \
--enable-dp-lm-head \
--moe-a2a-backend deepep \
--deepep-mode low_latency
@@ -394,13 +399,13 @@ python -m sglang.bench_serving \
--seed 1
```
### MiMo-V2-Flash 1P1D 12P IN32K OUT1 TTFT 5s
### MiMo-V2-Flash 1P1D 16P IN32K OUT1 TTFT 5s
**Model**: MiMo-V2-Flash
**Hardware**: Ascend A3 Series Products
**Cards**: 12
**Cards**: 16
**Deploy Mode**: PD Disaggregation
@@ -532,7 +537,7 @@ do
--speculative-num-draft-tokens 4 \
--enable-multi-layer-eagle \
--disable-radix-cache \
--attn-dp-size 2 \
--enable-dp-attention --dp-size 2 \
--enable-dp-lm-head \
--moe-a2a-backend deepep \
--deepep-mode low_latency
@@ -577,13 +582,13 @@ python -m sglang.bench_serving \
--seed 1
```
### MiMo-V2-Flash 1P1D 12P IN32K OUT1K TPOT 20ms
### MiMo-V2-Flash 1P1D 16P IN32K OUT1K TPOT 20ms
**Model**: MiMo-V2-Flash
**Hardware**: Ascend A3 Series Products
**Cards**: 12
**Cards**: 16
**Deploy Mode**: PD Disaggregation
@@ -715,7 +720,7 @@ do
--speculative-num-draft-tokens 4 \
--enable-multi-layer-eagle \
--disable-radix-cache \
--attn-dp-size 2 \
--enable-dp-attention --dp-size 2 \
--enable-dp-lm-head \
--moe-a2a-backend deepep \
--deepep-mode low_latency
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5.mdx
+8 -3
View File
@@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for MiniMax-M2.
On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.
</Note>
<Warning>
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes.
</Warning>
### Low Latency
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
@@ -312,7 +317,7 @@ python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 16 \
--attn-dp-size 2 \
--enable-dp-attention --dp-size 2 \
--mem-fraction-static 0.65 \
--max-running-requests 4 \
--reasoning-parser minimax-append-think \
@@ -421,7 +426,7 @@ python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 16 \
--attn-dp-size 16 \
--enable-dp-attention --dp-size 16 \
--mem-fraction-static 0.53 \
--max-running-requests 96 \
--disable-radix-cache \
@@ -525,7 +530,7 @@ python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 16 \
--attn-dp-size 16 \
--enable-dp-attention --dp-size 16 \
--mem-fraction-static 0.75 \
--max-running-requests 320 \
--disable-radix-cache \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b.mdx
+1
View File
@@ -11,6 +11,7 @@ On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count;
</Note>
<Warning>
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes.
</Warning>
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b.mdx
+7 -4
View File
@@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for Qwen3-30B-A
On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.
</Note>
<Warning>
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes.
</Warning>
### Low Latency
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
@@ -105,7 +110,7 @@ python3 -m sglang.launch_server \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 8 \
--tp-size 2 \
--attn-dp-size 2 \
--enable-dp-attention --dp-size 2 \
--mem-fraction-static 0.85 \
--cuda-graph-bs-decode 1 2 4 8 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 \
--dtype bfloat16 \
@@ -128,8 +133,7 @@ python -m sglang.bench_serving \
--random-input-len 1000 \
--random-output-len 100 \
--random-range-ratio 1 \
--seed 1 \
--max-attempts 4
--seed 1
```
### Qwen3-30B-A3B W8A8 1P IN3K5 OUT1K5 10ms
@@ -423,7 +427,6 @@ python -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 6688 \
--max-attempts 5 \
--max-concurrency 16 \
--num-prompts 16 \
--random-input-len 6144 \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b.mdx
+5
View File
@@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for Qwen3-32B o
On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.
</Note>
<Warning>
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes.
</Warning>
### Low Latency
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b.mdx
+5 -2
View File
@@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for Qwen3-8B on
On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.
</Note>
<Warning>
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes.
</Warning>
### Low Latency
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
@@ -213,7 +218,6 @@ python -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 6688 \
--max-attempts 5 \
--max-concurrency 1 \
--num-prompts 4 \
--random-input-len 3500 \
@@ -310,7 +314,6 @@ python -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 6688 \
--max-attempts 5 \
--max-concurrency 16 \
--num-prompts 16 \
--random-input-len 6144 \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct.mdx
+7 -3
View File
@@ -10,6 +10,11 @@ This page focuses on optimal configuration and benchmark results for Qwen3-Next-
On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.
</Note>
<Warning>
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes.
</Warning>
### Low Latency
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
@@ -228,7 +233,7 @@ python3 -m sglang.launch_server \
--speculative-num-draft-tokens 4 \
--speculative-draft-model-quantization unquant \
--speculative-draft-model-path $DRAFT_MODEL_PATH \
--attn-dp-size 2 \
--enable-dp-attention --dp-size 2 \
--enable-dp-lm-head \
--moe-a2a-backend deepep \
--deepep-mode auto \
@@ -336,7 +341,7 @@ python3 -m sglang.launch_server \
--max-prefill-tokens 28672 \
--context-length 81920 \
--max-total-tokens 122304 \
--attn-dp-size 2 \
--enable-dp-attention --dp-size 2 \
--enable-dp-lm-head \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
@@ -362,7 +367,6 @@ python -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 6688 \
--max-attempts 5 \
--max-concurrency 16 \
--num-prompts 16 \
--random-input-len 6144 \
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1.mdx
+7 -12
View File
@@ -14,9 +14,9 @@ reinforcement learning-based training.
This document demonstrates the deployment of DeepSeek-R1 on Ascend NPUs using SGLang, including single-node PD mixed
mode, multi-node PD disaggregation mode, feature configuration, and performance optimization.
This document is validated and written based on **SGLang v0.5.16**. The current model (DeepSeek-R1) is fully supported in
this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use
v0.5.16 or a later version.
This document is validated and written based on **SGLang v0.5.18**. The current model (DeepSeek-R1) is fully supported
in this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not
covered by this document and may introduce breaking changes.
## Supported features
@@ -90,17 +90,12 @@ The Docker image requires at least **30GB** of free space. Ensure sufficient dis
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -128,7 +123,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
```
<Tip>
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2.mdx
+7 -12
View File
@@ -15,9 +15,9 @@ improvements in long-context scenarios.
This document demonstrates the deployment of DeepSeek-V3.2 on Ascend NPUs using SGLang, including multi-node PD
disaggregation mode, feature configuration, and performance optimization.
This document is validated and written based on **SGLang v0.5.16**. The current model (DeepSeek-V3.2) is fully supported in
this version. To use the latest features (e.g., PD disaggregation, speculative decoding, DSA context parallel), it is
recommended to use v0.5.16 or a later version.
This document is validated and written based on **SGLang v0.5.18**. The current model (DeepSeek-V3.2) is fully supported
in this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not
covered by this document and may introduce breaking changes.
## Supported features
@@ -85,17 +85,12 @@ Ensure sufficient disk space before pulling images. The Docker image requires at
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -123,7 +118,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
```
<Tip>
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash.mdx
+5 -9
View File
@@ -84,17 +84,13 @@ Ensure sufficient disk space before pulling images. The Docker image requires at
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable release yet, so the following
command uses the validated build tag `cann9.1.0-a3-B105`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -122,7 +118,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
```
<Tip>
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1.mdx
+5 -9
View File
@@ -90,17 +90,13 @@ Ensure sufficient disk space before pulling images. The Docker image requires at
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable release yet, so the following
command uses the validated build tag `cann9.1.0-a3-B105`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -128,7 +124,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
```
<Tip>
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2.mdx
+5 -9
View File
@@ -81,17 +81,13 @@ Ensure sufficient disk space before pulling images. The Docker image requires at
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable release yet, so the following
command uses the validated build tag `cann9.1.0-a3-B105`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -119,7 +115,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
```
<Tip>
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3.mdx
+5 -9
View File
@@ -78,19 +78,15 @@ The dependencies required for the NPU runtime environment have been integrated i
online platform. You can directly pull it.
<Note>
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable release yet, so the following
command uses the validated build tag `cann9.1.0-a3-B105`.
For details, see
[Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
</Note>
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
docker run -itd --shm-size=64g --name ${NAME} \
--privileged=true --net=host \
@@ -118,7 +114,7 @@ docker run -itd --shm-size=64g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
```
<Tip>
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6.mdx
+9 -13
View File
@@ -16,9 +16,9 @@ This document demonstrates the deployment of Kimi-K2.6 on Ascend NPUs using SGLa
mode, multi-node PD mixed mode, multi-node PD disaggregation mode, feature configuration, and performance
optimization.
This document is validated and written based on **SGLang v0.5.16**. The current model (Kimi-K2.6) is fully supported in
this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to use
v0.5.16 or a later version.
This document is validated and written based on **SGLang v0.5.18**. The current model (Kimi-K2.6) is fully supported in
this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not
covered by this document and may introduce breaking changes.
## Supported features
@@ -91,17 +91,12 @@ Ensure sufficient disk space before pulling images. The Docker image requires at
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -129,7 +124,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
```
<Tip>
@@ -186,7 +181,8 @@ curl http://${HOST}:${PORT}/generate \
Expected result: an HTTP 200 response with the generated text containing "Paris".
For multimodal requests (text + image):
For multimodal requests (text + image). The example image URL is fetched by the server at request time, so the server
needs internet access; otherwise, replace it with a local image path or any URL reachable from the server:
```shell
# ============================================================
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash.mdx
+7 -12
View File
@@ -13,9 +13,9 @@ latency scenarios through PD disaggregation deployment.
This document demonstrates the deployment of MiMo-V2-Flash on Ascend NPUs using SGLang, including multi-node PD
disaggregation mode, feature configuration, and performance optimization.
This document is validated and written based on **SGLang v0.5.16**. The current model (MiMo-V2-Flash) is fully supported in
this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use
v0.5.16 or a later version.
This document is validated and written based on **SGLang v0.5.18**. The current model (MiMo-V2-Flash) is fully supported
in this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not
covered by this document and may introduce breaking changes.
## Supported features
@@ -83,17 +83,12 @@ The Docker image requires at least **30GB** of free space. Ensure sufficient dis
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -121,7 +116,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
```
<Tip>
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5.mdx
+8 -13
View File
@@ -8,16 +8,16 @@ metatags:
MiniMax-M2.5 is a Mixture-of-Experts (MoE) large language model developed by MiniMax, featuring a sparse MoE
architecture with approximately 230B total parameters and 10B active parameters. It supports native long-context
processing up to 200k tokens. The model supports EAGLE3 speculative decoding with a
processing up to 192k tokens. The model supports EAGLE3 speculative decoding with a
custom eagle model for accelerated inference, and excels at general language understanding, reasoning, and
long-context tasks.
This document demonstrates the deployment of MiniMax-M2.5 on Ascend NPUs using SGLang, including single-node PD mixed
mode, feature configuration, and performance optimization.
This document is validated and written based on **SGLang v0.5.16**. The current model (MiniMax-M2.5) is fully supported in
this version. To use the latest features (e.g., speculative decoding), it is recommended to use
v0.5.16 or a later version.
This document is validated and written based on **SGLang v0.5.18**. The current model (MiniMax-M2.5) is fully supported
in this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not
covered by this document and may introduce breaking changes.
## Supported features
@@ -83,17 +83,12 @@ Ensure sufficient disk space before pulling images. The Docker image requires at
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -121,7 +116,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
```
<Tip>
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b.mdx
+26 -2
View File
@@ -34,7 +34,7 @@ by this document and may introduce breaking changes.
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
| Context Parallelism | `--enable-prefill-cp \`<br/>`--attn-cp-size 2 \`<br/>`--moe-dp-size 2` |
| Context Parallelism | `--enable-prefill-cp \`<br/>`--cp-strategy zigzag \`<br/>`--attn-cp-size 2 \`<br/>`--moe-dp-size 2` |
<Note>
The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
@@ -94,6 +94,7 @@ online platform. You can directly pull it.
The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
```bash Command
docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
@@ -266,6 +267,7 @@ PD disaggregation is required. The following command is based on the **W8A8** qu
<Warning>
**Constraints:**
- `--cp-strategy` is required when `--enable-prefill-cp` is enabled (e.g., `--cp-strategy zigzag`)
- Prefill side must set `--max-running-requests 1` (PCP only supports batch_size=1)
- `--attn-cp-size` must evenly divide `--tp-size`; each CP rank occupies `tp_size / cp_size` NPUs
</Warning>
@@ -298,6 +300,7 @@ python3 -m sglang.launch_server \
--base-gpu-id 0 \
--tp-size 16 \
--enable-prefill-cp \
--cp-strategy zigzag \
--attn-cp-size 2 \
--moe-dp-size 2 \
--max-running-requests 1 \
@@ -313,6 +316,7 @@ Key parameters for PCP:
| Parameter | Value | Description |
|-----------|-------|-------------|
| `--enable-prefill-cp` | flag | Enable PCP feature |
| `--cp-strategy` | zigzag | Context sharding layout (required with `--enable-prefill-cp`) |
| `--attn-cp-size` | 2 | Split context across 2 CP ranks (each rank handles half the sequence) |
| `--moe-dp-size` | 2 | MoE DP size, should match `--attn-cp-size` |
| `--max-running-requests` | 1 | Required by PCP (batch_size=1 constraint) |
@@ -353,9 +357,29 @@ python3 -m sglang.launch_server \
--dist-init-addr <DECODE_HOST_IP>:6688
```
3. Launch the SGLang Router (on any reachable node):
```bash Router
#============================================================
# Before running, update the following variables:
# PREFILL_HOST_IP: prefill node IP address
# DECODE_HOST_IP: decode node IP address
# ROUTER_HOST_IP: router node IP address
#============================================================
python3 -m sglang_router.launch_router \
--pd-disaggregation \
--policy cache_aware \
--prefill http://<PREFILL_HOST_IP>:8000 8995 \
--decode http://<DECODE_HOST_IP>:8001 \
--host <ROUTER_HOST_IP> \
--port 6689
```
<Tip>
`ASCEND_MF_STORE_URL` on both nodes must point to the same KV store (typically the prefill node IP).
`ASCEND_USE_FIA=True` enables fast interconnect aggregation for KV transfer.
`ASCEND_USE_FIA=True` switches the Ascend attention backend to the FIA path, which PCP prefill
attention requires (the non-FIA CP path is not implemented).
PCP is a prefill-only feature; the decode side needs no CP-related flags.
</Tip>
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b.mdx
+7 -12
View File
@@ -15,9 +15,9 @@ and thinking/reasoning-enhanced editions.
This document demonstrates the deployment of Qwen3-30B-A3B on Ascend NPUs using SGLang, including single-node PD mixed
mode, feature configuration, and performance optimization.
This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-30B-A3B) is fully supported in
this version. To use the latest features (e.g., speculative decoding), it is recommended to use
v0.5.16 or a later version.
This document is validated and written based on **SGLang v0.5.18**. The current model (Qwen3-30B-A3B) is fully supported
in this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not
covered by this document and may introduce breaking changes.
## Supported features
@@ -82,17 +82,12 @@ The Docker image requires at least **30GB** of free space. Ensure sufficient dis
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -120,7 +115,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
```
<Tip>
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b.mdx
+7 -12
View File
@@ -14,9 +14,9 @@ decoding for accelerated inference and supports both standard and thinking/reaso
This document demonstrates the deployment of Qwen3-32B on Ascend NPUs using SGLang, including single-node PD mixed mode,
feature configuration, and performance optimization.
This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-32B) is fully supported in
this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later
version.
This document is validated and written based on **SGLang v0.5.18**. The current model (Qwen3-32B) is fully supported in
this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not
covered by this document and may introduce breaking changes.
## Supported features
@@ -82,17 +82,12 @@ Ensure sufficient disk space before pulling images. The Docker image requires at
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -120,7 +115,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
```
<Tip>
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b.mdx
+7 -10
View File
@@ -83,17 +83,13 @@ Ensure sufficient disk space before pulling images. The Docker image requires at
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable release yet, so the following
command uses the validated build tag `cann9.1.0-a3-B105`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -121,7 +117,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
```
<Tip>
@@ -163,7 +159,8 @@ curl http://${HOST}:${PORT}/generate \
Expected result: an HTTP 200 response with the generated text containing "Paris".
For multimodal requests (text + image):
For multimodal requests (text + image). The example image URL is fetched by the server at request time, so the server
needs internet access; otherwise, replace it with a local image path or any URL reachable from the server:
```shell
# ============================================================
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b.mdx
+7 -10
View File
@@ -80,17 +80,13 @@ Ensure sufficient disk space before pulling images. The Docker image requires at
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable release yet, so the following
command uses the validated build tag `cann9.1.0-a3-B105`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -118,7 +114,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
```
<Tip>
@@ -160,7 +156,8 @@ curl http://${HOST}:${PORT}/generate \
Expected result: an HTTP 200 response with the generated text containing "Paris".
For multimodal requests (text + image):
For multimodal requests (text + image). The example image URL is fetched by the server at request time, so the server
needs internet access; otherwise, replace it with a local image path or any URL reachable from the server:
```shell
# ============================================================
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b.mdx
+7 -10
View File
@@ -80,17 +80,13 @@ Ensure sufficient disk space before pulling images. The Docker image requires at
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable release yet, so the following
command uses the validated build tag `cann9.1.0-a3-B105`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -118,7 +114,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
```
<Tip>
@@ -160,7 +156,8 @@ curl http://${HOST}:${PORT}/generate \
Expected result: an HTTP 200 response with the generated text containing "Paris".
For multimodal requests (text + image):
For multimodal requests (text + image). The example image URL is fetched by the server at request time, so the server
needs internet access; otherwise, replace it with a local image path or any URL reachable from the server:
```shell
# ============================================================
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8_max.mdx
+5 -7
View File
@@ -83,15 +83,13 @@ Ensure sufficient disk space before pulling images. The Docker image requires at
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
The following command is based on the daily build tag, which contains the latest SGLang main branch changes.
CANN 9.1.0 has no stable release yet, so the following command is based on the validated build
tag `cann9.1.0-a3-B105`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* TODO: Interim explanation (CANN 9.1.0 has no stable release yet). Remove it and switch the tag below to the stable release once it is published. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -119,7 +117,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105
```
<Tip>
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b.mdx
+7 -12
View File
@@ -15,9 +15,9 @@ editions.
This document demonstrates the deployment of Qwen3-8B on Ascend NPUs using SGLang, including single-node PD mixed mode,
feature configuration, and performance optimization.
This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-8B) is fully supported in
this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later
version.
This document is validated and written based on **SGLang v0.5.18**. The current model (Qwen3-8B) is fully supported in
this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not
covered by this document and may introduce breaking changes.
## Supported features
@@ -83,17 +83,12 @@ Ensure sufficient disk space before pulling images. The Docker image requires at
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -121,7 +116,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
```
<Tip>
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct.mdx
+7 -12
View File
@@ -15,9 +15,9 @@ following, reasoning, and text generation tasks.
This document demonstrates the deployment of Qwen3-Next-80B-A3B-Instruct on Ascend NPUs using SGLang, including
single-node PD mixed mode, feature configuration, and performance optimization.
This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-Next-80B-A3B-Instruct) is
fully supported in this version. To use the latest features (e.g., speculative decoding), it is recommended to use
v0.5.16 or a later version.
This document is validated and written based on **SGLang v0.5.18**. The current model (Qwen3-Next-80B-A3B-Instruct) is
fully supported in this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later
versions are not covered by this document and may introduce breaking changes.
## Supported features
@@ -85,17 +85,12 @@ Ensure sufficient disk space before pulling images. The Docker image requires at
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.
Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).
{/* TODO: CANN 9.1.0 has no stable image yet, so this page temporarily references the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable release tag. */}
{/* NOTE(agent): The image version and scripts in this file are frozen (validated best practice). Skip this file in batch image-version updates and when deprecating or renaming parameters/flags. */}
```bash Command
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
@@ -123,7 +118,7 @@ docker run -itd --shm-size=16g --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
```
<Tip>
docs/docs/hardware-platforms/ascend-npus/reference/glossary.mdx
+2 -3
View File
@@ -20,9 +20,8 @@ SGLang supports the following Ascend inference hardware:
<Note>
Throughout these docs, **A3 Series** and **950PR/DT Series** are used as shorthand for the hardware above.
Docker image tags use `a3` for the A3 Series. For example, `main-cann9.1.0-a3`.
{/* TODO: CANN 9.1.0 has no stable image yet, so the example temporarily uses the daily build tag. Once the stable image is published, switch it back to a stable release tag. */}
Tags for the 950PR/DT Series use `950`. For example, `main-cann9.1.0-950`.
Docker image tags use `a3` for the A3 Series. For example, `cann9.1.0-a3-B105`.
Tags for the 950PR/DT Series use `950`. For example, `cann9.1.0-950-B105`.
On the A3 Series, each card has 2 dies, giving 16 devices per node.
Benchmark pages use "Cards" to refer to physical cards, so `Cards: 4` on the A3 Series means `--tp-size 8`.