MI50 / V620 — results, images and reproduction

Qwen3.8-27B · FP16, local RTN8 and RTN4 · TP2 / TP4 / TP8 · native MTP2

PUBLIC REVIEW — end-to-end command validation complete

47 of 48 executable configurations have all four commands independently verified; 1 capacity-blocked configuration(s) have a reviewed failure rather than a benchmark pass. 191 of 192 command executions verified. Published on LocalAIServers.com for review.

Docker Hub validation is shown separately from recorded and earlier exported-image results. A strict failure is retained even when the script itself completed successfully. The three unmeasured V620 TP8 placeholders remain unmeasured. The original services on the MI50 test host were restored and verified with real inference.

Validation coverage · Portable launch guide · Final command review · Restored-service verification
Earlier current-model validation · separate campaign

29 of 29 current-profile retests captured and checked · 28 full ladders · 1 without a full ladder · 0 running · 20 exact runtime IDs published to Docker Hub · Download guide

Earlier historical-model validation · separate campaign

6 of 6 distinct historical configurations retested · 6 complete fresh ladders · covers 19 preserved original rows · Fresh historical evidence

Recorded results remain intact. Exported-image retests appear in their own column. Strict checks correctness; the eligible speed sample is exact 2K, and 10K is diagnostic. A failed strict test makes both speed figures diagnostic (*). Historical Qwen3.6 uses a separate published runtime and model. Blank measurements are shown as “—”.
Download ExcelCSVExact JSONLaunch package guide

Serve / docker run leaves a service running. Benchmark reproduction runs the measurement sequence. Current command-validation coverage · All current commands (JSON) · Pinned model inputs

GPU / recordWeights / modelConfigurationRecorded strictRecorded 2K TPSRecorded 10K TPSPrefill / total secondsMemory per rankDocker Hub imagePortable serving commandBenchmark reproductionDocker Hub command validationEvidence
MI50Current Qwen3.8FP16Qwen/Qwen3.8-27BTP4MTP off PASS11170 tokens 53.55550.624 11.320 / 48.67610K: 11.950 / 209.497Strict: 11.748 / 233.402 12.53 / 31.14retained FP16 parameter extents; projection buffers separate / VRAM GiBSeparate packed-buffer extents: 9.381 GiB/rank; includes alternate views, not additive physical storage. Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:2c4733c2e813c2c9fc1cad8b8aeee0a1d8e7237f7a7a85bb0e9ecd4aa5a43fcc
Runtime image config: sha256:6bc353a76b3d75e16560d2c5e60a8c1ff301bd85314c95ed93bb31f88c08c8b4
Registry verificationDerived host-portability image; recorded benchmark parent: sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04
Exact compatibility change
Replaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately.
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp4-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-fp16-tp4-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \
  --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \
  -e HIP_VISIBLE_DEVICES=0,1,2,3 \
  -e VLLM_GFX906_ROWPAR_MUTABLE_AR=0 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_GFX906_QWEN_OUTPUT_FIXED3=1 \
  -e VLLM_GFX906_QWEN_QKV_FIXED3=1 \
  -e GFX906_MODEL_IDENTITY_DIR=/runwork \
  -e VLLM_GFX906_QWEN_MLP_FIXED3_STRICT=1 \
  -e VLLM_GFX906_QWEN_GDN_FIXED3=1 \
  -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \
  -e VLLM_GFX906_PERSISTENT_AR=0 \
  -e VLLM_GFX906_PERSISTENT_AR_KIND_ROUTE=0 \
  -e VLLM_GFX906_QWEN_MLP_FIXED3=1 \
  -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \
  -e HF_HUB_OFFLINE=1 \
  -e VLLM_GFX906_QWEN_MLP_FIXED3_EXT_PATH=/opt/vllm_patch_bundle/native/fixed3/v756f96_fixed3_core.so \
  -e VLLM_GFX906_ROWPAR_BOUNDARY_CUT=0 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp0/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp0/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp0,dst=/runwork \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp4-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-mi50-r3.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 55.211 · eligible10K 52.887 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 6447,
    "content_chars": 14192,
    "content_has_think_close": false,
    "elapsed_seconds": 129.0551943834871,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T13:10:56Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.008790487423539162,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 14192,
    "qwen_answer_text_sha256": "48ad4353316f3ec81d778cb0bd02f955629b0e42062d8e44609031916a6f0d4a",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 14192,
    "qwen_visible_text_sha256": "48ad4353316f3ec81d778cb0bd02f955629b0e42062d8e44609031916a6f0d4a",
    "raw_has_think_close": false,
    "reasoning_chars": 18865,
    "text_chars": 33058,
    "text_sha256": "3456fd7908c8bc4e5aa7ac66385d5f572f6a256afd8173ad9877fa582f015fa1",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 119.21210407651961,
    "vllm_decode_tps": 54.080078947871044,
    "vllm_generation_tokens_delta": 6447.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 9.833608750253916,
    "vllm_prefill_tps": 48.10034769664662,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 49.9553701096506
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 45.91352475900203,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T13:11:42Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.009055406786501408,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 10184,
    "text_chars": 10184,
    "text_sha256": "7dbe29445b58f7f49d4a0db3484a13f4e26bf4636101d736a3329a6e0b19c4de",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 36.224582785740495,
    "vllm_decode_tps": 55.21112587630087,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 9.681173008866608,
    "vllm_prefill_tps": 48.857715853936064,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 43.560149443936346
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 33642,
    "content_has_think_close": false,
    "elapsed_seconds": 198.85716584604234,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T13:15:01Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.005759751424193382,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 33642,
    "qwen_answer_text_sha256": "c9106d931f3a861571d0779d42737ee59d71c0701c09bf861c4f5c0e233c2698",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 33642,
    "qwen_visible_text_sha256": "c9106d931f3a861571d0779d42737ee59d71c0701c09bf861c4f5c0e233c2698",
    "raw_has_think_close": false,
    "reasoning_chars": 18865,
    "text_chars": 52508,
    "text_sha256": "e6fcb8784626552d714b997d261b8220f5355bb612b4596f16ad87c1274b5f64",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 189.0812345771119,
    "vllm_decode_tps": 52.88732127418894,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 9.767040423117578,
    "vllm_prefill_tps": 48.428180852047845,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 50.287350508365
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS6,668 natural tokens2K 53.74910K 51.418 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 137,
      "Error": "",
      "StartedAt": "2026-10-26T20:29:42.542233503Z",
      "FinishedAt": "2026-10-26T20:57:24.923491054Z"
    },
    "devices_after": [
      {
        "card": 1,
        "render": 129,
        "pci": "0000:06:00.0",
        "actual_card": "card0",
        "busy_percent": 0,
        "vram_bytes": 31794159616,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:0a:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 31561097216,
        "dpm": "auto"
      },
      {
        "card": 3,
        "render": 131,
        "pci": "0000:0d:00.0",
        "actual_card": "card3",
        "busy_percent": 0,
        "vram_bytes": 31559000064,
        "dpm": "auto"
      },
      {
        "card": 4,
        "render": 132,
        "pci": "0000:10:00.0",
        "actual_card": "card4",
        "busy_percent": 0,
        "vram_bytes": 31504474112,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T20:57:25.071040+00:00"
  },
  "retest_closure_note": "The reproduction runner issued Docker stop with a 60-second grace period. Final exit 137, PID 0 and OOM false are retained; this stop outcome is separate from any measurement verdict.",
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 53.74919605459173,
  "retest_10K_TPS": 51.41767637840134,
  "retest_strict_tokens": 6668,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 6668,
      "elapsed_seconds": 143.07483081799,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 127.12655968707986,
      "vllm_decode_tps": 52.45166719223098,
      "vllm_generation_tokens_delta": 6668.0,
      "vllm_prefill_seconds_delta": 15.934438918950036,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 53.29153924190905,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 37.20985887805,
      "vllm_decode_tps": 53.74919605459173,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 16.070583492983133,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 210.65881087898742,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 194.4856458780123,
      "vllm_decode_tps": 51.41767637840134,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 16.16023535700515,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 16.070583492983133,
  "retest_2K_total_seconds": 53.29153924190905,
  "retest_10K_prefill_seconds": 16.16023535700515,
  "retest_10K_total_seconds": 210.65881087898742,
  "retest_strict_prefill_seconds": 15.934438918950036,
  "retest_strict_total_seconds": 143.07483081799,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": null,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33421369344,
    "1": 33440952320,
    "2": 33438003200,
    "3": 33384120320
  },
  "retest_memory_observation_count": 27,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-fp16-tp4-mtp0/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 22.81,
  "retest_logged_model_load_seconds": 67.548319,
  "retest_logged_available_KV_GiB": 6.53,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 6.53,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 106800,
  "retest_logged_context_capacity_x": 3.23,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-fp16-tp4-mtp0",
    "image": "sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04",
    "container": "c8c0b898e9d81627eee172f20ca0783b4e5a809246b93619c4230e06528aae37",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 22.81,
    "retest_logged_model_load_seconds": 67.548319,
    "retest_logged_available_KV_GiB": 6.53,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 6.53,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 106800,
    "retest_logged_context_capacity_x": 3.23,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-fp16-tp4-mtp0/service-final.log",
    "source_log_sha256": "b6ce15ca515eb9d0dd6c076e8574f46fd917d7b27ba414136defce7cc7e4b42d",
    "sealed_index_sha256": "352796834416d1920746822000adb67699cb30990ae230cd3e920799f86c1230",
    "source_lines": {
      "model_load": {
        "line": 1161,
        "text": "(Worker pid=197) (Worker_TP0 pid=197) INFO 10-26 20:32:47 [gpu_model_runner.py:4584] Model loading took 22.81 GiB memory and 67.548319 seconds"
      },
      "automatic_KV_budget": {
        "line": 1240,
        "text": "(Worker pid=197) (Worker_TP0 pid=197) INFO 10-26 20:34:54 [gpu_worker.py:453] Available KV cache memory: 6.53 GiB"
      },
      "grouped_KV_tokens": {
        "line": 1245,
        "text": "(EngineCore_DP0 pid=179) INFO 10-26 20:34:54 [kv_cache_utils.py:1314] GPU KV cache size: 106,800 tokens"
      },
      "context_capacity": {
        "line": 1246,
        "text": "(EngineCore_DP0 pid=179) INFO 10-26 20:34:54 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 3.23x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-fp16-tp4-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 31.14431381225586
}
Retest KV budget: 6.530 GiB (automatic, rounded)106,800 grouped KV tokens · 3.23× reported capacityRetest sampled VRAM: 31.144 GiB max/cardRetest memory reports and limits
Evidence and limits

selected FP16 reference

{
  "strict_tokens": 11170,
  "source": "/home/joe/PhoneSync/Documents/gfx906/baseline/strict-control.json#$",
  "source_revision": "57f22fd70c859e4df9036fcc075a798fb032c31d",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "latency_sources": "[{\"path\": \"/home/joe/PhoneSync/Documents/gfx906/results/gfx906-down-serving-qwen38-706be24-20261021-r0/ladder-1/dense27b_tp4_fullbar_p2pon_20261021T172635Z/c1_2000_proxy/summary.json\", \"sha256\": \"71fe5aebeb67eed51da8a43721550c6dfde41fcad6636aaf0c95eb6ec6606635\"}, {\"path\": \"/home/joe/PhoneSync/Documents/gfx906/results/gfx906-down-serving-qwen38-706be24-20261021-r0/ladder-1/dense27b_tp4_fullbar_p2pon_20261021T172635Z/c1_10000_proxy/summary.json\", \"sha256\": \"a3e5bc940b57d15910b03176bd82187c6535e8d3a14b6f77f3ed0c6558406f16\"}]",
  "recorded_configuration_details": {
    "cell": "FP16_TP4_MTP_off",
    "status": "completed",
    "TP": 4,
    "native_MTP_tokens": 0,
    "host": "10.0.0.50",
    "strict_pass": true,
    "strict_tokens": 11170,
    "strict_diagnostic_TPS": 50.39692268013857,
    "official_2K_TPS": 53.55497506197457,
    "official_2K_tokens": 2000,
    "official_2K_prefill_seconds": 11.319955765997292,
    "official_2K_total_seconds": 48.67580153999734,
    "diagnostic_10K_TPS": 50.62447766842328,
    "diagnostic_10K_tokens": 10000,
    "diagnostic_10K_prefill_seconds": 11.950416606996441,
    "diagnostic_10K_total_seconds": 209.49722924898379,
    "SSM_dtype": "",
    "source": "57f22fd70c859e4df9036fcc075a798fb032c31d",
    "image": "sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04",
    "primary": "tuning/fixed3-down-early-repair/serving-result-57f22fd-20261021.json",
    "strict_prefill_seconds": 11.748157959023956,
    "strict_total_seconds": 233.402228802006,
    "per_rank_memory_JSON": [
      {
        "rank": 0,
        "parameter_logical_bytes": 13449011456,
        "lossless_packed_logical_bytes": 10072367608,
        "receipt": "results/gfx906-down-serving-qwen38-706be24-20261021-r0/loaded-model-rank0-pid250.json"
      },
      {
        "rank": 1,
        "parameter_logical_bytes": 13449011456,
        "lossless_packed_logical_bytes": 10071559696,
        "receipt": "results/gfx906-down-serving-qwen38-706be24-20261021-r0/loaded-model-rank1-pid251.json"
      },
      {
        "rank": 2,
        "parameter_logical_bytes": 13449011456,
        "lossless_packed_logical_bytes": 10071485104,
        "receipt": "results/gfx906-down-serving-qwen38-706be24-20261021-r0/loaded-model-rank2-pid252.json"
      },
      {
        "rank": 3,
        "parameter_logical_bytes": 13449011456,
        "lossless_packed_logical_bytes": 10072117712,
        "receipt": "results/gfx906-down-serving-qwen38-706be24-20261021-r0/loaded-model-rank3-pid253.json"
      }
    ],
    "KV_JSON": {
      "logged_available_cache_memory_GiB": 6.53,
      "available_memory_worker_rank": 0,
      "logged_GPU_KV_cache_tokens": 106800,
      "logged_max_concurrency_for_131072": 3.23,
      "configured_kv_cache_dtype": "auto",
      "actual_tensor_dtype": null,
      "actual_SSM_tensor_dtype": null,
      "actual_block_count": null,
      "actual_block_size": null,
      "evidence": "results/gfx906-down-serving-qwen38-706be24-20261021-r0/service-ready.log",
      "lines": {
        "configuration": 48,
        "available_memory": 1220,
        "token_capacity": 1225,
        "maximum_concurrency": 1226
      },
      "limit": "Engine allocation reports/configuration, not direct cache-tensor observations, full-context inference, or a simple token-count/concurrency arithmetic equivalence. Rounded GiB is not exact bytes."
    },
    "VRAM_JSON": {
      "after_supplemental_request_3_utc": "2026-10-21T17:26:34Z",
      "after_supplemental_by_physical_card_bytes": {
        "1": 33421770752,
        "2": 33441353728,
        "3": 33437638656,
        "4": 33384116224
      },
      "after_ladder1_by_physical_card_bytes": {
        "1": 33421770752,
        "2": 33441353728,
        "3": 33437638656,
        "4": 33384116224
      },
      "samples_match": true,
      "evidence": [
        "results/gfx906-down-serving-qwen38-706be24-20261021-r0/hardware-after-ladder1.txt",
        "results/gfx906-down-serving-qwen38-706be24-20261021-r0/supplemental-warmups/request-3/memory-after.txt"
      ],
      "limit": "Two retained samples, not peak/continuous sampling or model-only allocation; hardware snapshot reports busy0 for all four assigned cards."
    },
    "attention_KV_dtype": "",
    "context": 131072,
    "speculative_accounting_JSON": null
  },
  "detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/fp16-matrix.csv",
  "detail_source_sha256": "8643d90cf1f211a7f77be9f6212cd7706e6dbae3ae84b9bc80b5f42fa199ffe0",
  "detail_source_pointer": "FP16_TP4_MTP_off",
  "sampled_VRAM_after_ladder_bytes_by_rank": {
    "0": 33421770752,
    "1": 33441353728,
    "2": 33437638656,
    "3": 33384116224
  },
  "memory_accounting_limit": "All retained integer packed-buffer logical extents, separate from FP16 parameter extents; aliases/fallback representations are not deduplicated physical allocation. Physical VRAM is one post-ladder observation.",
  "memory_source": "/home/joe/PhoneSync/Documents/gfx906/results/gfx906-down-serving-qwen38-706be24-20261021-r0/hardware-after-ladder1.txt",
  "memory_source_sha256": "e8342ec2a51c7599ffafec4e5a7ab076b186cc3a93d40391d969a666d9055205",
  "retest_strict_tokens": 6668,
  "retest_MTP_accounting_by_tier": {},
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33421369344,
    "1": 33440952320,
    "2": 33438003200,
    "3": 33384120320
  },
  "retest_memory_observation_count": 27,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-fp16-tp4-mtp0/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 6.53,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 6.53,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 106800,
  "retest_memory_log_report": {
    "profile": "mi50-fp16-tp4-mtp0",
    "image": "sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04",
    "container": "c8c0b898e9d81627eee172f20ca0783b4e5a809246b93619c4230e06528aae37",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 22.81,
    "retest_logged_model_load_seconds": 67.548319,
    "retest_logged_available_KV_GiB": 6.53,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 6.53,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 106800,
    "retest_logged_context_capacity_x": 3.23,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-fp16-tp4-mtp0/service-final.log",
    "source_log_sha256": "b6ce15ca515eb9d0dd6c076e8574f46fd917d7b27ba414136defce7cc7e4b42d",
    "sealed_index_sha256": "352796834416d1920746822000adb67699cb30990ae230cd3e920799f86c1230",
    "source_lines": {
      "model_load": {
        "line": 1161,
        "text": "(Worker pid=197) (Worker_TP0 pid=197) INFO 10-26 20:32:47 [gpu_model_runner.py:4584] Model loading took 22.81 GiB memory and 67.548319 seconds"
      },
      "automatic_KV_budget": {
        "line": 1240,
        "text": "(Worker pid=197) (Worker_TP0 pid=197) INFO 10-26 20:34:54 [gpu_worker.py:453] Available KV cache memory: 6.53 GiB"
      },
      "grouped_KV_tokens": {
        "line": 1245,
        "text": "(EngineCore_DP0 pid=179) INFO 10-26 20:34:54 [kv_cache_utils.py:1314] GPU KV cache size: 106,800 tokens"
      },
      "context_capacity": {
        "line": 1246,
        "text": "(EngineCore_DP0 pid=179) INFO 10-26 20:34:54 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 3.23x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-fp16-tp4-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 31.14431381225586,
  "sampled_peak_VRAM_GiB": 31.14468765258789
}
MI50Current Qwen3.8FP16Qwen/Qwen3.8-27BTP2MTP off PASS9596 tokens 19.92519.343 16.503 / 116.89110K: 16.035 / 533.021Strict: 16.509 / 511.505 25.05 / 31.42FP16 parameter extents / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:1c6b141d4c18f1390f718971c6e234633c15b9dfcfaf71d7038772741e70bf88
Runtime image config: sha256:59e1ec913fbc1e54e9dd8c8a4181182779c02037fca6dfead3e5066f880e5dbb
Registry verificationDerived host-portability image; recorded benchmark parent: sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4
Exact compatibility change
Replaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately.
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp2-mtp0
GPU_RENDER_NODES=renderD128,renderD129

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-fp16-tp2-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --user root \
  --workdir /tmp \
  -e HIP_VISIBLE_DEVICES=0,1 \
  -e HF_HUB_OFFLINE=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \
  -e MI50_FP16_KV_CACHE_BYTES=4495769600 \
  -e MI50_FP16_TP=2 \
  -e MI50_FP16_MTP=0 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  -e MAX_JOBS=1 \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp0/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp0,dst=/runwork \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp0/cache/.triton,dst=/root/.triton \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp2-mtp0
GPU_RENDER_NODES=renderD128,renderD129

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-mi50-r3.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 19.973 · eligible10K 19.391 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 7872,
    "content_chars": 17192,
    "content_has_think_close": false,
    "elapsed_seconds": 416.38388748280704,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T14:00:57Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.008383386768400669,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 17192,
    "qwen_answer_text_sha256": "5b3ea80d3153926a3eea07f1ea606c32e149bd79991eb2d0c41e64904fdea9c4",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 17192,
    "qwen_visible_text_sha256": "5b3ea80d3153926a3eea07f1ea606c32e149bd79991eb2d0c41e64904fdea9c4",
    "raw_has_think_close": false,
    "reasoning_chars": 23183,
    "text_chars": 40376,
    "text_sha256": "067067e7523f151e2debbed8668a0976aca60ab82776edd0548deafa8a633379",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 402.27794865053147,
    "vllm_decode_tps": 19.568559565363092,
    "vllm_generation_tokens_delta": 7872.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.096986005082726,
    "vllm_prefill_tps": 33.55327158794496,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 18.9056306851572
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 114.23982309456915,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T14:02:52Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.006374645978212357,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 10371,
    "text_chars": 10371,
    "text_sha256": "e0dc5c200d6b0ff17362adf0ce94343d11e60a7c1bfdacabcdb8395eb52bbf65",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 100.13581881020218,
    "vllm_decode_tps": 19.972873081417628,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.095310986973345,
    "vllm_prefill_tps": 33.55725889532617,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 17.507029911490452
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 28172,
    "content_has_think_close": false,
    "elapsed_seconds": 529.8064922206104,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T14:11:42Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.005857612006366253,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 28172,
    "qwen_answer_text_sha256": "b0448c91cf4839e7736c792b9929579a50b908989111bbbf5d241369beeb9240",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 28172,
    "qwen_visible_text_sha256": "b0448c91cf4839e7736c792b9929579a50b908989111bbbf5d241369beeb9240",
    "raw_has_think_close": false,
    "reasoning_chars": 23183,
    "text_chars": 51356,
    "text_sha256": "1ddcb1e14e4c15ffe80602abbc888941612d36c5c7329d1732bda38660286044",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 515.7007571328431,
    "vllm_decode_tps": 19.391090398232684,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.096721733920276,
    "vllm_prefill_tps": 33.553900610937255,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 18.874815893792444
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS8,748 natural tokens2K 19.87310K 19.302 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T21:48:44.733823837Z",
      "FinishedAt": "2026-10-26T22:36:13.679245812Z"
    },
    "devices_after": [
      {
        "card": 1,
        "render": 129,
        "pci": "0000:06:00.0",
        "actual_card": "card0",
        "busy_percent": 0,
        "vram_bytes": 18457292800,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:0a:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 18462343168,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T22:36:13.825330+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 19.873362185229723,
  "retest_10K_TPS": 19.30213175950557,
  "retest_strict_tokens": 8748,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 8748,
      "elapsed_seconds": 467.49833614297677,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 450.85449509404134,
      "vllm_decode_tps": 19.403155774625915,
      "vllm_generation_tokens_delta": 8748.0,
      "vllm_prefill_seconds_delta": 16.6319247179199,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 116.15711760893464,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 100.63722390600014,
      "vllm_decode_tps": 19.873362185229723,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 15.506479750969447,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 532.3052741159918,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 518.077491367003,
      "vllm_decode_tps": 19.30213175950557,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 14.213958831969649,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 15.506479750969447,
  "retest_2K_total_seconds": 116.15711760893464,
  "retest_10K_prefill_seconds": 14.213958831969649,
  "retest_10K_total_seconds": 532.3052741159918,
  "retest_strict_prefill_seconds": 16.6319247179199,
  "retest_strict_total_seconds": 467.49833614297677,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": "Complete record: chart-details/row-01-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33733324800,
    "1": 33733308416
  },
  "retest_memory_observation_count": 45,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-fp16-tp2-mtp0/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 25.37,
  "retest_logged_model_load_seconds": 16.160363,
  "retest_logged_available_KV_GiB": null,
  "retest_explicit_KV_budget_bytes": 4495769600,
  "retest_KV_budget_GiB": 4.18701171875,
  "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
  "retest_logged_GPU_KV_tokens": 33712,
  "retest_logged_context_capacity_x": 1.02,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-fp16-tp2-mtp0",
    "image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
    "container": "99381d2f6797e72404e8e186fec8c4bd3744a515945517d05b5b8548a689c542",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 25.37,
    "retest_logged_model_load_seconds": 16.160363,
    "retest_logged_available_KV_GiB": null,
    "retest_explicit_KV_budget_bytes": 4495769600,
    "retest_KV_budget_GiB": 4.18701171875,
    "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
    "retest_logged_GPU_KV_tokens": 33712,
    "retest_logged_context_capacity_x": 1.02,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-fp16-tp2-mtp0/service-final.log",
    "source_log_sha256": "9baac6ec7141f36335e506c7777e974cae19185b140c8112860fc3fbb3c05edb",
    "sealed_index_sha256": "e4f8ebde61b26ca3d5a8e83d60d7ec38e19c442b556cfe06683fa5c49a7ecff3",
    "source_lines": {
      "model_load": {
        "line": 122,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 21:50:52 [gpu_model_runner.py:4584] Model loading took 25.37 GiB memory and 16.160363 seconds"
      },
      "explicit_KV_budget": {
        "line": 7,
        "text": "(APIServer pid=1) INFO 10-26 21:49:16 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM']}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.fp16_matrix_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 4495769600, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'optimization_level': '3'}"
      },
      "explicit_budget_worker_reports": [
        {
          "line": 134,
          "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 21:51:57 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.19 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        },
        {
          "line": 135,
          "text": "(Worker pid=197) (Worker_TP1 pid=197) INFO 10-26 21:51:57 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.19 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        }
      ],
      "grouped_KV_tokens": {
        "line": 136,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 21:51:57 [kv_cache_utils.py:1314] GPU KV cache size: 33,712 tokens"
      },
      "context_capacity": {
        "line": 137,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 21:51:57 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.02x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-fp16-tp2-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 31.416606903076172
}
Complete retest loaded admission
Retest KV budget: 4.187 GiB (explicit)33,712 grouped KV tokens · 1.02× reported capacityRetest sampled VRAM: 31.417 GiB max/cardRetest memory reports and limits
Evidence and limits

Completed, independently reviewed FP16 TP2 MTP-off matrix cell; one sample, no promotion or TP4 goal credit.

{
  "strict_tokens": 9596,
  "source": "/home/joe/PhoneSync/Documents/gfx906/tuning/qwen38-fp16-matrix/fp16-tp2-off-result-20261023.json#$",
  "source_revision": "90733170b8b007fca3401b68c04e5bc67e14a3e0",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "attention_KV_dtype": "FP16",
  "memory_accounting_limit": "Logical parameter/cache extents are not unique physical allocations. Hybrid runtime KV token/concurrency reports are retained separately from block geometry; no full-context request was executed.",
  "sampled_VRAM_before_ladder_bytes_by_rank": "{\"0\": 33733308416, \"1\": 33733312512}",
  "sampled_VRAM_after_ladder_bytes_by_rank": "{\"0\": 33733308416, \"1\": 33733312512}",
  "KV_capacity": "{\"SSM\": {\"bytes\": 275251200, \"dtype\": \"torch.float32\", \"shape\": [175, 24, 128, 128]}, \"attention\": {\"bytes\": 280985600, \"dtype\": \"torch.float16\", \"shape\": [175, 2, 784, 2, 256]}, \"block_size\": 784, \"blocks\": 175, \"conv\": {\"bytes\": 5376000, \"dtype\": \"torch.float16\", \"shape\": [175, 3, 5120]}, \"logged_KV_tokens\": 33712, \"logged_max_concurrency_at_131072\": 1.02}",
  "explicit_KV_cache_budget_bytes": 4495769600,
  "post_KV_allocator_allocated_bytes_by_rank": "{\"0\": 31604896256, \"1\": 31604896256}",
  "post_KV_allocator_reserved_bytes_by_rank": "{\"0\": 31878807552, \"1\": 31878807552}",
  "source_evidence_commit": "26c009d",
  "comparison_limit": "Explicit original dense FP16/collective fallback at TP2; TP4 packed/posted guards not widened and native DSOs unchanged. | Explicit 4495769600-byte KV budget, context131072 unchanged; actual capacity admission is configuration evidence, not full-context execution. | Actual target/attention KV/conv FP16 and SSM FP32. Non-MTP packed convolution and recurrent flags remain1. | Original greedy concurrency1, non-eager O3/maxseq2/maxbatched4 requests and scorer; eight warmups and three tracked startup requests excluded. | One official sample, no repeatability/causal gain or TP4 qualification. | Logical parameter/buffer bytes are not unique physical storage. Cache view bytes may alias; no summed physical cache claim. | Loaded route metadata/generated source/maps are not a per-kernel execution trace. | Outer metrics do not establish internal scorer scrape alignment/full settlement trajectory. | KV tokens/concurrency are logged hybrid-cache reports; do not equate175x784 with usable tokens or infer actual131072 execution. | Final kernel journal filter covers assigned four-card PCI scope, broader than TP2 before/after scope. | Read-only retained cache preserves regular files; socket endpoints inventoried, not byte-hashed. | No native MTP head in this cell; all prior failed/historical attempts preserved.",
  "recorded_configuration_details": {
    "cell": "FP16_TP2_MTPoff",
    "status": "completed",
    "TP": 2,
    "native_MTP_tokens": 0,
    "host": "10.0.0.50",
    "strict_pass": true,
    "strict_tokens": 9596,
    "strict_diagnostic_TPS": 19.386496043730823,
    "official_2K_TPS": 19.924717816026906,
    "official_2K_tokens": 2000,
    "official_2K_prefill_seconds": 16.502791117993183,
    "official_2K_total_seconds": 116.89143691700883,
    "diagnostic_10K_TPS": 19.34330520874968,
    "diagnostic_10K_tokens": 10000,
    "diagnostic_10K_prefill_seconds": 16.035188685986213,
    "diagnostic_10K_total_seconds": 533.0213894949993,
    "SSM_dtype": "torch.float32",
    "source": "90733170b8b007fca3401b68c04e5bc67e14a3e0",
    "image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
    "primary": "tuning/qwen38-fp16-matrix/fp16-tp2-off-result-20261023.json",
    "strict_prefill_seconds": 16.508799550996628,
    "strict_total_seconds": 511.50529021699913,
    "per_rank_memory_JSON": [
      {
        "rank": 0,
        "pid": 196,
        "parameter_count": 611,
        "parameter_logical_bytes": 26896673280,
        "parameter_dtypes": [
          "torch.float16"
        ],
        "packed_counts": {
          "_gfx906_fixed3_gate_primary": 0,
          "_gfx906_fixed3_primary": 0,
          "_gfx906_gdn_fixed3_primary": 0,
          "_gfx906_output_fixed3_primary": 0,
          "_gfx906_qkv_fixed3_primary": 0
        },
        "packed_logical_bytes": 0,
        "after_load_allocated_bytes": 27105545728,
        "after_load_reserved_bytes": 27231518720,
        "after_cache_allocated_bytes": 31604896256,
        "after_cache_reserved_bytes": 31878807552,
        "installed_manifest_sha256": "44ea477cf0ae8d0a6da890f35540f66d477c93913eb7d728b45ae58aacb57df0",
        "attention_layers": 16,
        "GDN_layers": 48
      },
      {
        "rank": 1,
        "pid": 197,
        "parameter_count": 611,
        "parameter_logical_bytes": 26896673280,
        "parameter_dtypes": [
          "torch.float16"
        ],
        "packed_counts": {
          "_gfx906_fixed3_gate_primary": 0,
          "_gfx906_fixed3_primary": 0,
          "_gfx906_gdn_fixed3_primary": 0,
          "_gfx906_output_fixed3_primary": 0,
          "_gfx906_qkv_fixed3_primary": 0
        },
        "packed_logical_bytes": 0,
        "after_load_allocated_bytes": 27105545728,
        "after_load_reserved_bytes": 27231518720,
        "after_cache_allocated_bytes": 31604896256,
        "after_cache_reserved_bytes": 31878807552,
        "installed_manifest_sha256": "44ea477cf0ae8d0a6da890f35540f66d477c93913eb7d728b45ae58aacb57df0",
        "attention_layers": 16,
        "GDN_layers": 48
      }
    ],
    "KV_JSON": {
      "attention": {
        "shape": [
          175,
          2,
          784,
          2,
          256
        ],
        "dtype": "torch.float16",
        "bytes": 280985600
      },
      "conv": {
        "shape": [
          175,
          3,
          5120
        ],
        "dtype": "torch.float16",
        "bytes": 5376000
      },
      "SSM": {
        "shape": [
          175,
          24,
          128,
          128
        ],
        "dtype": "torch.float32",
        "bytes": 275251200
      },
      "blocks": 175,
      "block_size": 784,
      "logged_KV_tokens": 33712,
      "logged_max_concurrency_at_131072": 1.02
    },
    "VRAM_JSON": {
      "before": "renderD129 33733308416\nrenderD130 33733312512",
      "after": "renderD129 33733308416\nrenderD130 33733312512"
    },
    "attention_KV_dtype": "torch.float16",
    "context": 131072,
    "speculative_accounting_JSON": null
  },
  "detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/fp16-matrix.csv",
  "detail_source_sha256": "8643d90cf1f211a7f77be9f6212cd7706e6dbae3ae84b9bc80b5f42fa199ffe0",
  "detail_source_pointer": "FP16_TP2_MTPoff",
  "retest_strict_tokens": 8748,
  "retest_MTP_accounting_by_tier": {},
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33733324800,
    "1": 33733308416
  },
  "retest_memory_observation_count": 45,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-fp16-tp2-mtp0/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": null,
  "retest_explicit_KV_budget_bytes": 4495769600,
  "retest_KV_budget_GiB": 4.18701171875,
  "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
  "retest_logged_GPU_KV_tokens": 33712,
  "retest_memory_log_report": {
    "profile": "mi50-fp16-tp2-mtp0",
    "image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
    "container": "99381d2f6797e72404e8e186fec8c4bd3744a515945517d05b5b8548a689c542",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 25.37,
    "retest_logged_model_load_seconds": 16.160363,
    "retest_logged_available_KV_GiB": null,
    "retest_explicit_KV_budget_bytes": 4495769600,
    "retest_KV_budget_GiB": 4.18701171875,
    "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
    "retest_logged_GPU_KV_tokens": 33712,
    "retest_logged_context_capacity_x": 1.02,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-fp16-tp2-mtp0/service-final.log",
    "source_log_sha256": "9baac6ec7141f36335e506c7777e974cae19185b140c8112860fc3fbb3c05edb",
    "sealed_index_sha256": "e4f8ebde61b26ca3d5a8e83d60d7ec38e19c442b556cfe06683fa5c49a7ecff3",
    "source_lines": {
      "model_load": {
        "line": 122,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 21:50:52 [gpu_model_runner.py:4584] Model loading took 25.37 GiB memory and 16.160363 seconds"
      },
      "explicit_KV_budget": {
        "line": 7,
        "text": "(APIServer pid=1) INFO 10-26 21:49:16 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM']}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.fp16_matrix_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 4495769600, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'optimization_level': '3'}"
      },
      "explicit_budget_worker_reports": [
        {
          "line": 134,
          "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 21:51:57 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.19 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        },
        {
          "line": 135,
          "text": "(Worker pid=197) (Worker_TP1 pid=197) INFO 10-26 21:51:57 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.19 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        }
      ],
      "grouped_KV_tokens": {
        "line": 136,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 21:51:57 [kv_cache_utils.py:1314] GPU KV cache size: 33,712 tokens"
      },
      "context_capacity": {
        "line": 137,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 21:51:57 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.02x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-fp16-tp2-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 31.416606903076172,
  "sampled_peak_VRAM_GiB": 31.416595458984375
}
MI50Current Qwen3.8FP16Qwen/Qwen3.8-27BTP4MTP22K accepted 1122 / 1756 (63.90%) PASS7114 tokens 27.81923.428 18.001 / 89.90710K: 18.015 / 444.865Strict: 26.326 / 311.798 12.53 / 29.47retained FP16 parameter extents; projection buffers separate / VRAM GiBSeparate packed-buffer extents: 9.381 GiB/rank; includes alternate views, not additive physical storage. Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:1c6b141d4c18f1390f718971c6e234633c15b9dfcfaf71d7038772741e70bf88
Runtime image config: sha256:59e1ec913fbc1e54e9dd8c8a4181182779c02037fca6dfead3e5066f880e5dbb
Registry verificationDerived host-portability image; recorded benchmark parent: sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4
Exact compatibility change
Replaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately.
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp4-mtp2
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-fp16-tp4-mtp2-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \
  --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \
  -e MI50_FP16_TP=4 \
  -e MI50_FP16_MTP=1 \
  -e HIP_VISIBLE_DEVICES=0,1,2,3 \
  -e HF_HUB_OFFLINE=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  -e MAX_JOBS=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp2/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp2/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp2/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp2,dst=/runwork \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp4-mtp2
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-mi50-r3.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 30.146 · eligible10K 23.866 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 8240,
    "content_chars": 16196,
    "content_has_think_close": false,
    "elapsed_seconds": 338.21908202953637,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T14:51:41Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.008258074522018433,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 16196,
    "qwen_answer_text_sha256": "850242d24439ba048f921216fb301df5a9ed94509f467bf49c1b4e69c45882e5",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 16196,
    "qwen_visible_text_sha256": "850242d24439ba048f921216fb301df5a9ed94509f467bf49c1b4e69c45882e5",
    "raw_has_think_close": false,
    "reasoning_chars": 26490,
    "text_chars": 42687,
    "text_sha256": "f0646379f7a2e598a9d348a84e9e4e9fc44af518966167d370e66b73b2c3259f",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 325.45470919366926,
    "vllm_decode_tps": 25.318423016262457,
    "vllm_generation_tokens_delta": 8240.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 12.75549241527915,
    "vllm_prefill_tps": 37.08206509012679,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 24.362906878448708
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 79.18325763661414,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T14:53:00Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.00859201792627573,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 8930,
    "text_chars": 8930,
    "text_sha256": "c3ada5acb4981cef88965e9d67305b457dc31cc108eea3e48c5b766e2dbfa012",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 66.3432620735839,
    "vllm_decode_tps": 30.146241494452322,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 12.832328528165817,
    "vllm_prefill_tps": 36.86002886863496,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 25.257864600347094
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 25611,
    "content_has_think_close": false,
    "elapsed_seconds": 431.6993595920503,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T15:00:12Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008361870422959328,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 25611,
    "qwen_answer_text_sha256": "f08a2e9e6af3876e798eddcc7b7904cdb6d70e093f95d7510c8e71b6d3ccf434",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 25611,
    "qwen_visible_text_sha256": "f08a2e9e6af3876e798eddcc7b7904cdb6d70e093f95d7510c8e71b6d3ccf434",
    "raw_has_think_close": false,
    "reasoning_chars": 26490,
    "text_chars": 52102,
    "text_sha256": "d7dd0e23e90221d9579b7b90513868c19cd7a8354623944daf195f0a241a3936",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 418.99922180734575,
    "vllm_decode_tps": 23.86639277482468,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 12.691327778622508,
    "vllm_prefill_tps": 37.26954407376739,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 23.16426878522557
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestMTP counter replayPASS11,252 natural tokens2K 27.10310K 22.724 · diagnostic2K MTP accepted 1109 / 1782 (62.23%)
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T20:59:39.988715707Z",
      "FinishedAt": "2026-10-26T21:46:24.608013465Z"
    },
    "devices_after": [
      {
        "card": 1,
        "render": 129,
        "pci": "0000:06:00.0",
        "actual_card": "card0",
        "busy_percent": 0,
        "vram_bytes": 25278005248,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:0a:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 25393348608,
        "dpm": "auto"
      },
      {
        "card": 3,
        "render": 131,
        "pci": "0000:0d:00.0",
        "actual_card": "card3",
        "busy_percent": 0,
        "vram_bytes": 25848430592,
        "dpm": "auto"
      },
      {
        "card": 4,
        "render": 132,
        "pci": "0000:10:00.0",
        "actual_card": "card4",
        "busy_percent": 0,
        "vram_bytes": 25334628352,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T21:46:24.749264+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 27.10269994925174,
  "retest_10K_TPS": 22.72398827389817,
  "retest_strict_tokens": 11252,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 11252,
      "elapsed_seconds": 532.8966229540529,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 507.4504561870126,
      "vllm_decode_tps": 22.17359323025864,
      "vllm_generation_tokens_delta": 11252.0,
      "vllm_prefill_seconds_delta": 25.432538520894013,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 98.17414218490012,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 73.7933860369958,
      "vllm_decode_tps": 27.10269994925174,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 24.37041282595601,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 464.32734813098796,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 440.06359620799776,
      "vllm_decode_tps": 22.72398827389817,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 24.250391291920096,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 24.37041282595601,
  "retest_2K_total_seconds": 98.17414218490012,
  "retest_10K_prefill_seconds": 24.250391291920096,
  "retest_10K_total_seconds": 464.32734813098796,
  "retest_strict_prefill_seconds": 25.432538520894013,
  "retest_strict_total_seconds": 532.8966229540529,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6926.0,
        "vllm:spec_decode_num_drafts_total": 4326.0,
        "vllm:request_generation_tokens_sum": 11252.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6926.0,
        "vllm:generation_tokens_total": 11252.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 8652.0
      },
      "accepted_per_position": {
        "0": 3729.0,
        "1": 3197.0
      },
      "accepted_fraction": 0.8005085529357374,
      "committed_emitted_tokens": 11252.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "29c2ed4ec314d2eb6a82fa83a792c5b39f1efa0c03863de816efe5de6b3540dc",
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "43f683bf9787a3fabc5c5d56c85b9e7fa4b9d476097a175297a7bd013ce52a0d"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 1782.0,
        "vllm:spec_decode_num_drafts_total": 891.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1109.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1109.0,
        "vllm:generation_tokens_total": 2000.0
      },
      "accepted_per_position": {
        "0": 667.0,
        "1": 442.0
      },
      "accepted_fraction": 0.622334455667789,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "44c060e7c7f6ad1f8d10cfb4974c2f609f97c8488318396b5e2d1d1776646fe8",
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "e0fe496bed1252fd10a56f00ec24c4b278fd954cce3899754f7921b693c9239c"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6092.0,
        "vllm:spec_decode_num_drafts_total": 3909.0,
        "vllm:spec_decode_num_draft_tokens_total": 7818.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6092.0
      },
      "accepted_per_position": {
        "0": 3312.0,
        "1": 2780.0
      },
      "accepted_fraction": 0.7792274238935789,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "c9b911e4b1a22b9368c93204f4938392aafed794a173dcf60d30f084df540283",
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "052278aac272623437a14aa1a168e01093f65f46c73b392ddecd652f3ce4e37c"
      }
    }
  },
  "retest_MTP_2K_accepted": 1109.0,
  "retest_MTP_2K_proposed": 1782.0,
  "retest_MTP_2K_draft_batches": 891.0,
  "retest_MTP_2K_acceptance_percent": 62.2334455667789,
  "retest_loaded_admission": "Complete record: chart-details/row-02-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 31621283840,
    "1": 31639273472,
    "2": 31637164032,
    "3": 31580545024
  },
  "retest_memory_observation_count": 44,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-fp16-tp4-mtp2/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 24.18,
  "retest_logged_model_load_seconds": 82.437392,
  "retest_logged_available_KV_GiB": 4.79,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 4.79,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 73600,
  "retest_logged_context_capacity_x": 2.13,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-fp16-tp4-mtp2",
    "image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
    "container": "a6900dc2002295b8c6786cb3cbe72fc0905467b837eb0c385bd9c52d17f73453",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 24.18,
    "retest_logged_model_load_seconds": 82.437392,
    "retest_logged_available_KV_GiB": 4.79,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 4.79,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 73600,
    "retest_logged_context_capacity_x": 2.13,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-fp16-tp4-mtp2/service-final.log",
    "source_log_sha256": "01da5b75e4ef49cb1d7d6bc6ad0f0bedd32b3e985961684407826dc906561b9a",
    "sealed_index_sha256": "287b09e42de876cf580b22e48e3779b0b496040a93d8af8611dfe440da40deec",
    "source_lines": {
      "model_load": {
        "line": 1173,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 21:03:21 [gpu_model_runner.py:4584] Model loading took 24.18 GiB memory and 82.437392 seconds"
      },
      "automatic_KV_budget": {
        "line": 1317,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 21:09:19 [gpu_worker.py:453] Available KV cache memory: 4.79 GiB"
      },
      "grouped_KV_tokens": {
        "line": 1321,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 21:09:19 [kv_cache_utils.py:1314] GPU KV cache size: 73,600 tokens"
      },
      "context_capacity": {
        "line": 1322,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 21:09:19 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 2.13x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-fp16-tp4-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 29.46636962890625
}
Complete retest loaded admission
Retest KV budget: 4.790 GiB (automatic, rounded)73,600 grouped KV tokens · 2.13× reported capacityRetest sampled VRAM: 29.466 GiB max/cardRetest memory reports and limits
Evidence and limits

Completed, independently reviewed FP16 TP4 native MTP2 matrix cell; separate from non-speculative FP16 qualification. No promotion.

{
  "strict_tokens": 7114,
  "source": "/home/joe/PhoneSync/Documents/gfx906/tuning/qwen38-fp16-matrix/fp16-tp4-mtp-result-20261023.json#$",
  "source_revision": "90733170b8b007fca3401b68c04e5bc67e14a3e0",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "attention_KV_dtype": "FP16",
  "memory_accounting_limit": "Parameter lists counted once by name and route buffer fields once by module/field. Parameter and packed-buffer logical extents are reported separately, not summed as physical storage. Shared draft embedding/head excluded from unshared head bytes; views/allocator aliases are not de-duplicated physical storage.",
  "sampled_VRAM_before_ladder_bytes_by_rank": "{\"0\": 31620239360, \"1\": 31640121344, \"2\": 31637413888, \"3\": 31580463104}",
  "sampled_VRAM_after_ladder_bytes_by_rank": "{\"0\": 31620239360, \"1\": 31640121344, \"2\": 31637413888, \"3\": 31580463104}",
  "KV_capacity": "{\"actual_blocks_per_physical_pool\": 368, \"full_attention_block_tokens\": 800, \"limit\": \"Different hybrid group accounting; do not substitute naive blocks\\u00d7tokens for logged capacity or claim actual full-context inference.\", \"logged_GPU_KV_cache_tokens\": 73600, \"logged_available_KV_GiB\": 4.79, \"logged_maximum_concurrency_for_131072\": 2.13, \"physical_pools\": 17}",
  "post_KV_allocator_allocated_bytes_by_rank": "{\"0\": 29316868096, \"1\": 29315055104, \"2\": 29315478016, \"3\": 29319622656}",
  "post_KV_allocator_reserved_bytes_by_rank": "{\"0\": 29806821376, \"1\": 29808918528, \"2\": 29806821376, \"3\": 29800529920}",
  "MTP_accounting_by_tier": {
    "tiers": {
      "strict": {
        "generation_tokens_total": 7114.0,
        "spec_decode_num_accepted_tokens_per_pos_total_0": 2349.0,
        "spec_decode_num_accepted_tokens_per_pos_total_1": 1986.0,
        "spec_decode_num_accepted_tokens_total": 4335.0,
        "spec_decode_num_draft_tokens_total": 5560.0,
        "spec_decode_num_drafts_total": 2780.0,
        "adjusted_unaccepted_proposed_minus_accepted": 1225.0,
        "iteration_histogram_count_delta": 2781.0,
        "iteration_histogram_sum_delta": 7587.0,
        "full_report": "c1_128_strict.json"
      },
      "fixed_2000": {
        "generation_tokens_total": 2000.0,
        "spec_decode_num_accepted_tokens_per_pos_total_0": 646.0,
        "spec_decode_num_accepted_tokens_per_pos_total_1": 476.0,
        "spec_decode_num_accepted_tokens_total": 1122.0,
        "spec_decode_num_draft_tokens_total": 1756.0,
        "spec_decode_num_drafts_total": 878.0,
        "adjusted_unaccepted_proposed_minus_accepted": 634.0,
        "iteration_histogram_count_delta": 879.0,
        "iteration_histogram_sum_delta": 2473.0,
        "full_report": "c1_2000.json"
      },
      "diagnostic_10000": {
        "generation_tokens_total": 10000.0,
        "spec_decode_num_accepted_tokens_per_pos_total_0": 3312.0,
        "spec_decode_num_accepted_tokens_per_pos_total_1": 2886.0,
        "spec_decode_num_accepted_tokens_total": 6198.0,
        "spec_decode_num_draft_tokens_total": 7602.0,
        "spec_decode_num_drafts_total": 3801.0,
        "adjusted_unaccepted_proposed_minus_accepted": 1404.0,
        "iteration_histogram_count_delta": 3802.0,
        "iteration_histogram_sum_delta": 10473.0,
        "full_report": "c1_10000.json"
      }
    },
    "limits": [
      "Returned completion tokens determine fixed-tier length; drafts and accepted proposals are separate counters.",
      "Proposed minus accepted is adjusted unaccepted, not scheduler rollback rejection.",
      "Iteration histogram counts and sums include prefill; they do not give pure decode-step emissions or exact 1/2/3 token chronology.",
      "Warmups are excluded from the three reported tiers; all eleven reports remain in the linked primary."
    ],
    "all_samples_and_histograms_primary": "tuning/qwen38-fp16-matrix/fp16-tp4-mtp-result-20261023.json"
  },
  "MTP_accounting_limit": "TPS uses committed returned tokens only. Proposed minus accepted is adjusted unaccepted, not a scheduler rollback count; iteration counts include prefill.",
  "comparison_limit": "Native two-token MTP and same-checkpoint unquantized draft head; no quantization. | Target TP4 fixed3/interleaved/posted routes retained; exact draft prefixes excluded from target packing. | MTP cache mode none; FP16 activation/attention KV and explicitly float32 SSM. | Packed QKVZ convolution disabled to preserve history5 compatibility; generic speculative rollback unchanged; packed non-spec recurrent decode remains enabled. | MTP compile cache disabled; original context/O3/non-eager/maxseq2/maxbatched4 retained. | One-time loaded metadata worker; no native arithmetic or binary change. | Native MTP cell is separate from the original non-speculative FP16 qualification. | Strict PASS is sole model correctness result; forced-length tier parser invalidity does not replace strict adjudication. | Accepted/proposed/draft counters differ from returned outputs; P-A is not scheduler rollback rejection. | Outer snapshots cannot recover per-step committed-token distribution or internal scrape trajectory. | No promotion, target credit, causal speed claim or repetition is inferred. | All earlier failed quantized/MTP and source/runtime attempts remain historical; none overwritten. | Actual368 blocks,800-token full-attention blocks,17 physical pools; logical tensor capacity is not a successful131072-token request or unique allocation proof. | Parameter lists counted once by name and route buffer fields once by module/field. Parameter and packed-buffer logical extents are reported separately, not summed as physical storage. Shared draft embedding/head excluded from unshared head bytes; views/allocator aliases are not de-duplicated physical storage. | Owner observed bash -lc no-op explicit exit0 returns1, versus bash -c returns0. Preserve outer SSH1 separately from remote startup/ladder/closure0; not a runtime failure or all-SSH0 claim.",
  "recorded_configuration_details": {
    "cell": "FP16_TP4_MTP2",
    "status": "completed",
    "TP": 4,
    "native_MTP_tokens": 2,
    "host": "10.0.0.50",
    "strict_pass": true,
    "strict_tokens": 7114,
    "strict_diagnostic_TPS": 24.92135282587446,
    "official_2K_TPS": 27.818932990951446,
    "official_2K_tokens": 2000,
    "official_2K_prefill_seconds": 18.001278546988033,
    "official_2K_total_seconds": 89.90659736702219,
    "diagnostic_10K_TPS": 23.42828193269942,
    "diagnostic_10K_tokens": 10000,
    "diagnostic_10K_prefill_seconds": 18.015202388050966,
    "diagnostic_10K_total_seconds": 444.86478443199303,
    "SSM_dtype": "float32",
    "source": "90733170b8b007fca3401b68c04e5bc67e14a3e0",
    "image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
    "primary": "tuning/qwen38-fp16-matrix/fp16-tp4-mtp-result-20261023.json",
    "strict_prefill_seconds": 26.3263557300088,
    "strict_total_seconds": 311.7976599709946,
    "per_rank_memory_JSON": [
      {
        "rank": 0,
        "target_parameter_logical_bytes": 13449011456,
        "packed_buffer_logical_bytes": 10072367608,
        "after_load_allocated_bytes": 24180564992,
        "after_load_reserved_bytes": 25945964544,
        "after_cache_allocated_bytes": 29316868096,
        "after_cache_reserved_bytes": 29806821376
      },
      {
        "rank": 1,
        "target_parameter_logical_bytes": 13449011456,
        "packed_buffer_logical_bytes": 10071559696,
        "after_load_allocated_bytes": 24178752000,
        "after_load_reserved_bytes": 25948061696,
        "after_cache_allocated_bytes": 29315055104,
        "after_cache_reserved_bytes": 29808918528
      },
      {
        "rank": 2,
        "target_parameter_logical_bytes": 13449011456,
        "packed_buffer_logical_bytes": 10071485104,
        "after_load_allocated_bytes": 24179174912,
        "after_load_reserved_bytes": 25945964544,
        "after_cache_allocated_bytes": 29315478016,
        "after_cache_reserved_bytes": 29806821376
      },
      {
        "rank": 3,
        "target_parameter_logical_bytes": 13449011456,
        "packed_buffer_logical_bytes": 10072117712,
        "after_load_allocated_bytes": 24183319552,
        "after_load_reserved_bytes": 25939673088,
        "after_cache_allocated_bytes": 29319622656,
        "after_cache_reserved_bytes": 29800529920
      }
    ],
    "KV_JSON": {
      "actual_blocks_per_physical_pool": 368,
      "full_attention_block_tokens": 800,
      "physical_pools": 17,
      "logged_GPU_KV_cache_tokens": 73600,
      "logged_maximum_concurrency_for_131072": 2.13,
      "logged_available_KV_GiB": 4.79,
      "limit": "Different hybrid group accounting; do not substitute naive blocks\u00d7tokens for logged capacity or claim actual full-context inference."
    },
    "VRAM_JSON": {
      "before": "renderD129 31620239360\nrenderD130 31640121344\nrenderD131 31637413888\nrenderD132 31580463104\n",
      "after": "renderD129 31620239360\nrenderD130 31640121344\nrenderD131 31637413888\nrenderD132 31580463104\n"
    },
    "attention_KV_dtype": "FP16",
    "context": 131072,
    "speculative_accounting_JSON": {
      "tiers": {
        "strict": {
          "generation_tokens_total": 7114.0,
          "spec_decode_num_accepted_tokens_per_pos_total_0": 2349.0,
          "spec_decode_num_accepted_tokens_per_pos_total_1": 1986.0,
          "spec_decode_num_accepted_tokens_total": 4335.0,
          "spec_decode_num_draft_tokens_total": 5560.0,
          "spec_decode_num_drafts_total": 2780.0,
          "adjusted_unaccepted_proposed_minus_accepted": 1225.0,
          "iteration_histogram_count_delta": 2781.0,
          "iteration_histogram_sum_delta": 7587.0,
          "full_report": "c1_128_strict.json"
        },
        "fixed_2000": {
          "generation_tokens_total": 2000.0,
          "spec_decode_num_accepted_tokens_per_pos_total_0": 646.0,
          "spec_decode_num_accepted_tokens_per_pos_total_1": 476.0,
          "spec_decode_num_accepted_tokens_total": 1122.0,
          "spec_decode_num_draft_tokens_total": 1756.0,
          "spec_decode_num_drafts_total": 878.0,
          "adjusted_unaccepted_proposed_minus_accepted": 634.0,
          "iteration_histogram_count_delta": 879.0,
          "iteration_histogram_sum_delta": 2473.0,
          "full_report": "c1_2000.json"
        },
        "diagnostic_10000": {
          "generation_tokens_total": 10000.0,
          "spec_decode_num_accepted_tokens_per_pos_total_0": 3312.0,
          "spec_decode_num_accepted_tokens_per_pos_total_1": 2886.0,
          "spec_decode_num_accepted_tokens_total": 6198.0,
          "spec_decode_num_draft_tokens_total": 7602.0,
          "spec_decode_num_drafts_total": 3801.0,
          "adjusted_unaccepted_proposed_minus_accepted": 1404.0,
          "iteration_histogram_count_delta": 3802.0,
          "iteration_histogram_sum_delta": 10473.0,
          "full_report": "c1_10000.json"
        }
      },
      "limits": [
        "Returned completion tokens determine fixed-tier length; drafts and accepted proposals are separate counters.",
        "Proposed minus accepted is adjusted unaccepted, not scheduler rollback rejection.",
        "Iteration histogram counts and sums include prefill; they do not give pure decode-step emissions or exact 1/2/3 token chronology.",
        "Warmups are excluded from the three reported tiers; all eleven reports remain in the linked primary."
      ],
      "all_samples_and_histograms_primary": "tuning/qwen38-fp16-matrix/fp16-tp4-mtp-result-20261023.json"
    }
  },
  "MTP_2K_draft_batches": 878.0,
  "MTP_2K_proposed": 1756.0,
  "MTP_2K_accepted": 1122.0,
  "MTP_2K_acceptance_percent": 63.89521640091116,
  "MTP_2K_committed": 2000.0,
  "detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/fp16-matrix.csv",
  "detail_source_sha256": "8643d90cf1f211a7f77be9f6212cd7706e6dbae3ae84b9bc80b5f42fa199ffe0",
  "detail_source_pointer": "FP16_TP4_MTP2",
  "retest_strict_tokens": 11252,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6926.0,
        "vllm:spec_decode_num_drafts_total": 4326.0,
        "vllm:request_generation_tokens_sum": 11252.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6926.0,
        "vllm:generation_tokens_total": 11252.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 8652.0
      },
      "accepted_per_position": {
        "0": 3729.0,
        "1": 3197.0
      },
      "accepted_fraction": 0.8005085529357374,
      "committed_emitted_tokens": 11252.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "29c2ed4ec314d2eb6a82fa83a792c5b39f1efa0c03863de816efe5de6b3540dc",
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "43f683bf9787a3fabc5c5d56c85b9e7fa4b9d476097a175297a7bd013ce52a0d"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 1782.0,
        "vllm:spec_decode_num_drafts_total": 891.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1109.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1109.0,
        "vllm:generation_tokens_total": 2000.0
      },
      "accepted_per_position": {
        "0": 667.0,
        "1": 442.0
      },
      "accepted_fraction": 0.622334455667789,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "44c060e7c7f6ad1f8d10cfb4974c2f609f97c8488318396b5e2d1d1776646fe8",
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "e0fe496bed1252fd10a56f00ec24c4b278fd954cce3899754f7921b693c9239c"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6092.0,
        "vllm:spec_decode_num_drafts_total": 3909.0,
        "vllm:spec_decode_num_draft_tokens_total": 7818.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6092.0
      },
      "accepted_per_position": {
        "0": 3312.0,
        "1": 2780.0
      },
      "accepted_fraction": 0.7792274238935789,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "c9b911e4b1a22b9368c93204f4938392aafed794a173dcf60d30f084df540283",
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "052278aac272623437a14aa1a168e01093f65f46c73b392ddecd652f3ce4e37c"
      }
    }
  },
  "retest_MTP_2K_accepted": 1109.0,
  "retest_MTP_2K_proposed": 1782.0,
  "retest_MTP_2K_draft_batches": 891.0,
  "retest_MTP_2K_acceptance_percent": 62.2334455667789,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 31621283840,
    "1": 31639273472,
    "2": 31637164032,
    "3": 31580545024
  },
  "retest_memory_observation_count": 44,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-fp16-tp4-mtp2/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 4.79,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 4.79,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 73600,
  "retest_memory_log_report": {
    "profile": "mi50-fp16-tp4-mtp2",
    "image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
    "container": "a6900dc2002295b8c6786cb3cbe72fc0905467b837eb0c385bd9c52d17f73453",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 24.18,
    "retest_logged_model_load_seconds": 82.437392,
    "retest_logged_available_KV_GiB": 4.79,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 4.79,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 73600,
    "retest_logged_context_capacity_x": 2.13,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-fp16-tp4-mtp2/service-final.log",
    "source_log_sha256": "01da5b75e4ef49cb1d7d6bc6ad0f0bedd32b3e985961684407826dc906561b9a",
    "sealed_index_sha256": "287b09e42de876cf580b22e48e3779b0b496040a93d8af8611dfe440da40deec",
    "source_lines": {
      "model_load": {
        "line": 1173,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 21:03:21 [gpu_model_runner.py:4584] Model loading took 24.18 GiB memory and 82.437392 seconds"
      },
      "automatic_KV_budget": {
        "line": 1317,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 21:09:19 [gpu_worker.py:453] Available KV cache memory: 4.79 GiB"
      },
      "grouped_KV_tokens": {
        "line": 1321,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 21:09:19 [kv_cache_utils.py:1314] GPU KV cache size: 73,600 tokens"
      },
      "context_capacity": {
        "line": 1322,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 21:09:19 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 2.13x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-fp16-tp4-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 29.46636962890625,
  "sampled_peak_VRAM_GiB": 29.467159271240234
}
MI50Current Qwen3.8FP16Qwen/Qwen3.8-27BTP2MTP22K accepted 1119 / 1764 (63.44%) PASS8452 tokens 18.15617.195 24.747 / 134.91610K: 24.715 / 606.308Strict: 18.314 / 495.761 25.05 / 31.50FP16 parameter extents / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:1c6b141d4c18f1390f718971c6e234633c15b9dfcfaf71d7038772741e70bf88
Runtime image config: sha256:59e1ec913fbc1e54e9dd8c8a4181182779c02037fca6dfead3e5066f880e5dbb
Registry verificationDerived host-portability image; recorded benchmark parent: sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4
Exact compatibility change
Replaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately.
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp2-mtp2
GPU_RENDER_NODES=renderD128,renderD129

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-fp16-tp2-mtp2-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --user root \
  --workdir /tmp \
  -e MI50_FP16_TP=2 \
  -e MI50_FP16_MTP=1 \
  -e HIP_VISIBLE_DEVICES=0,1 \
  -e HF_HUB_OFFLINE=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \
  -e MI50_FP16_KV_CACHE_BYTES=4929945600 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  -e MAX_JOBS=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp2/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp2/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp2/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp2,dst=/runwork \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp2-mtp2
GPU_RENDER_NODES=renderD128,renderD129

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-mi50-r3.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 18.303 · eligible10K 16.868 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 7733,
    "content_chars": 15664,
    "content_has_think_close": false,
    "elapsed_seconds": 454.4751425785944,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T15:47:21Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.008119790814816952,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 15664,
    "qwen_answer_text_sha256": "8164b1a96681e827dd0917ca18f9a1b538da8a1e71026cdf89d328166d5e6319",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 15664,
    "qwen_visible_text_sha256": "8164b1a96681e827dd0917ca18f9a1b538da8a1e71026cdf89d328166d5e6319",
    "raw_has_think_close": false,
    "reasoning_chars": 24035,
    "text_chars": 39700,
    "text_sha256": "a3b515cfead4289fa8a99c3d092d9bfe6915e7ad59e6d7142f40264fb59b53c7",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 439.371760125272,
    "vllm_decode_tps": 17.600129780291745,
    "vllm_generation_tokens_delta": 7733.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.093696534633636,
    "vllm_prefill_tps": 31.337585124668795,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 17.01523202374638
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 124.39862818829715,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T15:49:26Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008269759826362133,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9971,
    "text_chars": 9971,
    "text_sha256": "9eed22714e313734d169dd51b23d37c39b31db6c5bd9f411f04a191ebb554027",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 109.26931860670447,
    "vllm_decode_tps": 18.303399577320008,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.09930254612118,
    "vllm_prefill_tps": 31.325950225529304,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 16.077347709756744
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 27909,
    "content_has_think_close": false,
    "elapsed_seconds": 607.9516332549974,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T15:59:34Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.007648466154932976,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 27909,
    "qwen_answer_text_sha256": "a81f6effa1157fb8ecc4b9944da9aeb6aad02c9a5e3f1768aa7e23f383af6794",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 27909,
    "qwen_visible_text_sha256": "a81f6effa1157fb8ecc4b9944da9aeb6aad02c9a5e3f1768aa7e23f383af6794",
    "raw_has_think_close": false,
    "reasoning_chars": 24035,
    "text_chars": 51945,
    "text_sha256": "d4454027af91bd30f70c2174af129107b4b93f98393fb941f3c2f53db3a26159",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 592.8418835354969,
    "vllm_decode_tps": 16.86790403600295,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.100885062478483,
    "vllm_prefill_tps": 31.322667382938633,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 16.44867692263544
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestMTP counter replayPASS9,193 natural tokens2K 17.98710K 16.598 · diagnostic2K MTP accepted 1111 / 1780 (62.42%)
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T22:38:26.508242705Z",
      "FinishedAt": "2026-10-26T23:32:28.555393816Z"
    },
    "devices_after": [
      {
        "card": 1,
        "render": 129,
        "pci": "0000:06:00.0",
        "actual_card": "card0",
        "busy_percent": 0,
        "vram_bytes": 28219174912,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:0a:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 27819536384,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T23:32:28.696241+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 17.986992732445394,
  "retest_10K_TPS": 16.597632050314704,
  "retest_strict_tokens": 9193,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 9193,
      "elapsed_seconds": 562.7220106379827,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 543.6877332479926,
      "vllm_decode_tps": 16.908602931099775,
      "vllm_generation_tokens_delta": 9193.0,
      "vllm_prefill_seconds_delta": 19.022434039972723,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 130.33776032400783,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 111.19146094901953,
      "vllm_decode_tps": 17.986992732445394,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 19.13075027789455,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 625.5376734989695,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 602.4955830859253,
      "vllm_decode_tps": 16.597632050314704,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 23.02978788409382,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 19.13075027789455,
  "retest_2K_total_seconds": 130.33776032400783,
  "retest_10K_prefill_seconds": 23.02978788409382,
  "retest_10K_total_seconds": 625.5376734989695,
  "retest_strict_prefill_seconds": 19.022434039972723,
  "retest_strict_total_seconds": 562.7220106379827,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 5591.0,
        "vllm:spec_decode_num_draft_tokens_total": 7204.0,
        "vllm:generation_tokens_total": 9193.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 5591.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:request_generation_tokens_sum": 9193.0,
        "vllm:spec_decode_num_drafts_total": 3602.0
      },
      "accepted_per_position": {
        "0": 3054.0,
        "1": 2537.0
      },
      "accepted_fraction": 0.7760966129927818,
      "committed_emitted_tokens": 9193.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "27fa225cc8c4a6d7f8827719f2bd337459e89a5177a1bcf2cc19615df8298915",
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "88bc7a5f4e148bcaad2371bb058820ccffa01b02023c18216c6e58318503e5d5"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1111.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1111.0,
        "vllm:spec_decode_num_drafts_total": 890.0,
        "vllm:spec_decode_num_draft_tokens_total": 1780.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:generation_tokens_total": 2000.0
      },
      "accepted_per_position": {
        "0": 663.0,
        "1": 448.0
      },
      "accepted_fraction": 0.6241573033707866,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "88bc7a5f4e148bcaad2371bb058820ccffa01b02023c18216c6e58318503e5d5",
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "0a2b24d4501925f978e6ed227d810b69d2b565bbb4b2ac956c9270723cebf9ce"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:generation_tokens_total": 10000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6100.0,
        "vllm:spec_decode_num_drafts_total": 3900.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6100.0,
        "vllm:spec_decode_num_draft_tokens_total": 7800.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:request_generation_tokens_count": 1.0
      },
      "accepted_per_position": {
        "0": 3325.0,
        "1": 2775.0
      },
      "accepted_fraction": 0.782051282051282,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "9709d01df025fec6ae6aeaee1653bcd74d5bd21bfe14c67de92a1daa08723393",
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "106866b7537531838ae81399415e724a034b478609f3d59bf2ed18afd6dbc442"
      }
    }
  },
  "retest_MTP_2K_accepted": 1111.0,
  "retest_MTP_2K_proposed": 1780.0,
  "retest_MTP_2K_draft_batches": 890.0,
  "retest_MTP_2K_acceptance_percent": 62.41573033707866,
  "retest_loaded_admission": "Complete record: chart-details/row-03-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 34144878592,
    "1": 34144944128
  },
  "retest_memory_observation_count": 51,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-fp16-tp2-mtp2/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 28.13,
  "retest_logged_model_load_seconds": 17.482613,
  "retest_logged_available_KV_GiB": null,
  "retest_explicit_KV_budget_bytes": 4929945600,
  "retest_KV_budget_GiB": 4.59136962890625,
  "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
  "retest_logged_GPU_KV_tokens": 35200,
  "retest_logged_context_capacity_x": 1.02,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-fp16-tp2-mtp2",
    "image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
    "container": "01770f22e65cc4feb5e7ba102d94ef9688bb226605fc07e4cce66124eb50ca8a",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 28.13,
    "retest_logged_model_load_seconds": 17.482613,
    "retest_logged_available_KV_GiB": null,
    "retest_explicit_KV_budget_bytes": 4929945600,
    "retest_KV_budget_GiB": 4.59136962890625,
    "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
    "retest_logged_GPU_KV_tokens": 35200,
    "retest_logged_context_capacity_x": 1.02,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-fp16-tp2-mtp2/service-final.log",
    "source_log_sha256": "c9b1261d2fad38310f2950a7f12730e059bee01e33ed9344f95799be1edcf54f",
    "sealed_index_sha256": "c66a8e5ffb498b9534473eb9e911f1479f90a4415fd1b1f2266c5f3f84e3e2f8",
    "source_lines": {
      "model_load": {
        "line": 148,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 22:40:54 [gpu_model_runner.py:4584] Model loading took 28.13 GiB memory and 17.482613 seconds"
      },
      "explicit_KV_budget": {
        "line": 7,
        "text": "(APIServer pid=1) INFO 10-26 22:38:57 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM']}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.fp16_matrix_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 4929945600, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 2}, 'optimization_level': '3'}"
      },
      "explicit_budget_worker_reports": [
        {
          "line": 160,
          "text": "(Worker pid=217) (Worker_TP1 pid=217) INFO 10-26 22:41:46 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.59 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        },
        {
          "line": 161,
          "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 22:41:46 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.59 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        }
      ],
      "grouped_KV_tokens": {
        "line": 163,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 22:41:46 [kv_cache_utils.py:1314] GPU KV cache size: 35,200 tokens"
      },
      "context_capacity": {
        "line": 164,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 22:41:46 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.02x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-fp16-tp2-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 31.799957275390625
}
Complete retest loaded admission
Retest KV budget: 4.591 GiB (explicit)35,200 grouped KV tokens · 1.02× reported capacityRetest sampled VRAM: 31.800 GiB max/cardRetest memory reports and limits
Evidence and limits

Completed, independently reviewed FP16 TP2 native MTP2 cell; one strict-valid sample, no promotion or non-speculative FP16 qualification.

{
  "strict_tokens": 8452,
  "source": "/home/joe/PhoneSync/Documents/gfx906/tuning/qwen38-fp16-matrix/fp16-tp2-mtp-result-20261023.json#$",
  "source_revision": "90733170b8b007fca3401b68c04e5bc67e14a3e0",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "attention_KV_dtype": "torch.float16",
  "memory_accounting_limit": "Logical target weights and unshared native-head parameters are reported separately. Shared embedding/head overlaps are excluded from unshared head bytes. Cache views may alias; logical bytes, allocator counters and sampled physical VRAM must not be summed as disjoint storage. Allocation is not a full-context inference result.",
  "sampled_VRAM_before_ladder_bytes_by_rank": "{\"0\": 33825607680, \"1\": 33825673216}",
  "sampled_VRAM_after_ladder_bytes_by_rank": "{\"0\": 33825607680, \"1\": 33825673216}",
  "KV_capacity": "{\"SSM\": {\"bytes\": 278396928, \"dtype\": \"torch.float32\", \"shape\": [177, 24, 128, 128]}, \"attention\": {\"bytes\": 289996800, \"dtype\": \"torch.float16\", \"shape\": [177, 2, 800, 2, 256]}, \"block_size\": 800, \"blocks\": 177, \"conv\": {\"bytes\": 9062400, \"dtype\": \"torch.float16\", \"shape\": [177, 5, 5120]}, \"logged_KV_tokens\": 35200, \"logged_max_concurrency_at_131072\": 1.02}",
  "explicit_KV_cache_budget_bytes": 4929945600,
  "post_KV_allocator_allocated_bytes_by_rank": "{\"0\": 32466124800, \"1\": 32466124800}",
  "post_KV_allocator_reserved_bytes_by_rank": "{\"0\": 32828817408, \"1\": 32828817408}",
  "MTP_accounting_by_tier": {
    "tiers": {
      "strict": {
        "generation_tokens_total": 8452.0,
        "spec_decode_num_accepted_tokens_per_pos_total_0": 2804.0,
        "spec_decode_num_accepted_tokens_per_pos_total_1": 2429.0,
        "spec_decode_num_accepted_tokens_total": 5233.0,
        "spec_decode_num_draft_tokens_total": 6440.0,
        "spec_decode_num_drafts_total": 3220.0,
        "adjusted_unaccepted_proposed_minus_accepted": 1207.0,
        "iteration_histogram_count_delta": 3221.0,
        "iteration_histogram_sum_delta": 8925.0,
        "full_report": "/home/joe/PhoneSync/Documents/gfx906/results/mi50-fp16-tp2-mtp2-9073317-r0-mirror/offline_analysis/c1_128_strict-mtp-counters.json"
      },
      "fixed_2000": {
        "generation_tokens_total": 2000.0,
        "spec_decode_num_accepted_tokens_per_pos_total_0": 653.0,
        "spec_decode_num_accepted_tokens_per_pos_total_1": 466.0,
        "spec_decode_num_accepted_tokens_total": 1119.0,
        "spec_decode_num_draft_tokens_total": 1764.0,
        "spec_decode_num_drafts_total": 882.0,
        "adjusted_unaccepted_proposed_minus_accepted": 645.0,
        "iteration_histogram_count_delta": 883.0,
        "iteration_histogram_sum_delta": 2473.0,
        "full_report": "/home/joe/PhoneSync/Documents/gfx906/results/mi50-fp16-tp2-mtp2-9073317-r0-mirror/offline_analysis/c1_2000-mtp-counters.json"
      },
      "diagnostic_10000": {
        "generation_tokens_total": 10000.0,
        "spec_decode_num_accepted_tokens_per_pos_total_0": 3318.0,
        "spec_decode_num_accepted_tokens_per_pos_total_1": 2924.0,
        "spec_decode_num_accepted_tokens_total": 6242.0,
        "spec_decode_num_draft_tokens_total": 7514.0,
        "spec_decode_num_drafts_total": 3757.0,
        "adjusted_unaccepted_proposed_minus_accepted": 1272.0,
        "iteration_histogram_count_delta": 3758.0,
        "iteration_histogram_sum_delta": 10473.0,
        "full_report": "/home/joe/PhoneSync/Documents/gfx906/results/mi50-fp16-tp2-mtp2-9073317-r0-mirror/offline_analysis/c1_10000-mtp-counters.json"
      }
    },
    "limits": [
      "Returned completion tokens determine fixed-tier length; drafts and accepted proposals are separate counters.",
      "Proposed minus accepted is adjusted unaccepted, not scheduler rollback rejection.",
      "Iteration histogram counts and sums include prefill; they do not give pure decode-step emissions or exact 1/2/3 token chronology.",
      "Warmups are excluded from the three reported tiers; all eleven reports remain in the linked primary."
    ],
    "all_samples_and_histograms_primary": "tuning/qwen38-fp16-matrix/fp16-tp2-mtp-result-20261023.json"
  },
  "MTP_accounting_limit": "Only committed emitted tokens enter TPS. Proposed minus accepted is adjusted unaccepted, not scheduler rollback. Iteration histograms include prefill and do not measure pure decode-step emissions.",
  "source_evidence_commit": "69c6721",
  "comparison_limit": "Explicit original dense FP16/collective fallback at TP2; TP4 packed/posted guards not widened and native DSOs unchanged. | Explicit 4929945600-byte KV budget, context131072 unchanged; actual capacity admission is configuration evidence, not full-context execution. | Actual target/attention KV/conv FP16 and SSM FP32. MTP-only generic convolution fallback (packed convolution0), recurrent1; cache mode none and compile-cache disable, original graph policy. | Original greedy concurrency1, non-eager O3/maxseq2/maxbatched4 requests and scorer; eight warmups and three tracked startup requests excluded. | One official sample, no repeatability/causal gain or TP4 qualification. | Logical parameter/buffer bytes are not unique physical storage. Cache view bytes may alias; no summed physical cache claim. | Loaded route metadata/generated source/maps are not a per-kernel execution trace. | Outer metrics do not establish internal scorer scrape alignment/full settlement trajectory. | KV tokens/concurrency are logged hybrid-cache reports; do not equate177x800 with usable tokens or infer actual131072 execution. | Final kernel journal filter covers assigned four-card PCI scope, broader than TP2 before/after scope. | Read-only retained cache preserves regular files; socket endpoints inventoried, not byte-hashed. | Same-checkpoint native FP16 MTP head, shared embedding/head; all prior failed/historical attempts preserved. | Host exclusivity is unproven; runtime/cache/compiler differences prevent attributing differences solely to MTP. | Proposed minus accepted is adjusted unaccepted, not scheduler rollback rejection; aggregate iteration histograms are not per-step decode emission distributions. Full labeled before/after and histogram reports remain pinned outside hashed source. | One official sample, no repeatability, causal, promotion or TP4 qualification claim. | Original dense TP2 projection/generic collective fallback and generic convolution MTP route are composition differences. | Histogram mean includes prefill/engine output iterations and text clipping boundaries; not pure decode-step emission. | P-A is adjusted unaccepted proposals, not scheduler rollback rejections. | No per-token kernel trace or actual131072-token request. | Outer scrapes do not expose scorer internal scrapes/full settlement trajectory. | Before/after kernels cover selected PCI06/0a; final covers four assigned MI50 cards; selected intersection unchanged. | Cache same-filesystem rename/read-only; socket endpoint state not byte-hashed.",
  "recorded_configuration_details": {
    "cell": "FP16_TP2_MTP2",
    "status": "completed",
    "TP": 2,
    "native_MTP_tokens": 2,
    "host": "10.0.0.50",
    "strict_pass": true,
    "strict_tokens": 8452,
    "strict_diagnostic_TPS": 17.703028402850663,
    "official_2K_TPS": 18.15566891881604,
    "official_2K_tokens": 2000,
    "official_2K_prefill_seconds": 24.74742527602939,
    "official_2K_total_seconds": 134.91621195804328,
    "diagnostic_10K_TPS": 17.194536494951368,
    "diagnostic_10K_tokens": 10000,
    "diagnostic_10K_prefill_seconds": 24.714972343004774,
    "diagnostic_10K_total_seconds": 606.308061289019,
    "SSM_dtype": "torch.float32",
    "source": "90733170b8b007fca3401b68c04e5bc67e14a3e0",
    "image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
    "primary": "tuning/qwen38-fp16-matrix/fp16-tp2-mtp-result-20261023.json",
    "strict_prefill_seconds": 18.314323675003834,
    "strict_total_seconds": 495.76133046799805,
    "per_rank_memory_JSON": [
      {
        "rank": 0,
        "pid": 216,
        "parameter_count": 611,
        "parameter_logical_bytes": 26896673280,
        "parameter_dtypes": [
          "torch.float16"
        ],
        "packed_counts": {
          "_gfx906_fixed3_gate_primary": 0,
          "_gfx906_fixed3_primary": 0,
          "_gfx906_gdn_fixed3_primary": 0,
          "_gfx906_output_fixed3_primary": 0,
          "_gfx906_qkv_fixed3_primary": 0
        },
        "packed_logical_bytes": 0,
        "after_load_allocated_bytes": 27531406336,
        "after_load_reserved_bytes": 30203183104,
        "after_cache_allocated_bytes": 32466124800,
        "after_cache_reserved_bytes": 32828817408,
        "installed_manifest_sha256": "44ea477cf0ae8d0a6da890f35540f66d477c93913eb7d728b45ae58aacb57df0",
        "attention_layers": 16,
        "GDN_layers": 48
      },
      {
        "rank": 1,
        "pid": 217,
        "parameter_count": 611,
        "parameter_logical_bytes": 26896673280,
        "parameter_dtypes": [
          "torch.float16"
        ],
        "packed_counts": {
          "_gfx906_fixed3_gate_primary": 0,
          "_gfx906_fixed3_primary": 0,
          "_gfx906_gdn_fixed3_primary": 0,
          "_gfx906_output_fixed3_primary": 0,
          "_gfx906_qkv_fixed3_primary": 0
        },
        "packed_logical_bytes": 0,
        "after_load_allocated_bytes": 27531406336,
        "after_load_reserved_bytes": 30203183104,
        "after_cache_allocated_bytes": 32466124800,
        "after_cache_reserved_bytes": 32828817408,
        "installed_manifest_sha256": "44ea477cf0ae8d0a6da890f35540f66d477c93913eb7d728b45ae58aacb57df0",
        "attention_layers": 16,
        "GDN_layers": 48
      }
    ],
    "KV_JSON": {
      "attention": {
        "shape": [
          177,
          2,
          800,
          2,
          256
        ],
        "dtype": "torch.float16",
        "bytes": 289996800
      },
      "conv": {
        "shape": [
          177,
          5,
          5120
        ],
        "dtype": "torch.float16",
        "bytes": 9062400
      },
      "SSM": {
        "shape": [
          177,
          24,
          128,
          128
        ],
        "dtype": "torch.float32",
        "bytes": 278396928
      },
      "blocks": 177,
      "block_size": 800,
      "logged_KV_tokens": 35200,
      "logged_max_concurrency_at_131072": 1.02
    },
    "VRAM_JSON": {
      "before": "renderD129 33825607680\nrenderD130 33825673216",
      "after": "renderD129 33825607680\nrenderD130 33825673216"
    },
    "attention_KV_dtype": "torch.float16",
    "context": 131072,
    "speculative_accounting_JSON": {
      "tiers": {
        "strict": {
          "generation_tokens_total": 8452.0,
          "spec_decode_num_accepted_tokens_per_pos_total_0": 2804.0,
          "spec_decode_num_accepted_tokens_per_pos_total_1": 2429.0,
          "spec_decode_num_accepted_tokens_total": 5233.0,
          "spec_decode_num_draft_tokens_total": 6440.0,
          "spec_decode_num_drafts_total": 3220.0,
          "adjusted_unaccepted_proposed_minus_accepted": 1207.0,
          "iteration_histogram_count_delta": 3221.0,
          "iteration_histogram_sum_delta": 8925.0,
          "full_report": "/home/joe/PhoneSync/Documents/gfx906/results/mi50-fp16-tp2-mtp2-9073317-r0-mirror/offline_analysis/c1_128_strict-mtp-counters.json"
        },
        "fixed_2000": {
          "generation_tokens_total": 2000.0,
          "spec_decode_num_accepted_tokens_per_pos_total_0": 653.0,
          "spec_decode_num_accepted_tokens_per_pos_total_1": 466.0,
          "spec_decode_num_accepted_tokens_total": 1119.0,
          "spec_decode_num_draft_tokens_total": 1764.0,
          "spec_decode_num_drafts_total": 882.0,
          "adjusted_unaccepted_proposed_minus_accepted": 645.0,
          "iteration_histogram_count_delta": 883.0,
          "iteration_histogram_sum_delta": 2473.0,
          "full_report": "/home/joe/PhoneSync/Documents/gfx906/results/mi50-fp16-tp2-mtp2-9073317-r0-mirror/offline_analysis/c1_2000-mtp-counters.json"
        },
        "diagnostic_10000": {
          "generation_tokens_total": 10000.0,
          "spec_decode_num_accepted_tokens_per_pos_total_0": 3318.0,
          "spec_decode_num_accepted_tokens_per_pos_total_1": 2924.0,
          "spec_decode_num_accepted_tokens_total": 6242.0,
          "spec_decode_num_draft_tokens_total": 7514.0,
          "spec_decode_num_drafts_total": 3757.0,
          "adjusted_unaccepted_proposed_minus_accepted": 1272.0,
          "iteration_histogram_count_delta": 3758.0,
          "iteration_histogram_sum_delta": 10473.0,
          "full_report": "/home/joe/PhoneSync/Documents/gfx906/results/mi50-fp16-tp2-mtp2-9073317-r0-mirror/offline_analysis/c1_10000-mtp-counters.json"
        }
      },
      "limits": [
        "Returned completion tokens determine fixed-tier length; drafts and accepted proposals are separate counters.",
        "Proposed minus accepted is adjusted unaccepted, not scheduler rollback rejection.",
        "Iteration histogram counts and sums include prefill; they do not give pure decode-step emissions or exact 1/2/3 token chronology.",
        "Warmups are excluded from the three reported tiers; all eleven reports remain in the linked primary."
      ],
      "all_samples_and_histograms_primary": "tuning/qwen38-fp16-matrix/fp16-tp2-mtp-result-20261023.json"
    }
  },
  "MTP_2K_draft_batches": 882.0,
  "MTP_2K_proposed": 1764.0,
  "MTP_2K_accepted": 1119.0,
  "MTP_2K_acceptance_percent": 63.435374149659864,
  "MTP_2K_committed": 2000.0,
  "detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/fp16-matrix.csv",
  "detail_source_sha256": "8643d90cf1f211a7f77be9f6212cd7706e6dbae3ae84b9bc80b5f42fa199ffe0",
  "detail_source_pointer": "FP16_TP2_MTP2",
  "retest_strict_tokens": 9193,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 5591.0,
        "vllm:spec_decode_num_draft_tokens_total": 7204.0,
        "vllm:generation_tokens_total": 9193.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 5591.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:request_generation_tokens_sum": 9193.0,
        "vllm:spec_decode_num_drafts_total": 3602.0
      },
      "accepted_per_position": {
        "0": 3054.0,
        "1": 2537.0
      },
      "accepted_fraction": 0.7760966129927818,
      "committed_emitted_tokens": 9193.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "27fa225cc8c4a6d7f8827719f2bd337459e89a5177a1bcf2cc19615df8298915",
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "88bc7a5f4e148bcaad2371bb058820ccffa01b02023c18216c6e58318503e5d5"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1111.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1111.0,
        "vllm:spec_decode_num_drafts_total": 890.0,
        "vllm:spec_decode_num_draft_tokens_total": 1780.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:generation_tokens_total": 2000.0
      },
      "accepted_per_position": {
        "0": 663.0,
        "1": 448.0
      },
      "accepted_fraction": 0.6241573033707866,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "88bc7a5f4e148bcaad2371bb058820ccffa01b02023c18216c6e58318503e5d5",
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "0a2b24d4501925f978e6ed227d810b69d2b565bbb4b2ac956c9270723cebf9ce"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:generation_tokens_total": 10000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6100.0,
        "vllm:spec_decode_num_drafts_total": 3900.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6100.0,
        "vllm:spec_decode_num_draft_tokens_total": 7800.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:request_generation_tokens_count": 1.0
      },
      "accepted_per_position": {
        "0": 3325.0,
        "1": 2775.0
      },
      "accepted_fraction": 0.782051282051282,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "9709d01df025fec6ae6aeaee1653bcd74d5bd21bfe14c67de92a1daa08723393",
        "/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "106866b7537531838ae81399415e724a034b478609f3d59bf2ed18afd6dbc442"
      }
    }
  },
  "retest_MTP_2K_accepted": 1111.0,
  "retest_MTP_2K_proposed": 1780.0,
  "retest_MTP_2K_draft_batches": 890.0,
  "retest_MTP_2K_acceptance_percent": 62.41573033707866,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 34144878592,
    "1": 34144944128
  },
  "retest_memory_observation_count": 51,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-fp16-tp2-mtp2/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": null,
  "retest_explicit_KV_budget_bytes": 4929945600,
  "retest_KV_budget_GiB": 4.59136962890625,
  "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
  "retest_logged_GPU_KV_tokens": 35200,
  "retest_memory_log_report": {
    "profile": "mi50-fp16-tp2-mtp2",
    "image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
    "container": "01770f22e65cc4feb5e7ba102d94ef9688bb226605fc07e4cce66124eb50ca8a",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 28.13,
    "retest_logged_model_load_seconds": 17.482613,
    "retest_logged_available_KV_GiB": null,
    "retest_explicit_KV_budget_bytes": 4929945600,
    "retest_KV_budget_GiB": 4.59136962890625,
    "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
    "retest_logged_GPU_KV_tokens": 35200,
    "retest_logged_context_capacity_x": 1.02,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-fp16-tp2-mtp2/service-final.log",
    "source_log_sha256": "c9b1261d2fad38310f2950a7f12730e059bee01e33ed9344f95799be1edcf54f",
    "sealed_index_sha256": "c66a8e5ffb498b9534473eb9e911f1479f90a4415fd1b1f2266c5f3f84e3e2f8",
    "source_lines": {
      "model_load": {
        "line": 148,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 22:40:54 [gpu_model_runner.py:4584] Model loading took 28.13 GiB memory and 17.482613 seconds"
      },
      "explicit_KV_budget": {
        "line": 7,
        "text": "(APIServer pid=1) INFO 10-26 22:38:57 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM']}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.fp16_matrix_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 4929945600, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 2}, 'optimization_level': '3'}"
      },
      "explicit_budget_worker_reports": [
        {
          "line": 160,
          "text": "(Worker pid=217) (Worker_TP1 pid=217) INFO 10-26 22:41:46 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.59 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        },
        {
          "line": 161,
          "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 22:41:46 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.59 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        }
      ],
      "grouped_KV_tokens": {
        "line": 163,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 22:41:46 [kv_cache_utils.py:1314] GPU KV cache size: 35,200 tokens"
      },
      "context_capacity": {
        "line": 164,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 22:41:46 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.02x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-fp16-tp2-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 31.799957275390625,
  "sampled_peak_VRAM_GiB": 31.502613067626953
}
MI50Current Qwen3.88-bit RTNQwen/Qwen3.8-27BTP4MTP off PASS14098 tokens 40.86139.515 15.177 / 64.13710K: 16.029 / 269.109Strict: 13.244 / 376.013 5.80 / 25.35packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:a506e684a1f3142d071857d89f8fee8f17f0a30e92970c76d163c03380004961
Runtime image config: sha256:fe0d81ba97b6b8a432db928d19d03b511f15de9e40e5d25be81797d9c3e1694b
Registry verificationDerived host-portability image; recorded benchmark parent: sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a
Exact compatibility change
Replaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately.
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp4-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-w8-tp4-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \
  --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \
  -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \
  -e MI50_RTN_TP=4 \
  -e HIP_VISIBLE_DEVICES=0,1,2,3 \
  -e HF_HUB_OFFLINE=1 \
  -e MI50_RTN_BITS=8 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp0/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp0/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp0,dst=/runwork \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp4-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-mi50-r3.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 41.516 · eligible10K 40.124 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 9254,
    "content_chars": 17704,
    "content_has_think_close": false,
    "elapsed_seconds": 239.1448707189411,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T16:30:45Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.008517238311469555,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 17704,
    "qwen_answer_text_sha256": "2136309f6e466a777e2cc34c8bf82f4d72be9c4cefff367e37709495d31b0a1f",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 17704,
    "qwen_visible_text_sha256": "2136309f6e466a777e2cc34c8bf82f4d72be9c4cefff367e37709495d31b0a1f",
    "raw_has_think_close": false,
    "reasoning_chars": 27545,
    "text_chars": 45250,
    "text_sha256": "fb06c0d2793348020838c839569222efaa9132372c78d6fcdcc657af1da0e551",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 229.69770350586623,
    "vllm_decode_tps": 40.28773409031346,
    "vllm_generation_tokens_delta": 9254.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 9.437211033888161,
    "vllm_prefill_tps": 50.12073994122843,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 38.696209423934974
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 57.735280628316104,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T16:31:43Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.00803779810667038,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 6532,
    "text_chars": 6532,
    "text_sha256": "ef46298b33bf1677f80fde93f2648cae9345517992f93fc8d2b1d90d68cf392e",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 48.173785145394504,
    "vllm_decode_tps": 41.516355710138825,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 9.553459473885596,
    "vllm_prefill_tps": 49.5108605728581,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 34.64086392643436
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 21707,
    "content_has_think_close": false,
    "elapsed_seconds": 258.7847386011854,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T16:36:02Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008156619034707546,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 21707,
    "qwen_answer_text_sha256": "99fb6e9f1c94acc87a5110d7453e700d80ed6ffed621a85fee60a797d181be6e",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 21707,
    "qwen_visible_text_sha256": "99fb6e9f1c94acc87a5110d7453e700d80ed6ffed621a85fee60a797d181be6e",
    "raw_has_think_close": false,
    "reasoning_chars": 27545,
    "text_chars": 49253,
    "text_sha256": "3e84f6a3408e1b88b0a7d7ab7aaf7e6f673e8abb2a6c53a4ae0dd2fd99bcf36d",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 249.2266788072884,
    "vllm_decode_tps": 40.1241153148471,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 9.549467314966023,
    "vllm_prefill_tps": 49.53155860942207,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 38.64215507472817
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS16,128 natural tokens2K 40.83310K 39.500 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T15:30:09.216444552Z",
      "FinishedAt": "2026-10-26T16:04:27.834638844Z"
    },
    "devices_after": [
      {
        "card": 1,
        "render": 129,
        "pci": "0000:06:00.0",
        "actual_card": "card0",
        "busy_percent": 0,
        "vram_bytes": 7933067264,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:0a:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 4747493376,
        "dpm": "auto"
      },
      {
        "card": 3,
        "render": 131,
        "pci": "0000:0d:00.0",
        "actual_card": "card3",
        "busy_percent": 0,
        "vram_bytes": 9868738560,
        "dpm": "auto"
      },
      {
        "card": 4,
        "render": 132,
        "pci": "0000:10:00.0",
        "actual_card": "card4",
        "busy_percent": 0,
        "vram_bytes": 9818406912,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T16:04:27.971854+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 40.83310567872094,
  "retest_10K_TPS": 39.49955384999737,
  "retest_strict_tokens": 16128,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 16128,
      "elapsed_seconds": 439.7262021850329,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 418.7522614010377,
      "vllm_decode_tps": 38.514418873918075,
      "vllm_generation_tokens_delta": 16128.0,
      "vllm_prefill_seconds_delta": 20.95829319197219,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 70.49048449797556,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 48.97986491001211,
      "vllm_decode_tps": 40.83310567872094,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 21.49934186495375,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 275.08431653201114,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 253.16741647198796,
      "vllm_decode_tps": 39.49955384999737,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 21.903418170055375,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 21.49934186495375,
  "retest_2K_total_seconds": 70.49048449797556,
  "retest_10K_prefill_seconds": 21.903418170055375,
  "retest_10K_total_seconds": 275.08431653201114,
  "retest_strict_prefill_seconds": 20.95829319197219,
  "retest_strict_total_seconds": 439.7262021850329,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": "Complete record: chart-details/row-04-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_logged_model_load_delta_GiB": 13.25,
  "retest_logged_model_load_seconds": 64.44897,
  "retest_logged_available_KV_GiB": 15.92,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 15.92,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 260288,
  "retest_logged_context_capacity_x": 7.77,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-w8-tp4-mtp0",
    "image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
    "container": "5a3463ca7a0d4134be09562c677c092b3a8f8952eb1d9d6f0f6a24795e966fc0",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 13.25,
    "retest_logged_model_load_seconds": 64.44897,
    "retest_logged_available_KV_GiB": 15.92,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 15.92,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 260288,
    "retest_logged_context_capacity_x": 7.77,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w8-tp4-mtp0/service-final.log",
    "source_log_sha256": "00785c973ab4b2f5244959832bc6928c3b8483fd02cedd719bad4475c078b8b4",
    "sealed_index_sha256": "6bf9c86f222b1baf9d1e5e4e47cf58baa38ee70a9f0b2006431655307d80db75",
    "source_lines": {
      "model_load": {
        "line": 144,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 15:33:02 [gpu_model_runner.py:4584] Model loading took 13.25 GiB memory and 64.448970 seconds"
      },
      "automatic_KV_budget": {
        "line": 166,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 15:34:08 [gpu_worker.py:453] Available KV cache memory: 15.92 GiB"
      },
      "grouped_KV_tokens": {
        "line": 171,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 15:34:08 [kv_cache_utils.py:1314] GPU KV cache size: 260,288 tokens"
      },
      "context_capacity": {
        "line": 172,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 15:34:08 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 7.77x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w8-tp4-mtp0/memory-log-report.json"
}
Complete retest loaded admission
Retest KV budget: 15.920 GiB (automatic, rounded)260,288 grouped KV tokens · 7.77× reported capacityRetest memory reports and limits
Evidence and limits

completed matrix cell

{
  "strict_tokens": 14098,
  "source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w8-tp4-mtpoff",
  "source_revision": "d72edcb54835d75267f4e852d0290847824c584e",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "sampled_VRAM_after_ladder_bytes_by_rank": "[27206279168,27223011328,27222994944,27172745216]",
  "comparison_limit": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
  "recorded_configuration_details": {
    "cell": "w8-tp4-mtpoff",
    "bits": 8,
    "group_size": 128,
    "TP": 4,
    "native_MTP": false,
    "num_speculative_tokens": 0,
    "strict_valid": true,
    "official_2k_eligible": true,
    "strict_returned_tokens": 14098,
    "strict_decode_TPS": 38.863979519826344,
    "strict_prefill_seconds": 13.244176653999602,
    "strict_total_seconds": 376.0125174979912,
    "fixed_2000_returned_tokens": 2000,
    "fixed_2000_decode_TPS": 40.860679847930975,
    "fixed_2000_prefill_seconds": 15.176513558981242,
    "fixed_2000_total_seconds": 64.13704536799924,
    "fixed_10000_returned_tokens": 10000,
    "fixed_10000_decode_TPS": 39.51520953183532,
    "fixed_10000_prefill_seconds": 16.028607391985133,
    "fixed_10000_total_seconds": 269.10898952701245,
    "packed_weight_bytes_per_rank": 6230318080,
    "packed_weight_bytes_total": 24921272320,
    "packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
    "render_nodes_by_rank": [
      "renderD129",
      "renderD130",
      "renderD131",
      "renderD132"
    ],
    "sampled_VRAM_before_ladder_bytes_by_rank": [
      27206131712,
      27222863872,
      27222847488,
      27172597760
    ],
    "sampled_VRAM_after_ladder_bytes_by_rank": [
      27206279168,
      27223011328,
      27222994944,
      27172745216
    ],
    "allocated_after_load_bytes_by_rank": [
      7698443776,
      7698443776,
      7698443776,
      7698443776
    ],
    "allocated_after_cache_bytes_by_rank": [
      24779252224,
      24779252224,
      24779252224,
      24779252224
    ],
    "reserved_after_cache_bytes_by_rank": [
      25272778752,
      25272778752,
      25272778752,
      25272778752
    ],
    "sampled_attention_KV_first_layer_views_by_rank": [
      {
        "rank": 0,
        "views": [
          {
            "shape": [
              1329,
              2,
              784,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 1066942464
          }
        ]
      },
      {
        "rank": 1,
        "views": [
          {
            "shape": [
              1329,
              2,
              784,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 1066942464
          }
        ]
      },
      {
        "rank": 2,
        "views": [
          {
            "shape": [
              1329,
              2,
              784,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 1066942464
          }
        ]
      },
      {
        "rank": 3,
        "views": [
          {
            "shape": [
              1329,
              2,
              784,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 1066942464
          }
        ]
      }
    ],
    "KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
    "KV_budget": "automatic; actual allocation receipts retained",
    "context": 131072,
    "greedy": true,
    "concurrency": 1,
    "activation_dtype": "FP16",
    "attention_KV_dtype": "FP16",
    "SSM_dtype": "float32",
    "native_MTP_head_dtype": "not used",
    "native_MTP_unshared_parameter_bytes_by_rank": [],
    "model_repository": "Qwen/Qwen3.8-27B",
    "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "serving_source": "d72edcb54835d75267f4e852d0290847824c584e",
    "image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
    "native_MTP_model_source_sha256": "not used",
    "MTP_request_accounting_by_tier": {},
    "MTP_accounting_scope": "not used",
    "runtime_difference": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
    "MTP_scheduler_rollback_rejection_count": "not used",
    "host_exclusivity": "unproven",
    "serving_CPUs": "0-7,16-23",
    "cache_placement": "fresh private admitted rootFS",
    "compile_cache_disabled": true,
    "official_2k_samples": 1,
    "repeatability_established": false,
    "FP16_goal_qualifying_ladders": 0,
    "evidence": "../../tuning/qwen38-rtn-mi50/w8-tp4-result-20261022.json"
  },
  "detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
  "detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
  "detail_source_pointer": "w8-tp4-mtpoff",
  "retest_strict_tokens": 16128,
  "retest_MTP_accounting_by_tier": {},
  "retest_logged_available_KV_GiB": 15.92,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 15.92,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 260288,
  "retest_memory_log_report": {
    "profile": "mi50-w8-tp4-mtp0",
    "image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
    "container": "5a3463ca7a0d4134be09562c677c092b3a8f8952eb1d9d6f0f6a24795e966fc0",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 13.25,
    "retest_logged_model_load_seconds": 64.44897,
    "retest_logged_available_KV_GiB": 15.92,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 15.92,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 260288,
    "retest_logged_context_capacity_x": 7.77,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w8-tp4-mtp0/service-final.log",
    "source_log_sha256": "00785c973ab4b2f5244959832bc6928c3b8483fd02cedd719bad4475c078b8b4",
    "sealed_index_sha256": "6bf9c86f222b1baf9d1e5e4e47cf58baa38ee70a9f0b2006431655307d80db75",
    "source_lines": {
      "model_load": {
        "line": 144,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 15:33:02 [gpu_model_runner.py:4584] Model loading took 13.25 GiB memory and 64.448970 seconds"
      },
      "automatic_KV_budget": {
        "line": 166,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 15:34:08 [gpu_worker.py:453] Available KV cache memory: 15.92 GiB"
      },
      "grouped_KV_tokens": {
        "line": 171,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 15:34:08 [kv_cache_utils.py:1314] GPU KV cache size: 260,288 tokens"
      },
      "context_capacity": {
        "line": 172,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 15:34:08 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 7.77x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w8-tp4-mtp0/memory-log-report.json",
  "sampled_peak_VRAM_GiB": 25.353404998779297
}
MI50Current Qwen3.88-bit RTNQwen/Qwen3.8-27BTP2MTP off PASS7711 tokens 24.02223.177 17.135 / 100.40610K: 17.255 / 448.730Strict: 17.178 / 345.797 11.60 / 22.82packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:a506e684a1f3142d071857d89f8fee8f17f0a30e92970c76d163c03380004961
Runtime image config: sha256:fe0d81ba97b6b8a432db928d19d03b511f15de9e40e5d25be81797d9c3e1694b
Registry verificationDerived host-portability image; recorded benchmark parent: sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a
Exact compatibility change
Replaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately.
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp2-mtp0
GPU_RENDER_NODES=renderD128,renderD129

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-w8-tp2-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --user root \
  --workdir /tmp \
  -e HF_HUB_OFFLINE=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \
  -e MI50_RTN_BITS=8 \
  -e MI50_RTN_TP=2 \
  -e HIP_VISIBLE_DEVICES=0,1 \
  -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \
  -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp0/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp0,dst=/runwork \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp0/cache/.triton,dst=/root/.triton \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp2-mtp0
GPU_RENDER_NODES=renderD128,renderD129

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-mi50-r3.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 23.958 · eligible10K 23.139 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 7319,
    "content_chars": 16222,
    "content_has_think_close": false,
    "elapsed_seconds": 329.2658855440095,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T17:16:41Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.008773885667324066,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 16222,
    "qwen_answer_text_sha256": "a7be1ad298a7592e1101ba37a866b77cbefeeb356eeaa898e32541ef83392157",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 16222,
    "qwen_visible_text_sha256": "a7be1ad298a7592e1101ba37a866b77cbefeeb356eeaa898e32541ef83392157",
    "raw_has_think_close": false,
    "reasoning_chars": 22466,
    "text_chars": 38689,
    "text_sha256": "bc50a4ac34f0594c65f428bf83f691971fdfc2cbf8b1dfd3bf586d539eb1b6c9",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 312.1824915353209,
    "vllm_decode_tps": 23.444620369339052,
    "vllm_generation_tokens_delta": 7319.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 17.07360925897956,
    "vllm_prefill_tps": 27.703574143307403,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 22.228236575154536
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 100.557428477332,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T17:18:22Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.00600853469222784,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 10572,
    "text_chars": 10572,
    "text_sha256": "15ddcc3add7dbf167ef060d6e3340b39995f49d194e0d06e516e1ea573caee41",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 83.47772400546819,
    "vllm_decode_tps": 23.958487414786134,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 17.07128318399191,
    "vllm_prefill_tps": 27.70734893810102,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 19.889132312595354
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 31057,
    "content_has_think_close": false,
    "elapsed_seconds": 449.2523580007255,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T17:25:51Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.006314177066087723,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 31057,
    "qwen_answer_text_sha256": "218046609c9b388c7a955b81e313225e4f1c3821840a1bdf99771f1e43421044",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 31057,
    "qwen_visible_text_sha256": "218046609c9b388c7a955b81e313225e4f1c3821840a1bdf99771f1e43421044",
    "raw_has_think_close": false,
    "reasoning_chars": 22466,
    "text_chars": 53524,
    "text_sha256": "711f175195df588992b8ece3772dcee43db2f47e50fd5c6d8a23ff24fab7c1e3",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 432.1707717254758,
    "vllm_decode_tps": 23.139001187133072,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 17.072354597970843,
    "vllm_prefill_tps": 27.705610101152597,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 22.25920425771889
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS13,641 natural tokens2K 24.06710K 23.240 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T16:57:55.889934026Z",
      "FinishedAt": "2026-10-26T17:44:23.886246589Z"
    },
    "devices_after": [
      {
        "card": 1,
        "render": 129,
        "pci": "0000:06:00.0",
        "actual_card": "card0",
        "busy_percent": 0,
        "vram_bytes": 10047279104,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:0a:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 8687751168,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T17:44:24.026755+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 24.06705207684622,
  "retest_10K_TPS": 23.240077979944296,
  "retest_strict_tokens": 13641,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 13641,
      "elapsed_seconds": 614.2203862669412,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 596.9754239380127,
      "vllm_decode_tps": 22.85018688041741,
      "vllm_generation_tokens_delta": 13641.0,
      "vllm_prefill_seconds_delta": 17.231190228951164,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 100.30374919692986,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 83.10116227006074,
      "vllm_decode_tps": 24.06705207684622,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 17.19083948188927,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 447.68319084704854,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 430.2911551600555,
      "vllm_decode_tps": 23.240077979944296,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 17.377528968965635,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 17.19083948188927,
  "retest_2K_total_seconds": 100.30374919692986,
  "retest_10K_prefill_seconds": 17.377528968965635,
  "retest_10K_total_seconds": 447.68319084704854,
  "retest_strict_prefill_seconds": 17.231190228951164,
  "retest_strict_total_seconds": 614.2203862669412,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": "Complete record: chart-details/row-05-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 29245403136,
    "1": 29245374464
  },
  "retest_memory_observation_count": 47,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w8-tp2-mtp0/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 25.72,
  "retest_logged_model_load_seconds": 117.649267,
  "retest_logged_available_KV_GiB": null,
  "retest_explicit_KV_budget_bytes": 6442450944,
  "retest_KV_budget_GiB": 6.0,
  "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
  "retest_logged_GPU_KV_tokens": 48608,
  "retest_logged_context_capacity_x": 1.46,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-w8-tp2-mtp0",
    "image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
    "container": "08a36b4fd94cf9307b1d837bee7665ffdd40b6e7a7cca6c1a12120691a552194",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 25.72,
    "retest_logged_model_load_seconds": 117.649267,
    "retest_logged_available_KV_GiB": null,
    "retest_explicit_KV_budget_bytes": 6442450944,
    "retest_KV_budget_GiB": 6.0,
    "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
    "retest_logged_GPU_KV_tokens": 48608,
    "retest_logged_context_capacity_x": 1.46,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w8-tp2-mtp0/service-final.log",
    "source_log_sha256": "6383592438234ae67e293c662569aaced83d7ca748bd67ddeed7c4e75341295f",
    "sealed_index_sha256": "4cc128355232a7a0e150440bc784203cf03d7b7562ba35660c0b5577679dc4cc",
    "source_lines": {
      "model_load": {
        "line": 118,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:01:41 [gpu_model_runner.py:4584] Model loading took 25.72 GiB memory and 117.649267 seconds"
      },
      "explicit_KV_budget": {
        "line": 7,
        "text": "(APIServer pid=1) INFO 10-26 16:58:27 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 8, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'optimization_level': '3'}"
      },
      "explicit_budget_worker_reports": [
        {
          "line": 125,
          "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:02:26 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        },
        {
          "line": 126,
          "text": "(Worker pid=197) (Worker_TP1 pid=197) INFO 10-26 17:02:26 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        }
      ],
      "grouped_KV_tokens": {
        "line": 127,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 17:02:26 [kv_cache_utils.py:1314] GPU KV cache size: 48,608 tokens"
      },
      "context_capacity": {
        "line": 128,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 17:02:26 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.46x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w8-tp2-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 27.23690414428711
}
Complete retest loaded admission
Retest KV budget: 6.000 GiB (explicit)48,608 grouped KV tokens · 1.46× reported capacityRetest sampled VRAM: 27.237 GiB max/cardRetest memory reports and limits
Evidence and limits

completed matrix cell

{
  "strict_tokens": 7711,
  "source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w8-tp2-mtpoff",
  "source_revision": "d72edcb54835d75267f4e852d0290847824c584e",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "sampled_VRAM_after_ladder_bytes_by_rank": "[24503320576,24482340864]",
  "comparison_limit": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
  "recorded_configuration_details": {
    "cell": "w8-tp2-mtpoff",
    "bits": 8,
    "group_size": 128,
    "TP": 2,
    "native_MTP": false,
    "num_speculative_tokens": 0,
    "strict_valid": true,
    "official_2k_eligible": true,
    "strict_returned_tokens": 7711,
    "strict_decode_TPS": 23.46576383615205,
    "strict_prefill_seconds": 17.17835429200204,
    "strict_total_seconds": 345.7971052789944,
    "fixed_2000_returned_tokens": 2000,
    "fixed_2000_decode_TPS": 24.02199034482997,
    "fixed_2000_prefill_seconds": 17.13484327500919,
    "fixed_2000_total_seconds": 100.40560724699753,
    "fixed_10000_returned_tokens": 10000,
    "fixed_10000_decode_TPS": 23.17706206378481,
    "fixed_10000_prefill_seconds": 17.25501243400504,
    "fixed_10000_total_seconds": 448.7302281940065,
    "packed_weight_bytes_per_rank": 12460636160,
    "packed_weight_bytes_total": 24921272320,
    "packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
    "render_nodes_by_rank": [
      "renderD129",
      "renderD130"
    ],
    "sampled_VRAM_before_ladder_bytes_by_rank": [
      24503320576,
      24482336768
    ],
    "sampled_VRAM_after_ladder_bytes_by_rank": [
      24503320576,
      24482340864
    ],
    "allocated_after_load_bytes_by_rank": [
      15213055488,
      15213055488
    ],
    "allocated_after_cache_bytes_by_rank": [
      21639049728,
      21639049728
    ],
    "reserved_after_cache_bytes_by_rank": [
      22716350464,
      22716350464
    ],
    "sampled_attention_KV_first_layer_views_by_rank": [
      {
        "rank": 0,
        "views": [
          {
            "shape": [
              250,
              2,
              784,
              2,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 401408000
          }
        ]
      },
      {
        "rank": 1,
        "views": [
          {
            "shape": [
              250,
              2,
              784,
              2,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 401408000
          }
        ]
      }
    ],
    "KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
    "KV_budget": "6442450944 bytes explicit",
    "context": 131072,
    "greedy": true,
    "concurrency": 1,
    "activation_dtype": "FP16",
    "attention_KV_dtype": "FP16",
    "SSM_dtype": "float32",
    "native_MTP_head_dtype": "not used",
    "native_MTP_unshared_parameter_bytes_by_rank": [],
    "model_repository": "Qwen/Qwen3.8-27B",
    "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "serving_source": "d72edcb54835d75267f4e852d0290847824c584e",
    "image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
    "native_MTP_model_source_sha256": "not used",
    "MTP_request_accounting_by_tier": {},
    "MTP_accounting_scope": "not used",
    "runtime_difference": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
    "MTP_scheduler_rollback_rejection_count": "not used",
    "host_exclusivity": "unproven",
    "serving_CPUs": "0-7,16-23",
    "cache_placement": "fresh private admitted rootFS",
    "compile_cache_disabled": true,
    "official_2k_samples": 1,
    "repeatability_established": false,
    "FP16_goal_qualifying_ladders": 0,
    "evidence": "../../tuning/qwen38-rtn-mi50/w8-tp2-result-20261022.json"
  },
  "detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
  "detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
  "detail_source_pointer": "w8-tp2-mtpoff",
  "retest_strict_tokens": 13641,
  "retest_MTP_accounting_by_tier": {},
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 29245403136,
    "1": 29245374464
  },
  "retest_memory_observation_count": 47,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w8-tp2-mtp0/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": null,
  "retest_explicit_KV_budget_bytes": 6442450944,
  "retest_KV_budget_GiB": 6.0,
  "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
  "retest_logged_GPU_KV_tokens": 48608,
  "retest_memory_log_report": {
    "profile": "mi50-w8-tp2-mtp0",
    "image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
    "container": "08a36b4fd94cf9307b1d837bee7665ffdd40b6e7a7cca6c1a12120691a552194",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 25.72,
    "retest_logged_model_load_seconds": 117.649267,
    "retest_logged_available_KV_GiB": null,
    "retest_explicit_KV_budget_bytes": 6442450944,
    "retest_KV_budget_GiB": 6.0,
    "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
    "retest_logged_GPU_KV_tokens": 48608,
    "retest_logged_context_capacity_x": 1.46,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w8-tp2-mtp0/service-final.log",
    "source_log_sha256": "6383592438234ae67e293c662569aaced83d7ca748bd67ddeed7c4e75341295f",
    "sealed_index_sha256": "4cc128355232a7a0e150440bc784203cf03d7b7562ba35660c0b5577679dc4cc",
    "source_lines": {
      "model_load": {
        "line": 118,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:01:41 [gpu_model_runner.py:4584] Model loading took 25.72 GiB memory and 117.649267 seconds"
      },
      "explicit_KV_budget": {
        "line": 7,
        "text": "(APIServer pid=1) INFO 10-26 16:58:27 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 8, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'optimization_level': '3'}"
      },
      "explicit_budget_worker_reports": [
        {
          "line": 125,
          "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:02:26 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        },
        {
          "line": 126,
          "text": "(Worker pid=197) (Worker_TP1 pid=197) INFO 10-26 17:02:26 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        }
      ],
      "grouped_KV_tokens": {
        "line": 127,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 17:02:26 [kv_cache_utils.py:1314] GPU KV cache size: 48,608 tokens"
      },
      "context_capacity": {
        "line": 128,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 17:02:26 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.46x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w8-tp2-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 27.23690414428711,
  "sampled_peak_VRAM_GiB": 22.82049560546875
}
MI50Current Qwen3.88-bit RTNQwen/Qwen3.8-27BTP4MTP22K accepted 1085 / 1832 (59.22%) PASS6410 tokens 30.28025.717 14.502 / 80.56510K: 14.462 / 403.327Strict: 14.460 / 236.118 5.80 / 23.93packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:1ee2a65021b0a54e0f8784ea5e7487504cb48741de9af1bad226b5427c4fa96b
Runtime image config: sha256:b1450af604a4f69af53a3ac83f783d9572fcfafb3705c44d5730d50ba99baaad
Registry verificationDerived host-portability image; recorded benchmark parent: sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7
Exact compatibility change
Replaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately.
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp4-mtp2
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-w8-tp4-mtp2-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \
  --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \
  --user root \
  --workdir /tmp \
  -e MI50_RTN_TP=4 \
  -e HIP_VISIBLE_DEVICES=0,1,2,3 \
  -e HF_HUB_OFFLINE=1 \
  -e MI50_RTN_BITS=8 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \
  -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \
  -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  -e MAX_JOBS=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp2/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp2/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp2/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp2,dst=/runwork \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp4-mtp2
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-mi50-r3.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 34.170 · eligible10K 27.357 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 6977,
    "content_chars": 15337,
    "content_has_think_close": false,
    "elapsed_seconds": 241.16507857106626,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T17:59:50Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.007626298815011978,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 15337,
    "qwen_answer_text_sha256": "3d4173726447a9c1f69d408bd8ce8c8fbcf609a93736ca85a0879ec41ea6dd58",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 15337,
    "qwen_visible_text_sha256": "3d4173726447a9c1f69d408bd8ce8c8fbcf609a93736ca85a0879ec41ea6dd58",
    "raw_has_think_close": false,
    "reasoning_chars": 20610,
    "text_chars": 35948,
    "text_sha256": "ef0d1ca01a68cb4926196ff3a217df69b6d80218a4a3509a6f6eb16ff6a0368d",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 230.64176619797945,
    "vllm_decode_tps": 30.250375354874137,
    "vllm_generation_tokens_delta": 6977.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 10.514229037798941,
    "vllm_prefill_tps": 44.986655540748835,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 28.930390922846755
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 68.95196704845876,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T18:01:00Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.005916460417211056,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 10087,
    "text_chars": 10087,
    "text_sha256": "0e8ac24e709139ba097dd23070a7a19e68a3eb2378b79f656bd3e03ffa3763ae",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 58.531507852487266,
    "vllm_decode_tps": 34.16963056958067,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 10.413011768832803,
    "vllm_prefill_tps": 45.42393790581672,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 29.005698975845316
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 31161,
    "content_has_think_close": false,
    "elapsed_seconds": 376.00577663164586,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T18:07:16Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.007972470484673977,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 31161,
    "qwen_answer_text_sha256": "7f3809ce4035f946acab56de8b0a0f62f14f1542f6c57c680a966286083b7820",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 31161,
    "qwen_visible_text_sha256": "7f3809ce4035f946acab56de8b0a0f62f14f1542f6c57c680a966286083b7820",
    "raw_has_think_close": false,
    "reasoning_chars": 20610,
    "text_chars": 51772,
    "text_sha256": "5efdd6da02c5c6693aab66ce303140d9d7a0ffae7e22abdc30844352272582c9",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 365.5408342368901,
    "vllm_decode_tps": 27.35672478527929,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 10.456269898451865,
    "vllm_prefill_tps": 45.23601672428439,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 26.595336086542368
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completed; strict failed; fixed speeds diagnostic onlyVerification · Raw retestMTP counter replayFAIL5,065 natural tokens2K 33.452*10K 26.146* · diagnostic2K MTP accepted 1178 / 1644 (71.65%)
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T18:58:23.196020003Z",
      "FinishedAt": "2026-10-26T19:33:39.95166382Z"
    },
    "devices_after": [
      {
        "card": 1,
        "render": 129,
        "pci": "0000:06:00.0",
        "actual_card": "card0",
        "busy_percent": 0,
        "vram_bytes": 19930054656,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:0a:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 19457867776,
        "dpm": "auto"
      },
      {
        "card": 3,
        "render": 131,
        "pci": "0000:0d:00.0",
        "actual_card": "card3",
        "busy_percent": 0,
        "vram_bytes": 19892056064,
        "dpm": "auto"
      },
      {
        "card": 4,
        "render": 132,
        "pci": "0000:10:00.0",
        "actual_card": "card4",
        "busy_percent": 0,
        "vram_bytes": 19969708032,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T19:33:40.094119+00:00"
  },
  "retest_strict": "FAIL",
  "retest_official_2K_eligible": false,
  "retest_2K_TPS": 33.45204890335249,
  "retest_10K_TPS": 26.145610878939074,
  "retest_strict_tokens": 5065,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 5065,
      "elapsed_seconds": 198.25056633690838,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 175.8506427659886,
      "vllm_decode_tps": 28.802851785649686,
      "vllm_generation_tokens_delta": 5065.0,
      "vllm_prefill_seconds_delta": 22.387725064065307,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 82.28246686095372,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 59.78707031602971,
      "vllm_decode_tps": 33.45204890335249,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 22.482500477926806,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 404.386289892951,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 382.4733736879425,
      "vllm_decode_tps": 26.145610878939074,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 21.900252138031647,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 22.482500477926806,
  "retest_2K_total_seconds": 82.28246686095372,
  "retest_10K_prefill_seconds": 21.900252138031647,
  "retest_10K_total_seconds": 404.386289892951,
  "retest_strict_prefill_seconds": 22.387725064065307,
  "retest_strict_total_seconds": 198.25056633690838,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 4156.0,
        "vllm:request_generation_tokens_sum": 5065.0,
        "vllm:spec_decode_num_accepted_tokens_total": 2986.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 2986.0,
        "vllm:spec_decode_num_drafts_total": 2078.0,
        "vllm:generation_tokens_total": 5065.0,
        "vllm:request_generation_tokens_count": 1.0
      },
      "accepted_per_position": {
        "0": 1725.0,
        "1": 1261.0
      },
      "accepted_fraction": 0.7184793070259865,
      "committed_emitted_tokens": 5065.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "52626453501dda855f1e312dff99fb8be8ab0168b8ceb2aed5f39894f51dcec0",
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "d08dc97c94d35a272c3296a02796beec082e2f0f769b29ee9dbc79fd8d61a501"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_drafts_total": 822.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1178.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1178.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_draft_tokens_total": 1644.0
      },
      "accepted_per_position": {
        "0": 669.0,
        "1": 509.0
      },
      "accepted_fraction": 0.7165450121654501,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "afedac1311f18a5f412481eecac893a8c4e54b59275f102c389b4eb471af9cd8",
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "91830c16667855b20d0b7897c58007bd337986836036f4de1a6476a12faf324b"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 6275.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6275.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:spec_decode_num_draft_tokens_total": 7452.0,
        "vllm:spec_decode_num_drafts_total": 3726.0
      },
      "accepted_per_position": {
        "0": 3373.0,
        "1": 2902.0
      },
      "accepted_fraction": 0.8420558239398819,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "7f92ce7ac776a0f277eabd37bb200446633e17d526b88b9190bc2ad03f51111a",
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "5016c951dee5e71c622875bcd6b1bd81c125066f0d8a119b07898f086e4ad657"
      }
    }
  },
  "retest_MTP_2K_accepted": 1178.0,
  "retest_MTP_2K_proposed": 1644.0,
  "retest_MTP_2K_draft_batches": 822.0,
  "retest_MTP_2K_acceptance_percent": 71.65450121654501,
  "retest_loaded_admission": "Complete record: chart-details/row-06-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 1, \"fixed\": 1, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
  "retest_diagnostic_counter_audit": "retest-evidence/mi50-w8-tp4-mtp2/local-diagnostic-fixed-audit.json",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 25681870848,
    "1": 25698664448,
    "2": 25698631680,
    "3": 25648300032
  },
  "retest_memory_observation_count": 34,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w8-tp4-mtp2/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 14.43,
  "retest_logged_model_load_seconds": 63.480969,
  "retest_logged_available_KV_GiB": 14.35,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 14.35,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 220800,
  "retest_logged_context_capacity_x": 6.39,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-w8-tp4-mtp2",
    "image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
    "container": "2901da8586316a8b0a053199945a78afd8c877762f777773550bd1a8463cbba4",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 14.43,
    "retest_logged_model_load_seconds": 63.480969,
    "retest_logged_available_KV_GiB": 14.35,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 14.35,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 220800,
    "retest_logged_context_capacity_x": 6.39,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w8-tp4-mtp2/service-final.log",
    "source_log_sha256": "37ad86fe2abc818ffd5877f746e3891fdfc6c948a6eec70e813d4b171a7d0e3d",
    "sealed_index_sha256": "cad06a029821c6b96a4260e7a9e1f3938969ecef85de855a0ab156d226b112e4",
    "source_lines": {
      "model_load": {
        "line": 178,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:01:37 [gpu_model_runner.py:4584] Model loading took 14.43 GiB memory and 63.480969 seconds"
      },
      "automatic_KV_budget": {
        "line": 265,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:06:40 [gpu_worker.py:453] Available KV cache memory: 14.35 GiB"
      },
      "grouped_KV_tokens": {
        "line": 269,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 19:06:40 [kv_cache_utils.py:1314] GPU KV cache size: 220,800 tokens"
      },
      "context_capacity": {
        "line": 270,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 19:06:40 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.39x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w8-tp4-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 23.933746337890625
}
Complete retest loaded admission
Retest KV budget: 14.350 GiB (automatic, rounded)220,800 grouped KV tokens · 6.39× reported capacityRetest sampled VRAM: 23.934 GiB max/cardRetest memory reports and limits
Evidence and limits

completed matrix cell

{
  "strict_tokens": 6410,
  "source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w8-tp4-mtp2",
  "source_revision": "016143b00c7a08e25ff85eb83dc052be148215ab",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "sampled_VRAM_after_ladder_bytes_by_rank": "[25681883136,25698689024,25698684928,25648316416]",
  "comparison_limit": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
  "recorded_configuration_details": {
    "cell": "w8-tp4-mtp2",
    "bits": 8,
    "group_size": 128,
    "TP": 4,
    "native_MTP": true,
    "num_speculative_tokens": 2,
    "strict_valid": true,
    "official_2k_eligible": true,
    "strict_returned_tokens": 6410,
    "strict_decode_TPS": 28.920106957529065,
    "strict_prefill_seconds": 14.459649450000143,
    "strict_total_seconds": 236.11842984199757,
    "fixed_2000_returned_tokens": 2000,
    "fixed_2000_decode_TPS": 30.279763374132372,
    "fixed_2000_prefill_seconds": 14.502473675995134,
    "fixed_2000_total_seconds": 80.56469545399887,
    "fixed_10000_returned_tokens": 10000,
    "fixed_10000_decode_TPS": 25.71673654115124,
    "fixed_10000_prefill_seconds": 14.462411346990848,
    "fixed_10000_total_seconds": 403.3265637109871,
    "packed_weight_bytes_per_rank": 6230318080,
    "packed_weight_bytes_total": 24921272320,
    "packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
    "render_nodes_by_rank": [
      "renderD129",
      "renderD130",
      "renderD131",
      "renderD132"
    ],
    "sampled_VRAM_before_ladder_bytes_by_rank": [
      25681883136,
      25698689024,
      25698684928,
      25648316416
    ],
    "sampled_VRAM_after_ladder_bytes_by_rank": [
      25681883136,
      25698689024,
      25698684928,
      25648316416
    ],
    "allocated_after_load_bytes_by_rank": [
      7913016320,
      7913016320,
      7913016320,
      7913016320
    ],
    "allocated_after_cache_bytes_by_rank": [
      23317684224,
      23317684224,
      23317684224,
      23317684224
    ],
    "reserved_after_cache_bytes_by_rank": [
      23886561280,
      23886561280,
      23886561280,
      23886561280
    ],
    "sampled_attention_KV_first_layer_views_by_rank": [
      {
        "rank": 0,
        "views": [
          {
            "shape": [
              1105,
              2,
              800,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 905216000
          }
        ]
      },
      {
        "rank": 1,
        "views": [
          {
            "shape": [
              1105,
              2,
              800,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 905216000
          }
        ]
      },
      {
        "rank": 2,
        "views": [
          {
            "shape": [
              1105,
              2,
              800,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 905216000
          }
        ]
      },
      {
        "rank": 3,
        "views": [
          {
            "shape": [
              1105,
              2,
              800,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 905216000
          }
        ]
      }
    ],
    "KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
    "KV_budget": "automatic; actual allocation receipts retained",
    "context": 131072,
    "greedy": true,
    "concurrency": 1,
    "activation_dtype": "FP16",
    "attention_KV_dtype": "FP16",
    "SSM_dtype": "float32",
    "native_MTP_head_dtype": "FP16",
    "native_MTP_unshared_parameter_bytes_by_rank": [
      212388864,
      212388864,
      212388864,
      212388864
    ],
    "model_repository": "Qwen/Qwen3.8-27B",
    "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "serving_source": "016143b00c7a08e25ff85eb83dc052be148215ab",
    "image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
    "native_MTP_model_source_sha256": "46f57b9c511c9199aa83421fa7fb167b08d5d80c14377948690cdf67d75453c3",
    "MTP_request_accounting_by_tier": {
      "c1_128_strict": {
        "drafts": 2517.0,
        "proposed": 5034.0,
        "accepted": 3894.0,
        "adjusted_unaccepted": 1140.0,
        "generation": 6410.0,
        "prompt": 473.0,
        "histogram_count": 2518.0,
        "histogram_sum": 6883.0,
        "means": {
          "prompt_plus_generation_per_observed_iteration": 2.733518665607625,
          "generation_per_all_observed_iterations": 2.545671167593328,
          "sum_equals_prompt_plus_generation": true
        }
      },
      "c1_2000": {
        "drafts": 916.0,
        "proposed": 1832.0,
        "accepted": 1085.0,
        "adjusted_unaccepted": 747.0,
        "generation": 2000.0,
        "prompt": 473.0,
        "histogram_count": 917.0,
        "histogram_sum": 2473.0,
        "means": {
          "prompt_plus_generation_per_observed_iteration": 2.6968375136314067,
          "generation_per_all_observed_iterations": 2.1810250817884405,
          "sum_equals_prompt_plus_generation": true
        }
      },
      "c1_10000": {
        "drafts": 3808.0,
        "proposed": 7616.0,
        "accepted": 6191.0,
        "adjusted_unaccepted": 1425.0,
        "generation": 10000.0,
        "prompt": 473.0,
        "histogram_count": 3809.0,
        "histogram_sum": 10473.0,
        "means": {
          "prompt_plus_generation_per_observed_iteration": 2.749540561827251,
          "generation_per_all_observed_iterations": 2.625360987135731,
          "sum_equals_prompt_plus_generation": true
        }
      }
    },
    "MTP_accounting_scope": "P-A is adjusted unaccepted, not scheduler rollback rejection; iteration means include prefill, not exact decode-only emissions.",
    "runtime_difference": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
    "MTP_scheduler_rollback_rejection_count": "unobservable",
    "host_exclusivity": "unproven",
    "serving_CPUs": "0-7,16-23",
    "cache_placement": "fresh private admitted rootFS",
    "compile_cache_disabled": true,
    "official_2k_samples": 1,
    "repeatability_established": false,
    "FP16_goal_qualifying_ladders": 0,
    "evidence": "../../tuning/qwen38-rtn-mi50/w8-tp4-mtp-result-20261022.json"
  },
  "MTP_accounting_by_tier": {
    "c1_128_strict": {
      "drafts": 2517.0,
      "proposed": 5034.0,
      "accepted": 3894.0,
      "adjusted_unaccepted": 1140.0,
      "generation": 6410.0,
      "prompt": 473.0,
      "histogram_count": 2518.0,
      "histogram_sum": 6883.0,
      "means": {
        "prompt_plus_generation_per_observed_iteration": 2.733518665607625,
        "generation_per_all_observed_iterations": 2.545671167593328,
        "sum_equals_prompt_plus_generation": true
      }
    },
    "c1_2000": {
      "drafts": 916.0,
      "proposed": 1832.0,
      "accepted": 1085.0,
      "adjusted_unaccepted": 747.0,
      "generation": 2000.0,
      "prompt": 473.0,
      "histogram_count": 917.0,
      "histogram_sum": 2473.0,
      "means": {
        "prompt_plus_generation_per_observed_iteration": 2.6968375136314067,
        "generation_per_all_observed_iterations": 2.1810250817884405,
        "sum_equals_prompt_plus_generation": true
      }
    },
    "c1_10000": {
      "drafts": 3808.0,
      "proposed": 7616.0,
      "accepted": 6191.0,
      "adjusted_unaccepted": 1425.0,
      "generation": 10000.0,
      "prompt": 473.0,
      "histogram_count": 3809.0,
      "histogram_sum": 10473.0,
      "means": {
        "prompt_plus_generation_per_observed_iteration": 2.749540561827251,
        "generation_per_all_observed_iterations": 2.625360987135731,
        "sum_equals_prompt_plus_generation": true
      }
    }
  },
  "MTP_2K_draft_batches": 916.0,
  "MTP_2K_proposed": 1832.0,
  "MTP_2K_accepted": 1085.0,
  "MTP_2K_acceptance_percent": 59.224890829694324,
  "MTP_2K_committed": 2000.0,
  "detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
  "detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
  "detail_source_pointer": "w8-tp4-mtp2",
  "retest_strict_tokens": 5065,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 4156.0,
        "vllm:request_generation_tokens_sum": 5065.0,
        "vllm:spec_decode_num_accepted_tokens_total": 2986.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 2986.0,
        "vllm:spec_decode_num_drafts_total": 2078.0,
        "vllm:generation_tokens_total": 5065.0,
        "vllm:request_generation_tokens_count": 1.0
      },
      "accepted_per_position": {
        "0": 1725.0,
        "1": 1261.0
      },
      "accepted_fraction": 0.7184793070259865,
      "committed_emitted_tokens": 5065.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "52626453501dda855f1e312dff99fb8be8ab0168b8ceb2aed5f39894f51dcec0",
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "d08dc97c94d35a272c3296a02796beec082e2f0f769b29ee9dbc79fd8d61a501"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_drafts_total": 822.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1178.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1178.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_draft_tokens_total": 1644.0
      },
      "accepted_per_position": {
        "0": 669.0,
        "1": 509.0
      },
      "accepted_fraction": 0.7165450121654501,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "afedac1311f18a5f412481eecac893a8c4e54b59275f102c389b4eb471af9cd8",
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "91830c16667855b20d0b7897c58007bd337986836036f4de1a6476a12faf324b"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 6275.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6275.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:spec_decode_num_draft_tokens_total": 7452.0,
        "vllm:spec_decode_num_drafts_total": 3726.0
      },
      "accepted_per_position": {
        "0": 3373.0,
        "1": 2902.0
      },
      "accepted_fraction": 0.8420558239398819,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "7f92ce7ac776a0f277eabd37bb200446633e17d526b88b9190bc2ad03f51111a",
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "5016c951dee5e71c622875bcd6b1bd81c125066f0d8a119b07898f086e4ad657"
      }
    }
  },
  "retest_MTP_2K_accepted": 1178.0,
  "retest_MTP_2K_proposed": 1644.0,
  "retest_MTP_2K_draft_batches": 822.0,
  "retest_MTP_2K_acceptance_percent": 71.65450121654501,
  "retest_diagnostic_counter_audit": "retest-evidence/mi50-w8-tp4-mtp2/local-diagnostic-fixed-audit.json",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 25681870848,
    "1": 25698664448,
    "2": 25698631680,
    "3": 25648300032
  },
  "retest_memory_observation_count": 34,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w8-tp4-mtp2/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 14.35,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 14.35,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 220800,
  "retest_memory_log_report": {
    "profile": "mi50-w8-tp4-mtp2",
    "image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
    "container": "2901da8586316a8b0a053199945a78afd8c877762f777773550bd1a8463cbba4",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 14.43,
    "retest_logged_model_load_seconds": 63.480969,
    "retest_logged_available_KV_GiB": 14.35,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 14.35,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 220800,
    "retest_logged_context_capacity_x": 6.39,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w8-tp4-mtp2/service-final.log",
    "source_log_sha256": "37ad86fe2abc818ffd5877f746e3891fdfc6c948a6eec70e813d4b171a7d0e3d",
    "sealed_index_sha256": "cad06a029821c6b96a4260e7a9e1f3938969ecef85de855a0ab156d226b112e4",
    "source_lines": {
      "model_load": {
        "line": 178,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:01:37 [gpu_model_runner.py:4584] Model loading took 14.43 GiB memory and 63.480969 seconds"
      },
      "automatic_KV_budget": {
        "line": 265,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:06:40 [gpu_worker.py:453] Available KV cache memory: 14.35 GiB"
      },
      "grouped_KV_tokens": {
        "line": 269,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 19:06:40 [kv_cache_utils.py:1314] GPU KV cache size: 220,800 tokens"
      },
      "context_capacity": {
        "line": 270,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 19:06:40 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.39x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w8-tp4-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 23.933746337890625,
  "sampled_peak_VRAM_GiB": 23.93376922607422
}
MI50Current Qwen3.88-bit RTNQwen/Qwen3.8-27BTP2MTP22K accepted 1138 / 1724 (66.01%) PASS8595 tokens 19.47917.083 18.051 / 120.73810K: 18.045 / 603.450Strict: 18.056 / 501.280 11.60 / 23.27packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:1ee2a65021b0a54e0f8784ea5e7487504cb48741de9af1bad226b5427c4fa96b
Runtime image config: sha256:b1450af604a4f69af53a3ac83f783d9572fcfafb3705c44d5730d50ba99baaad
Registry verificationDerived host-portability image; recorded benchmark parent: sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7
Exact compatibility change
Replaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately.
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp2-mtp2
GPU_RENDER_NODES=renderD128,renderD129

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-w8-tp2-mtp2-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \
  -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \
  -e MI50_RTN_BITS=8 \
  -e HIP_VISIBLE_DEVICES=0,1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \
  -e MI50_RTN_TP=2 \
  -e HF_HUB_OFFLINE=1 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  -e MAX_JOBS=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp2/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp2/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp2/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp2,dst=/runwork \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp2-mtp2
GPU_RENDER_NODES=renderD128,renderD129

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-mi50-r3.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 18.427 · eligible10K 16.555 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 17304,
    "content_chars": 15852,
    "content_has_think_close": false,
    "elapsed_seconds": 1191.0941870175302,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T19:08:11Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.00875687412917614,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 15852,
    "qwen_answer_text_sha256": "36bfb02ddbea6ad955edaccfcbdc78935636fadd3c34ac94c890d877d6ade3d7",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 15852,
    "qwen_visible_text_sha256": "36bfb02ddbea6ad955edaccfcbdc78935636fadd3c34ac94c890d877d6ade3d7",
    "raw_has_think_close": false,
    "reasoning_chars": 74240,
    "text_chars": 90093,
    "text_sha256": "af766f503825569b7901850243fb5fb03e584e6691d634ed3a89c9873e3b730a",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 1173.0919499928132,
    "vllm_decode_tps": 14.75076186492117,
    "vllm_generation_tokens_delta": 17304.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 17.993215464055538,
    "vllm_prefill_tps": 26.287686097290212,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 14.527818361139667
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 126.62693756446242,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T19:10:18Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008112105540931225,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9243,
    "text_chars": 9243,
    "text_sha256": "c23f3a143c4c1bb3d52df4c7713c11ee2fa0b4c7ef0afc41c3f94f658443aa44",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 108.535968715325,
    "vllm_decode_tps": 18.427070985524868,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 17.998336000368,
    "vllm_prefill_tps": 26.280207236398347,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 15.79442761917742
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 622.0402731243521,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T19:20:40Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.007797887548804283,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 51149,
    "text_chars": 51149,
    "text_sha256": "7df23c557b4477aa92c6e33883414281ad35c7c170650fa101d9593bd5e39d64",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 604.0367387775332,
    "vllm_decode_tps": 16.555284402465794,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 17.99445286858827,
    "vllm_prefill_tps": 26.28587840120913,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 16.076129524174554
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestMTP counter replayPASS6,643 natural tokens2K 18.08110K 17.682 · diagnostic2K MTP accepted 1077 / 1844 (58.41%)
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T19:35:52.777581181Z",
      "FinishedAt": "2026-10-26T20:27:29.68673334Z"
    },
    "devices_after": [
      {
        "card": 1,
        "render": 129,
        "pci": "0000:06:00.0",
        "actual_card": "card0",
        "busy_percent": 0,
        "vram_bytes": 17268699136,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:0a:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 17447673856,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T20:27:29.842167+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 18.08137387676853,
  "retest_10K_TPS": 17.681917045281846,
  "retest_strict_tokens": 6643,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 6643,
      "elapsed_seconds": 381.4662659961032,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 361.9941003039712,
      "vllm_decode_tps": 18.351127806839354,
      "vllm_generation_tokens_delta": 6643.0,
      "vllm_prefill_seconds_delta": 19.46000845602248,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 129.1342354209628,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 110.61106382904109,
      "vllm_decode_tps": 18.08137387676853,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 18.51244649104774,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 586.0752006319817,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 565.5495371000143,
      "vllm_decode_tps": 17.681917045281846,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 20.5129211419262,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 18.51244649104774,
  "retest_2K_total_seconds": 129.1342354209628,
  "retest_10K_prefill_seconds": 20.5129211419262,
  "retest_10K_total_seconds": 586.0752006319817,
  "retest_strict_prefill_seconds": 19.46000845602248,
  "retest_strict_total_seconds": 381.4662659961032,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_drafts_total": 2651.0,
        "vllm:spec_decode_num_draft_tokens_total": 5302.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_total": 3991.0,
        "vllm:request_generation_tokens_sum": 6643.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 3991.0,
        "vllm:generation_tokens_total": 6643.0
      },
      "accepted_per_position": {
        "0": 2190.0,
        "1": 1801.0
      },
      "accepted_fraction": 0.7527348170501698,
      "committed_emitted_tokens": 6643.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "ee5967d10ba0e760ed2c0a067f815ef122fb81d0ea3f9fc41067bbc21c23493b",
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "13f8368194830434abc764415caecd01230d8c66b06bfa2260089fd04e785f83"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 1844.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1077.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1077.0,
        "vllm:spec_decode_num_drafts_total": 922.0,
        "vllm:generation_tokens_total": 2000.0
      },
      "accepted_per_position": {
        "0": 653.0,
        "1": 424.0
      },
      "accepted_fraction": 0.584056399132321,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "0c4587f09022dd78756b9c5bf8a50385d3399caa00b05b618df181b715b0190f",
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "0f55e3c3423fdc6d85ab588c3654e51882febc921946a4ec66cac49e21e74a78"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 7582.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6208.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6208.0,
        "vllm:spec_decode_num_drafts_total": 3791.0,
        "vllm:generation_tokens_total": 10000.0
      },
      "accepted_per_position": {
        "0": 3310.0,
        "1": 2898.0
      },
      "accepted_fraction": 0.8187813241888684,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "e806c3d489768238aaa0c29d2db857e00cbefa0f0aba50d5e411a7344842ee89",
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "67a912927c33d611ba2f51361c6d0b04ffbbae8ff08a9f92a137738da181cb21"
      }
    }
  },
  "retest_MTP_2K_accepted": 1077.0,
  "retest_MTP_2K_proposed": 1844.0,
  "retest_MTP_2K_draft_batches": 922.0,
  "retest_MTP_2K_acceptance_percent": 58.4056399132321,
  "retest_loaded_admission": "Complete record: chart-details/row-07-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30519541760,
    "1": 30519545856
  },
  "retest_memory_observation_count": 49,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w8-tp2-mtp2/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 28.09,
  "retest_logged_model_load_seconds": 123.709991,
  "retest_logged_available_KV_GiB": null,
  "retest_explicit_KV_budget_bytes": 6442450944,
  "retest_KV_budget_GiB": 6.0,
  "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
  "retest_logged_GPU_KV_tokens": 45600,
  "retest_logged_context_capacity_x": 1.34,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-w8-tp2-mtp2",
    "image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
    "container": "4b05f105aed3db4ec411d4497fe439e146cb346b8d83fd5b1527647707fdcef3",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 28.09,
    "retest_logged_model_load_seconds": 123.709991,
    "retest_logged_available_KV_GiB": null,
    "retest_explicit_KV_budget_bytes": 6442450944,
    "retest_KV_budget_GiB": 6.0,
    "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
    "retest_logged_GPU_KV_tokens": 45600,
    "retest_logged_context_capacity_x": 1.34,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w8-tp2-mtp2/service-final.log",
    "source_log_sha256": "553939c869da8d99f5604c5504ab31cb9157600462eed3a586e6c78531ee843f",
    "sealed_index_sha256": "0bae66b49db8c96e0a89da91a15f660adb325e42c22238bd46faecbe24194dbe",
    "source_lines": {
      "model_load": {
        "line": 144,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:40:05 [gpu_model_runner.py:4584] Model loading took 28.09 GiB memory and 123.709991 seconds"
      },
      "explicit_KV_budget": {
        "line": 7,
        "text": "(APIServer pid=1) INFO 10-26 19:36:24 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 8, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 2}, 'optimization_level': '3'}"
      },
      "explicit_budget_worker_reports": [
        {
          "line": 156,
          "text": "(Worker pid=217) (Worker_TP1 pid=217) INFO 10-26 19:41:01 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        },
        {
          "line": 157,
          "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:41:01 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        }
      ],
      "grouped_KV_tokens": {
        "line": 159,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 19:41:01 [kv_cache_utils.py:1314] GPU KV cache size: 45,600 tokens"
      },
      "context_capacity": {
        "line": 160,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 19:41:01 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.34x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w8-tp2-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.423542022705078
}
Complete retest loaded admission
Retest KV budget: 6.000 GiB (explicit)45,600 grouped KV tokens · 1.34× reported capacityRetest sampled VRAM: 28.424 GiB max/cardRetest memory reports and limits
Evidence and limits

completed matrix cell

{
  "strict_tokens": 8595,
  "source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w8-tp2-mtp2",
  "source_revision": "016143b00c7a08e25ff85eb83dc052be148215ab",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "sampled_VRAM_after_ladder_bytes_by_rank": "[24989736960,24989675520]",
  "comparison_limit": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
  "recorded_configuration_details": {
    "cell": "w8-tp2-mtp2",
    "bits": 8,
    "group_size": 128,
    "TP": 2,
    "native_MTP": true,
    "num_speculative_tokens": 2,
    "strict_valid": true,
    "official_2k_eligible": true,
    "strict_returned_tokens": 8595,
    "strict_decode_TPS": 17.78735896682357,
    "strict_prefill_seconds": 18.055707121035084,
    "strict_total_seconds": 501.27962843398564,
    "fixed_2000_returned_tokens": 2000,
    "fixed_2000_decode_TPS": 19.47859055870054,
    "fixed_2000_prefill_seconds": 18.05051256698789,
    "fixed_2000_total_seconds": 120.73758545302553,
    "fixed_10000_returned_tokens": 10000,
    "fixed_10000_decode_TPS": 17.08257754036289,
    "fixed_10000_prefill_seconds": 18.045061738986988,
    "fixed_10000_total_seconds": 603.449919919949,
    "packed_weight_bytes_per_rank": 12460636160,
    "packed_weight_bytes_total": 24921272320,
    "packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
    "render_nodes_by_rank": [
      "renderD129",
      "renderD130"
    ],
    "sampled_VRAM_before_ladder_bytes_by_rank": [
      24989736960,
      24989675520
    ],
    "sampled_VRAM_after_ladder_bytes_by_rank": [
      24989736960,
      24989675520
    ],
    "allocated_after_load_bytes_by_rank": [
      15637867520,
      15637867520
    ],
    "allocated_after_cache_bytes_by_rank": [
      22076637184,
      22076637184
    ],
    "reserved_after_cache_bytes_by_rank": [
      23408410624,
      23408410624
    ],
    "sampled_attention_KV_first_layer_views_by_rank": [
      {
        "rank": 0,
        "views": [
          {
            "shape": [
              231,
              2,
              800,
              2,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 378470400
          }
        ]
      },
      {
        "rank": 1,
        "views": [
          {
            "shape": [
              231,
              2,
              800,
              2,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 378470400
          }
        ]
      }
    ],
    "KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
    "KV_budget": "6442450944 bytes explicit",
    "context": 131072,
    "greedy": true,
    "concurrency": 1,
    "activation_dtype": "FP16",
    "attention_KV_dtype": "FP16",
    "SSM_dtype": "float32",
    "native_MTP_head_dtype": "FP16",
    "native_MTP_unshared_parameter_bytes_by_rank": [
      424725504,
      424725504
    ],
    "model_repository": "Qwen/Qwen3.8-27B",
    "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "serving_source": "016143b00c7a08e25ff85eb83dc052be148215ab",
    "image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
    "native_MTP_model_source_sha256": "46f57b9c511c9199aa83421fa7fb167b08d5d80c14377948690cdf67d75453c3",
    "MTP_request_accounting_by_tier": {
      "c1_128_strict": {
        "drafts": 3364.0,
        "proposed": 6728.0,
        "accepted": 5230.0,
        "adjusted_unaccepted": 1498.0,
        "generation": 8595.0,
        "prompt": 473.0,
        "histogram_count": 3365.0,
        "histogram_sum": 9068.0,
        "means": {
          "prompt_plus_generation_per_observed_iteration": 2.69479940564636,
          "generation_per_all_observed_iterations": 2.554234769687964,
          "sum_equals_prompt_plus_generation": true
        }
      },
      "c1_2000": {
        "drafts": 862.0,
        "proposed": 1724.0,
        "accepted": 1138.0,
        "adjusted_unaccepted": 586.0,
        "generation": 2000.0,
        "prompt": 473.0,
        "histogram_count": 863.0,
        "histogram_sum": 2473.0,
        "means": {
          "prompt_plus_generation_per_observed_iteration": 2.86558516801854,
          "generation_per_all_observed_iterations": 2.317497103128621,
          "sum_equals_prompt_plus_generation": true
        }
      },
      "c1_10000": {
        "drafts": 3914.0,
        "proposed": 7828.0,
        "accepted": 6087.0,
        "adjusted_unaccepted": 1741.0,
        "generation": 10000.0,
        "prompt": 473.0,
        "histogram_count": 3915.0,
        "histogram_sum": 10473.0,
        "means": {
          "prompt_plus_generation_per_observed_iteration": 2.675095785440613,
          "generation_per_all_observed_iterations": 2.554278416347382,
          "sum_equals_prompt_plus_generation": true
        }
      }
    },
    "MTP_accounting_scope": "P-A is adjusted unaccepted, not scheduler rollback rejection; iteration means include prefill, not exact decode-only emissions.",
    "runtime_difference": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
    "MTP_scheduler_rollback_rejection_count": "unobservable",
    "host_exclusivity": "unproven",
    "serving_CPUs": "0-7,16-23",
    "cache_placement": "fresh private admitted rootFS",
    "compile_cache_disabled": true,
    "official_2k_samples": 1,
    "repeatability_established": false,
    "FP16_goal_qualifying_ladders": 0,
    "evidence": "../../tuning/qwen38-rtn-mi50/w8-tp2-mtp-result-20261023.json"
  },
  "MTP_accounting_by_tier": {
    "c1_128_strict": {
      "drafts": 3364.0,
      "proposed": 6728.0,
      "accepted": 5230.0,
      "adjusted_unaccepted": 1498.0,
      "generation": 8595.0,
      "prompt": 473.0,
      "histogram_count": 3365.0,
      "histogram_sum": 9068.0,
      "means": {
        "prompt_plus_generation_per_observed_iteration": 2.69479940564636,
        "generation_per_all_observed_iterations": 2.554234769687964,
        "sum_equals_prompt_plus_generation": true
      }
    },
    "c1_2000": {
      "drafts": 862.0,
      "proposed": 1724.0,
      "accepted": 1138.0,
      "adjusted_unaccepted": 586.0,
      "generation": 2000.0,
      "prompt": 473.0,
      "histogram_count": 863.0,
      "histogram_sum": 2473.0,
      "means": {
        "prompt_plus_generation_per_observed_iteration": 2.86558516801854,
        "generation_per_all_observed_iterations": 2.317497103128621,
        "sum_equals_prompt_plus_generation": true
      }
    },
    "c1_10000": {
      "drafts": 3914.0,
      "proposed": 7828.0,
      "accepted": 6087.0,
      "adjusted_unaccepted": 1741.0,
      "generation": 10000.0,
      "prompt": 473.0,
      "histogram_count": 3915.0,
      "histogram_sum": 10473.0,
      "means": {
        "prompt_plus_generation_per_observed_iteration": 2.675095785440613,
        "generation_per_all_observed_iterations": 2.554278416347382,
        "sum_equals_prompt_plus_generation": true
      }
    }
  },
  "MTP_2K_draft_batches": 862.0,
  "MTP_2K_proposed": 1724.0,
  "MTP_2K_accepted": 1138.0,
  "MTP_2K_acceptance_percent": 66.0092807424594,
  "MTP_2K_committed": 2000.0,
  "detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
  "detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
  "detail_source_pointer": "w8-tp2-mtp2",
  "retest_strict_tokens": 6643,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_drafts_total": 2651.0,
        "vllm:spec_decode_num_draft_tokens_total": 5302.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_total": 3991.0,
        "vllm:request_generation_tokens_sum": 6643.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 3991.0,
        "vllm:generation_tokens_total": 6643.0
      },
      "accepted_per_position": {
        "0": 2190.0,
        "1": 1801.0
      },
      "accepted_fraction": 0.7527348170501698,
      "committed_emitted_tokens": 6643.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "ee5967d10ba0e760ed2c0a067f815ef122fb81d0ea3f9fc41067bbc21c23493b",
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "13f8368194830434abc764415caecd01230d8c66b06bfa2260089fd04e785f83"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 1844.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1077.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1077.0,
        "vllm:spec_decode_num_drafts_total": 922.0,
        "vllm:generation_tokens_total": 2000.0
      },
      "accepted_per_position": {
        "0": 653.0,
        "1": 424.0
      },
      "accepted_fraction": 0.584056399132321,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "0c4587f09022dd78756b9c5bf8a50385d3399caa00b05b618df181b715b0190f",
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "0f55e3c3423fdc6d85ab588c3654e51882febc921946a4ec66cac49e21e74a78"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 7582.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6208.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6208.0,
        "vllm:spec_decode_num_drafts_total": 3791.0,
        "vllm:generation_tokens_total": 10000.0
      },
      "accepted_per_position": {
        "0": 3310.0,
        "1": 2898.0
      },
      "accepted_fraction": 0.8187813241888684,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "e806c3d489768238aaa0c29d2db857e00cbefa0f0aba50d5e411a7344842ee89",
        "/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "67a912927c33d611ba2f51361c6d0b04ffbbae8ff08a9f92a137738da181cb21"
      }
    }
  },
  "retest_MTP_2K_accepted": 1077.0,
  "retest_MTP_2K_proposed": 1844.0,
  "retest_MTP_2K_draft_batches": 922.0,
  "retest_MTP_2K_acceptance_percent": 58.4056399132321,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30519541760,
    "1": 30519545856
  },
  "retest_memory_observation_count": 49,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w8-tp2-mtp2/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": null,
  "retest_explicit_KV_budget_bytes": 6442450944,
  "retest_KV_budget_GiB": 6.0,
  "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
  "retest_logged_GPU_KV_tokens": 45600,
  "retest_memory_log_report": {
    "profile": "mi50-w8-tp2-mtp2",
    "image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
    "container": "4b05f105aed3db4ec411d4497fe439e146cb346b8d83fd5b1527647707fdcef3",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 28.09,
    "retest_logged_model_load_seconds": 123.709991,
    "retest_logged_available_KV_GiB": null,
    "retest_explicit_KV_budget_bytes": 6442450944,
    "retest_KV_budget_GiB": 6.0,
    "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
    "retest_logged_GPU_KV_tokens": 45600,
    "retest_logged_context_capacity_x": 1.34,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w8-tp2-mtp2/service-final.log",
    "source_log_sha256": "553939c869da8d99f5604c5504ab31cb9157600462eed3a586e6c78531ee843f",
    "sealed_index_sha256": "0bae66b49db8c96e0a89da91a15f660adb325e42c22238bd46faecbe24194dbe",
    "source_lines": {
      "model_load": {
        "line": 144,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:40:05 [gpu_model_runner.py:4584] Model loading took 28.09 GiB memory and 123.709991 seconds"
      },
      "explicit_KV_budget": {
        "line": 7,
        "text": "(APIServer pid=1) INFO 10-26 19:36:24 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 8, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 2}, 'optimization_level': '3'}"
      },
      "explicit_budget_worker_reports": [
        {
          "line": 156,
          "text": "(Worker pid=217) (Worker_TP1 pid=217) INFO 10-26 19:41:01 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        },
        {
          "line": 157,
          "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:41:01 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        }
      ],
      "grouped_KV_tokens": {
        "line": 159,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 19:41:01 [kv_cache_utils.py:1314] GPU KV cache size: 45,600 tokens"
      },
      "context_capacity": {
        "line": 160,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 19:41:01 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.34x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w8-tp2-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.423542022705078,
  "sampled_peak_VRAM_GiB": 23.27350616455078
}
MI50Current Qwen3.84-bit RTNQwen/Qwen3.8-27BTP4MTP off FAIL3407 tokens 66.085*62.679 11.545 / 41.82210K: 11.992 / 171.549Strict: 11.261 / 63.139 2.95 / 23.25packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:4faf22e8db7711abff5cd83ee1a7bb8e65ae8041fc5552b4652083b093aba065
Runtime image config: sha256:8376baaa8d3491c010ada3c623d856873b60675a433fd512e258d730cb7b6513
Registry verificationDerived host-portability image; recorded benchmark parent: sha256:72f25a82297a6d478d4f4fee305b8c7853999edeb0c86d852b8370849f810fe3
Exact compatibility change
Replaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately.
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp4-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-w4-tp4-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \
  --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \
  -e MI50_RTN_TP=4 \
  -e HIP_VISIBLE_DEVICES=0,1,2,3 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \
  -e MI50_RTN_BITS=4 \
  -e HF_HUB_OFFLINE=1 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp0/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp0,dst=/runwork \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp0/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:72f25a82297a6d478d4f4fee305b8c7853999edeb0c86d852b8370849f810fe3
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp4-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-mi50-r3.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 67.903 · eligible10K 64.254 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3973,
    "content_chars": 7161,
    "content_has_think_close": false,
    "elapsed_seconds": 67.34906083345413,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T19:45:03Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.008497083559632301,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 7161,
    "qwen_answer_text_sha256": "734dd5ae16a2c3fa33a02e8e60a6948b7777cfd78b2a6a161ac39b16f47824d3",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 7161,
    "qwen_visible_text_sha256": "734dd5ae16a2c3fa33a02e8e60a6948b7777cfd78b2a6a161ac39b16f47824d3",
    "raw_has_think_close": false,
    "reasoning_chars": 13048,
    "text_chars": 20210,
    "text_sha256": "23747931019b8d463b4463290119d973395421ba6b542dbe1079699ef06d0062",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 59.143225917592645,
    "vllm_decode_tps": 67.1759096390141,
    "vllm_generation_tokens_delta": 3973.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 8.197571682743728,
    "vllm_prefill_tps": 57.70001389505225,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 58.99117152984116
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 37.77216265629977,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T19:45:41Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.005713380873203278,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9486,
    "text_chars": 9486,
    "text_sha256": "62d22931eb970f7b66da3c3cb82ac07ae6bc058eb360faff682ef9ccb1d67817",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 29.453894706442952,
    "vllm_decode_tps": 67.90273476337599,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 8.310188611969352,
    "vllm_prefill_tps": 56.91808237886772,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 52.949046582230395
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 6738,
    "content_has_think_close": false,
    "elapsed_seconds": 164.05896650161594,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T19:48:26Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008259626105427742,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 6738,
    "qwen_answer_text_sha256": "ceea9ed46978d39a3f977aea6b797cdfb9234d9d38cdbb94405b5f282eb6536d",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 6738,
    "qwen_visible_text_sha256": "ceea9ed46978d39a3f977aea6b797cdfb9234d9d38cdbb94405b5f282eb6536d",
    "raw_has_think_close": false,
    "reasoning_chars": 33979,
    "text_chars": 40718,
    "text_sha256": "560fb1ae582b985f835b761352697d70b5f65cdb19bf1a40c75d0c44407bb112",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 155.63261205423623,
    "vllm_decode_tps": 64.25388527511902,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 8.417382342740893,
    "vllm_prefill_tps": 56.19324164452536,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 60.95369374341087
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS3,098 natural tokens2K 66.39710K 62.979 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T16:06:48.027490738Z",
      "FinishedAt": "2026-10-26T16:29:20.154019642Z"
    },
    "devices_after": [
      {
        "card": 1,
        "render": 129,
        "pci": "0000:06:00.0",
        "actual_card": "card0",
        "busy_percent": 0,
        "vram_bytes": 9762500608,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:0a:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 10040770560,
        "dpm": "auto"
      },
      {
        "card": 3,
        "render": 131,
        "pci": "0000:0d:00.0",
        "actual_card": "card3",
        "busy_percent": 0,
        "vram_bytes": 10040770560,
        "dpm": "auto"
      },
      {
        "card": 4,
        "render": 132,
        "pci": "0000:10:00.0",
        "actual_card": "card4",
        "busy_percent": 0,
        "vram_bytes": 9990438912,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T16:29:20.298170+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 66.39721642663191,
  "retest_10K_TPS": 62.979410578946336,
  "retest_strict_tokens": 3098,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 3098,
      "elapsed_seconds": 62.61535412305966,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 46.883774308953434,
      "vllm_decode_tps": 66.07829778346945,
      "vllm_generation_tokens_delta": 3098.0,
      "vllm_prefill_seconds_delta": 15.720751523040235,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 47.03673107700888,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 30.12174466997385,
      "vllm_decode_tps": 66.39721642663191,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 16.904422716936097,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 175.82567254197784,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 158.78205127792899,
      "vllm_decode_tps": 62.979410578946336,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 17.030237389029935,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 16.904422716936097,
  "retest_2K_total_seconds": 47.03673107700888,
  "retest_10K_prefill_seconds": 17.030237389029935,
  "retest_10K_total_seconds": 175.82567254197784,
  "retest_strict_prefill_seconds": 15.720751523040235,
  "retest_strict_total_seconds": 62.61535412305966,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": "Complete record: chart-details/row-08-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 24943259648,
    "1": 24960045056,
    "2": 24959987712,
    "3": 24909742080
  },
  "retest_memory_observation_count": 3,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w4-tp4-mtp0/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 13.16,
  "retest_logged_model_load_seconds": 79.138209,
  "retest_logged_available_KV_GiB": 16.01,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 16.01,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 261856,
  "retest_logged_context_capacity_x": 7.82,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-w4-tp4-mtp0",
    "image": "sha256:72f25a82297a6d478d4f4fee305b8c7853999edeb0c86d852b8370849f810fe3",
    "container": "562cda56103fb421e6adf66cd6726f7ba830c1f89fbcd0e042644e91a1d8e6e8",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 13.16,
    "retest_logged_model_load_seconds": 79.138209,
    "retest_logged_available_KV_GiB": 16.01,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 16.01,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 261856,
    "retest_logged_context_capacity_x": 7.82,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w4-tp4-mtp0/service-final.log",
    "source_log_sha256": "82234bdbf1435b75c9834c4de1afa8954c2357b23180cc78f3c19bc1ab470fab",
    "sealed_index_sha256": "ce64d445ea0f358d6dcc1f3f3a09f310416abdf4782a992701442ea1c778834b",
    "source_lines": {
      "model_load": {
        "line": 144,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:10:29 [gpu_model_runner.py:4584] Model loading took 13.16 GiB memory and 79.138209 seconds"
      },
      "automatic_KV_budget": {
        "line": 167,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:11:47 [gpu_worker.py:453] Available KV cache memory: 16.01 GiB"
      },
      "grouped_KV_tokens": {
        "line": 171,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:11:47 [kv_cache_utils.py:1314] GPU KV cache size: 261,856 tokens"
      },
      "context_capacity": {
        "line": 172,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:11:47 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 7.82x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w4-tp4-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 23.245853424072266
}
Complete retest loaded admission
Retest KV budget: 16.010 GiB (automatic, rounded)261,856 grouped KV tokens · 7.82× reported capacityRetest sampled VRAM: 23.246 GiB max/cardRetest memory reports and limits
Evidence and limits

strict failed; speeds diagnostic/ineligible

{
  "strict_tokens": 3407,
  "source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w4-tp4-mtpoff",
  "source_revision": "e60a9ec31b6bed8a09f6d88972d861f4253127a1",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "sampled_VRAM_after_ladder_bytes_by_rank": "[24943263744,24960000000,24960081920,24909647872]",
  "comparison_limit": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
  "recorded_configuration_details": {
    "cell": "w4-tp4-mtpoff",
    "bits": 4,
    "group_size": 128,
    "TP": 4,
    "native_MTP": false,
    "num_speculative_tokens": 0,
    "strict_valid": false,
    "official_2k_eligible": false,
    "strict_returned_tokens": 3407,
    "strict_decode_TPS": 65.68917102453123,
    "strict_prefill_seconds": 11.260670032002963,
    "strict_total_seconds": 63.138696900976356,
    "fixed_2000_returned_tokens": 2000,
    "fixed_2000_decode_TPS": 66.08481149139995,
    "fixed_2000_prefill_seconds": 11.544884058996104,
    "fixed_2000_total_seconds": 41.821840891003376,
    "fixed_10000_returned_tokens": 10000,
    "fixed_10000_decode_TPS": 62.67869554052399,
    "fixed_10000_prefill_seconds": 11.992151816986734,
    "fixed_10000_total_seconds": 171.5490621229983,
    "packed_weight_bytes_per_rank": 3162718720,
    "packed_weight_bytes_total": 12650874880,
    "packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
    "render_nodes_by_rank": [
      "renderD129",
      "renderD130",
      "renderD131",
      "renderD132"
    ],
    "sampled_VRAM_before_ladder_bytes_by_rank": [
      24943263744,
      24960000000,
      24960081920,
      24909647872
    ],
    "sampled_VRAM_after_ladder_bytes_by_rank": [
      24943263744,
      24960000000,
      24960081920,
      24909647872
    ],
    "allocated_after_load_bytes_by_rank": [
      4590513664,
      4590513664,
      4590513664,
      4590513664
    ],
    "allocated_after_cache_bytes_by_rank": [
      21839094272,
      21839094272,
      21839094272,
      21839094272
    ],
    "reserved_after_cache_bytes_by_rank": [
      23108517888,
      23108517888,
      23108517888,
      23108517888
    ],
    "sampled_attention_KV_first_layer_views_by_rank": [
      {
        "rank": 0,
        "views": [
          {
            "shape": [
              1337,
              2,
              784,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 1073364992
          }
        ]
      },
      {
        "rank": 1,
        "views": [
          {
            "shape": [
              1337,
              2,
              784,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 1073364992
          }
        ]
      },
      {
        "rank": 2,
        "views": [
          {
            "shape": [
              1337,
              2,
              784,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 1073364992
          }
        ]
      },
      {
        "rank": 3,
        "views": [
          {
            "shape": [
              1337,
              2,
              784,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 1073364992
          }
        ]
      }
    ],
    "KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
    "KV_budget": "automatic; actual allocation receipts retained",
    "context": 131072,
    "greedy": true,
    "concurrency": 1,
    "activation_dtype": "FP16",
    "attention_KV_dtype": "FP16",
    "SSM_dtype": "float32",
    "native_MTP_head_dtype": "not used",
    "native_MTP_unshared_parameter_bytes_by_rank": [],
    "model_repository": "Qwen/Qwen3.8-27B",
    "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "serving_source": "e60a9ec31b6bed8a09f6d88972d861f4253127a1",
    "image": "sha256:72f25a82297a6d478d4f4fee305b8c7853999edeb0c86d852b8370849f810fe3",
    "native_MTP_model_source_sha256": "not used",
    "MTP_request_accounting_by_tier": {},
    "MTP_accounting_scope": "not used",
    "runtime_difference": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
    "MTP_scheduler_rollback_rejection_count": "not used",
    "host_exclusivity": "unproven",
    "serving_CPUs": "0-7,16-23",
    "cache_placement": "fresh private admitted rootFS",
    "compile_cache_disabled": true,
    "official_2k_samples": 1,
    "repeatability_established": false,
    "FP16_goal_qualifying_ladders": 0,
    "evidence": "../../tuning/qwen38-rtn-mi50/w4-tp4-result-20261022.json"
  },
  "detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
  "detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
  "detail_source_pointer": "w4-tp4-mtpoff",
  "retest_strict_tokens": 3098,
  "retest_MTP_accounting_by_tier": {},
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 24943259648,
    "1": 24960045056,
    "2": 24959987712,
    "3": 24909742080
  },
  "retest_memory_observation_count": 3,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w4-tp4-mtp0/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 16.01,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 16.01,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 261856,
  "retest_memory_log_report": {
    "profile": "mi50-w4-tp4-mtp0",
    "image": "sha256:72f25a82297a6d478d4f4fee305b8c7853999edeb0c86d852b8370849f810fe3",
    "container": "562cda56103fb421e6adf66cd6726f7ba830c1f89fbcd0e042644e91a1d8e6e8",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 13.16,
    "retest_logged_model_load_seconds": 79.138209,
    "retest_logged_available_KV_GiB": 16.01,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 16.01,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 261856,
    "retest_logged_context_capacity_x": 7.82,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w4-tp4-mtp0/service-final.log",
    "source_log_sha256": "82234bdbf1435b75c9834c4de1afa8954c2357b23180cc78f3c19bc1ab470fab",
    "sealed_index_sha256": "ce64d445ea0f358d6dcc1f3f3a09f310416abdf4782a992701442ea1c778834b",
    "source_lines": {
      "model_load": {
        "line": 144,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:10:29 [gpu_model_runner.py:4584] Model loading took 13.16 GiB memory and 79.138209 seconds"
      },
      "automatic_KV_budget": {
        "line": 167,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:11:47 [gpu_worker.py:453] Available KV cache memory: 16.01 GiB"
      },
      "grouped_KV_tokens": {
        "line": 171,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:11:47 [kv_cache_utils.py:1314] GPU KV cache size: 261,856 tokens"
      },
      "context_capacity": {
        "line": 172,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:11:47 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 7.82x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w4-tp4-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 23.245853424072266,
  "sampled_peak_VRAM_GiB": 23.245887756347656
}
MI50Current Qwen3.84-bit RTNQwen/Qwen3.8-27BTP2MTP off PASS9449 tokens 51.18147.579 16.011 / 55.10010K: 15.533 / 225.722Strict: 14.604 / 211.881 5.89 / 17.89packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:a506e684a1f3142d071857d89f8fee8f17f0a30e92970c76d163c03380004961
Runtime image config: sha256:fe0d81ba97b6b8a432db928d19d03b511f15de9e40e5d25be81797d9c3e1694b
Registry verificationDerived host-portability image; recorded benchmark parent: sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a
Exact compatibility change
Replaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately.
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp2-mtp0
GPU_RENDER_NODES=renderD128,renderD129

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-w4-tp2-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --user root \
  --workdir /tmp \
  -e HIP_VISIBLE_DEVICES=0,1 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \
  -e MI50_RTN_BITS=4 \
  -e MI50_RTN_TP=2 \
  -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \
  -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \
  -e HF_HUB_OFFLINE=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp0/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp0,dst=/runwork \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp0/cache/.triton,dst=/root/.triton \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp2-mtp0
GPU_RENDER_NODES=renderD128,renderD129

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-mi50-r3.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 51.245 · eligible10K 47.628 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3523,
    "content_chars": 5608,
    "content_has_think_close": false,
    "elapsed_seconds": 77.33335476461798,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T20:14:37Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.00762627087533474,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 5608,
    "qwen_answer_text_sha256": "8dd65b4456cd761c6f12ec9bc4fff7e98a8f640fc7a21baad81ed563a4a236ab",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 5608,
    "qwen_visible_text_sha256": "8dd65b4456cd761c6f12ec9bc4fff7e98a8f640fc7a21baad81ed563a4a236ab",
    "raw_has_think_close": false,
    "reasoning_chars": 11957,
    "text_chars": 17566,
    "text_sha256": "a0a97116032c440598376109954cb6d4cc4fbfb0762c8537e312d97fdbfecb05",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 69.49865089077502,
    "vllm_decode_tps": 50.69163148989456,
    "vllm_generation_tokens_delta": 3523.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 7.825316600501537,
    "vllm_prefill_tps": 60.44483873913608,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 45.556021857878385
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 47.061583733186126,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T20:15:24Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.006401632912456989,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9891,
    "text_chars": 9891,
    "text_sha256": "1f3ca45660e8651054584f47d321fb72ef4d84418ec484185e47892b95bedb23",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 39.02831846382469,
    "vllm_decode_tps": 51.24484166167185,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 8.024602881632745,
    "vllm_prefill_tps": 58.94372680829284,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 42.49750733717175
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 217.98573838174343,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T20:19:02Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.005619527772068977,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 18708,
    "text_chars": 18708,
    "text_sha256": "631eb28bb7d6d51e464222a4b4838025b684054be2c3a363f83eaa830161d1e0",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 209.96061483211815,
    "vllm_decode_tps": 47.6279801714044,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 8.016675692982972,
    "vllm_prefill_tps": 59.00201256912747,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 45.874560759051526
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS3,227 natural tokens2K 51.08310K 47.512 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T16:31:32.061488539Z",
      "FinishedAt": "2026-10-26T16:55:44.011700778Z"
    },
    "devices_after": [
      {
        "card": 1,
        "render": 129,
        "pci": "0000:06:00.0",
        "actual_card": "card0",
        "busy_percent": 0,
        "vram_bytes": 5398183936,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:0a:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 6673534976,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T16:55:44.148546+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 51.08344546346769,
  "retest_10K_TPS": 47.51204361074414,
  "retest_strict_tokens": 3227,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 3227,
      "elapsed_seconds": 76.7849499079166,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 63.73261856404133,
      "vllm_decode_tps": 50.63341304197895,
      "vllm_generation_tokens_delta": 3227.0,
      "vllm_prefill_seconds_delta": 13.041100402013399,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 54.03698424703907,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 39.151626947917975,
      "vllm_decode_tps": 51.08344546346769,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 14.875148234074004,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 224.74859275098424,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 210.47295043605845,
      "vllm_decode_tps": 47.51204361074414,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 14.263138746959157,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 14.875148234074004,
  "retest_2K_total_seconds": 54.03698424703907,
  "retest_10K_prefill_seconds": 14.263138746959157,
  "retest_10K_total_seconds": 224.74859275098424,
  "retest_strict_prefill_seconds": 13.041100402013399,
  "retest_strict_total_seconds": 76.7849499079166,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": "Complete record: chart-details/row-09-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 27801952256,
    "1": 27801944064
  },
  "retest_memory_observation_count": 24,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w4-tp2-mtp0/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 25.57,
  "retest_logged_model_load_seconds": 102.064312,
  "retest_logged_available_KV_GiB": null,
  "retest_explicit_KV_budget_bytes": 6442450944,
  "retest_KV_budget_GiB": 6.0,
  "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
  "retest_logged_GPU_KV_tokens": 48608,
  "retest_logged_context_capacity_x": 1.46,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-w4-tp2-mtp0",
    "image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
    "container": "03dcc2c0c19b25cf8daefae1396e31ccfe2cc837186d1972d193e2f122174499",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 25.57,
    "retest_logged_model_load_seconds": 102.064312,
    "retest_logged_available_KV_GiB": null,
    "retest_explicit_KV_budget_bytes": 6442450944,
    "retest_KV_budget_GiB": 6.0,
    "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
    "retest_logged_GPU_KV_tokens": 48608,
    "retest_logged_context_capacity_x": 1.46,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w4-tp2-mtp0/service-final.log",
    "source_log_sha256": "a408ac50b7c0f5e6e9df9755c37e8b000d217b6ce9b28a0465a7d8b9bf3436a3",
    "sealed_index_sha256": "9eb60e96cc59b2775938ceb3c9ca2747e620a5c010e06ff616f4da65f7050403",
    "source_lines": {
      "model_load": {
        "line": 118,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:35:01 [gpu_model_runner.py:4584] Model loading took 25.57 GiB memory and 102.064312 seconds"
      },
      "explicit_KV_budget": {
        "line": 7,
        "text": "(APIServer pid=1) INFO 10-26 16:32:03 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 4, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'optimization_level': '3'}"
      },
      "explicit_budget_worker_reports": [
        {
          "line": 125,
          "text": "(Worker pid=197) (Worker_TP1 pid=197) INFO 10-26 16:35:45 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        },
        {
          "line": 126,
          "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:35:45 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        }
      ],
      "grouped_KV_tokens": {
        "line": 127,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:35:45 [kv_cache_utils.py:1314] GPU KV cache size: 48,608 tokens"
      },
      "context_capacity": {
        "line": 128,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:35:45 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.46x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w4-tp2-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 25.89258575439453
}
Complete retest loaded admission
Retest KV budget: 6.000 GiB (explicit)48,608 grouped KV tokens · 1.46× reported capacityRetest sampled VRAM: 25.893 GiB max/cardRetest memory reports and limits
Evidence and limits

completed matrix cell

{
  "strict_tokens": 9449,
  "source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w4-tp2-mtpoff",
  "source_revision": "d72edcb54835d75267f4e852d0290847824c584e",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "sampled_VRAM_after_ladder_bytes_by_rank": "[19207479296,19207450624]",
  "comparison_limit": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
  "recorded_configuration_details": {
    "cell": "w4-tp2-mtpoff",
    "bits": 4,
    "group_size": 128,
    "TP": 2,
    "native_MTP": false,
    "num_speculative_tokens": 0,
    "strict_valid": true,
    "official_2k_eligible": true,
    "strict_returned_tokens": 9449,
    "strict_decode_TPS": 47.89995832486492,
    "strict_prefill_seconds": 14.604201089998242,
    "strict_total_seconds": 211.88102579300175,
    "fixed_2000_returned_tokens": 2000,
    "fixed_2000_decode_TPS": 51.18094703903334,
    "fixed_2000_prefill_seconds": 16.0112418270146,
    "fixed_2000_total_seconds": 55.10041077597998,
    "fixed_10000_returned_tokens": 10000,
    "fixed_10000_decode_TPS": 47.57945031867218,
    "fixed_10000_prefill_seconds": 15.533215646981262,
    "fixed_10000_total_seconds": 225.72210267998162,
    "packed_weight_bytes_per_rank": 6325437440,
    "packed_weight_bytes_total": 12650874880,
    "packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
    "render_nodes_by_rank": [
      "renderD129",
      "renderD130"
    ],
    "sampled_VRAM_before_ladder_bytes_by_rank": [
      19207479296,
      19207450624
    ],
    "sampled_VRAM_after_ladder_bytes_by_rank": [
      19207479296,
      19207450624
    ],
    "allocated_after_load_bytes_by_rank": [
      9030266368,
      9030266368
    ],
    "allocated_after_cache_bytes_by_rank": [
      15490159104,
      15490159104
    ],
    "reserved_after_cache_bytes_by_rank": [
      16882073600,
      16882073600
    ],
    "sampled_attention_KV_first_layer_views_by_rank": [
      {
        "rank": 0,
        "views": [
          {
            "shape": [
              250,
              2,
              784,
              2,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 401408000
          }
        ]
      },
      {
        "rank": 1,
        "views": [
          {
            "shape": [
              250,
              2,
              784,
              2,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 401408000
          }
        ]
      }
    ],
    "KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
    "KV_budget": "6442450944 bytes explicit",
    "context": 131072,
    "greedy": true,
    "concurrency": 1,
    "activation_dtype": "FP16",
    "attention_KV_dtype": "FP16",
    "SSM_dtype": "float32",
    "native_MTP_head_dtype": "not used",
    "native_MTP_unshared_parameter_bytes_by_rank": [],
    "model_repository": "Qwen/Qwen3.8-27B",
    "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "serving_source": "d72edcb54835d75267f4e852d0290847824c584e",
    "image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
    "native_MTP_model_source_sha256": "not used",
    "MTP_request_accounting_by_tier": {},
    "MTP_accounting_scope": "not used",
    "runtime_difference": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
    "MTP_scheduler_rollback_rejection_count": "not used",
    "host_exclusivity": "unproven",
    "serving_CPUs": "0-7,16-23",
    "cache_placement": "fresh private admitted rootFS",
    "compile_cache_disabled": true,
    "official_2k_samples": 1,
    "repeatability_established": false,
    "FP16_goal_qualifying_ladders": 0,
    "evidence": "../../tuning/qwen38-rtn-mi50/w4-tp2-result-20261022.json"
  },
  "detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
  "detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
  "detail_source_pointer": "w4-tp2-mtpoff",
  "retest_strict_tokens": 3227,
  "retest_MTP_accounting_by_tier": {},
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 27801952256,
    "1": 27801944064
  },
  "retest_memory_observation_count": 24,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w4-tp2-mtp0/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": null,
  "retest_explicit_KV_budget_bytes": 6442450944,
  "retest_KV_budget_GiB": 6.0,
  "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
  "retest_logged_GPU_KV_tokens": 48608,
  "retest_memory_log_report": {
    "profile": "mi50-w4-tp2-mtp0",
    "image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
    "container": "03dcc2c0c19b25cf8daefae1396e31ccfe2cc837186d1972d193e2f122174499",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 25.57,
    "retest_logged_model_load_seconds": 102.064312,
    "retest_logged_available_KV_GiB": null,
    "retest_explicit_KV_budget_bytes": 6442450944,
    "retest_KV_budget_GiB": 6.0,
    "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
    "retest_logged_GPU_KV_tokens": 48608,
    "retest_logged_context_capacity_x": 1.46,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w4-tp2-mtp0/service-final.log",
    "source_log_sha256": "a408ac50b7c0f5e6e9df9755c37e8b000d217b6ce9b28a0465a7d8b9bf3436a3",
    "sealed_index_sha256": "9eb60e96cc59b2775938ceb3c9ca2747e620a5c010e06ff616f4da65f7050403",
    "source_lines": {
      "model_load": {
        "line": 118,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:35:01 [gpu_model_runner.py:4584] Model loading took 25.57 GiB memory and 102.064312 seconds"
      },
      "explicit_KV_budget": {
        "line": 7,
        "text": "(APIServer pid=1) INFO 10-26 16:32:03 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 4, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'optimization_level': '3'}"
      },
      "explicit_budget_worker_reports": [
        {
          "line": 125,
          "text": "(Worker pid=197) (Worker_TP1 pid=197) INFO 10-26 16:35:45 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        },
        {
          "line": 126,
          "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:35:45 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        }
      ],
      "grouped_KV_tokens": {
        "line": 127,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:35:45 [kv_cache_utils.py:1314] GPU KV cache size: 48,608 tokens"
      },
      "context_capacity": {
        "line": 128,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:35:45 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.46x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w4-tp2-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 25.89258575439453,
  "sampled_peak_VRAM_GiB": 17.88835906982422
}
MI50Current Qwen3.84-bit RTNQwen/Qwen3.8-27BTP4MTP22K accepted 1243 / 1512 (82.21%) PASS5882 tokens 63.60239.246 22.624 / 54.08110K: 23.475 / 278.288Strict: 23.362 / 152.955 2.95 / 22.01packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:1ee2a65021b0a54e0f8784ea5e7487504cb48741de9af1bad226b5427c4fa96b
Runtime image config: sha256:b1450af604a4f69af53a3ac83f783d9572fcfafb3705c44d5730d50ba99baaad
Registry verificationDerived host-portability image; recorded benchmark parent: sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7
Exact compatibility change
Replaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately.
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp4-mtp2
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-w4-tp4-mtp2-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \
  --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \
  --user root \
  --workdir /tmp \
  -e HF_HUB_OFFLINE=1 \
  -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \
  -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \
  -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \
  -e MI50_RTN_BITS=4 \
  -e MI50_RTN_TP=4 \
  -e HIP_VISIBLE_DEVICES=0,1,2,3 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  -e MAX_JOBS=1 \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp2/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp2,dst=/runwork \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp2/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp2/cache/.triton,dst=/root/.triton \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp4-mtp2
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-mi50-r3.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 58.906 · eligible10K 41.409 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 4001,
    "content_chars": 6233,
    "content_has_think_close": false,
    "elapsed_seconds": 83.75218992121518,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T20:45:47Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.008191794157028198,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 6233,
    "qwen_answer_text_sha256": "554b14189ce88a7e754ef12db9cccfcc3806d2a4e6aec967f964cbfc932b3064",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 6233,
    "qwen_visible_text_sha256": "554b14189ce88a7e754ef12db9cccfcc3806d2a4e6aec967f964cbfc932b3064",
    "raw_has_think_close": false,
    "reasoning_chars": 13030,
    "text_chars": 19264,
    "text_sha256": "596f08acb3c9150c6168651db02f471064a7f17a4cc99fa1680722d02507b8a8",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 72.83459853101522,
    "vllm_decode_tps": 54.93268420085066,
    "vllm_generation_tokens_delta": 4001.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 10.909531355835497,
    "vllm_prefill_tps": 43.356582842304476,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 47.77188517415126
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 44.91906757093966,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T20:46:32Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.005639274604618549,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 8944,
    "text_chars": 8944,
    "text_sha256": "d8ddbdc35e64d91111afb2dcbcc23ad1d82db5e1f23d4c6783b5c3ae59526314",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 33.952197367325425,
    "vllm_decode_tps": 58.90634937003341,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 10.95891628973186,
    "vllm_prefill_tps": 43.161202028998545,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 44.5245217265796
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 252.5263803936541,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T20:50:45Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.005935841239988804,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 15159,
    "text_chars": 15159,
    "text_sha256": "7d0dad85f37cf14d6d03b0b1681550a739e02f5b7a62ace29c039b0137dac6ee",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 241.49360792525113,
    "vllm_decode_tps": 41.40896351631499,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 11.024237572215497,
    "vllm_prefill_tps": 42.90546143454917,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 39.59982313297869
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestMTP counter replayPASS7,228 natural tokens2K 52.13410K 33.819 · diagnostic2K MTP accepted 1107 / 1786 (61.98%)
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T17:46:35.884279265Z",
      "FinishedAt": "2026-10-26T18:15:55.378635525Z"
    },
    "devices_after": [
      {
        "card": 1,
        "render": 129,
        "pci": "0000:06:00.0",
        "actual_card": "card0",
        "busy_percent": 0,
        "vram_bytes": 17836716032,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:0a:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 17957421056,
        "dpm": "auto"
      },
      {
        "card": 3,
        "render": 131,
        "pci": "0000:0d:00.0",
        "actual_card": "card3",
        "busy_percent": 0,
        "vram_bytes": 17957421056,
        "dpm": "auto"
      },
      {
        "card": 4,
        "render": 132,
        "pci": "0000:10:00.0",
        "actual_card": "card4",
        "busy_percent": 0,
        "vram_bytes": 17907089408,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T18:15:55.526176+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 52.13382312317558,
  "retest_10K_TPS": 33.81858503497666,
  "retest_strict_tokens": 7228,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 7228,
      "elapsed_seconds": 201.84848926996347,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 177.56178416404873,
      "vllm_decode_tps": 40.70695749104478,
      "vllm_generation_tokens_delta": 7228.0,
      "vllm_prefill_seconds_delta": 24.27340279892087,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 59.321681477013044,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 38.36281093896832,
      "vllm_decode_tps": 52.13382312317558,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 20.947546551004052,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 318.76875816693064,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 295.69539913209155,
      "vllm_decode_tps": 33.81858503497666,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 23.061395875993185,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 20.947546551004052,
  "retest_2K_total_seconds": 59.321681477013044,
  "retest_10K_prefill_seconds": 23.061395875993185,
  "retest_10K_total_seconds": 318.76875816693064,
  "retest_strict_prefill_seconds": 24.27340279892087,
  "retest_strict_total_seconds": 201.84848926996347,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 4373.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:generation_tokens_total": 7228.0,
        "vllm:request_generation_tokens_sum": 7228.0,
        "vllm:spec_decode_num_accepted_tokens_total": 4373.0,
        "vllm:spec_decode_num_drafts_total": 2856.0,
        "vllm:spec_decode_num_draft_tokens_total": 5712.0
      },
      "accepted_per_position": {
        "0": 2402.0,
        "1": 1971.0
      },
      "accepted_fraction": 0.7655812324929971,
      "committed_emitted_tokens": 7228.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "1411456f9ac2de0cf4abed25becd1de45e5fcb81ad379267079da1f541901438",
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "1f754390fbaad7155236e4c71b33c173c3b158180135b13c88b7fe6ec6a3b504"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 1107.0,
        "vllm:spec_decode_num_drafts_total": 893.0,
        "vllm:spec_decode_num_draft_tokens_total": 1786.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1107.0
      },
      "accepted_per_position": {
        "0": 649.0,
        "1": 458.0
      },
      "accepted_fraction": 0.6198208286674132,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "f08b3a808c79b818598e5ffcea671b6909a8135f90f44bbe34f705ece68ef125",
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "9e4e97ae4f7e360b146b2d710dff319fd670df1e82436d97242a34bcd33a8b1c"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_accepted_tokens_total": 5990.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 5990.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 8022.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:spec_decode_num_drafts_total": 4011.0
      },
      "accepted_per_position": {
        "0": 3306.0,
        "1": 2684.0
      },
      "accepted_fraction": 0.7466965843929195,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "d96d24c021bd72fd6ce7030da03d9a08a49067fb54eacf17fbcc1063e7042ab0",
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "5eb6bf177d61d9131f899ccbf23344e6b01ad427f1062215d80cef034553981e"
      }
    }
  },
  "retest_MTP_2K_accepted": 1107.0,
  "retest_MTP_2K_proposed": 1786.0,
  "retest_MTP_2K_draft_batches": 893.0,
  "retest_MTP_2K_acceptance_percent": 61.982082866741315,
  "retest_loaded_admission": "Complete record: chart-details/row-10-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 23613620224,
    "1": 23630397440,
    "2": 23630405632,
    "3": 23580045312
  },
  "retest_memory_observation_count": 29,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w4-tp4-mtp2/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 14.35,
  "retest_logged_model_load_seconds": 55.690794,
  "retest_logged_available_KV_GiB": 14.41,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 14.41,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 221600,
  "retest_logged_context_capacity_x": 6.42,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-w4-tp4-mtp2",
    "image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
    "container": "4d74b43dece8d4d95c3aa6b916fe0d831886909ec0266e72762306371a15892c",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 14.35,
    "retest_logged_model_load_seconds": 55.690794,
    "retest_logged_available_KV_GiB": 14.41,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 14.41,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 221600,
    "retest_logged_context_capacity_x": 6.42,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w4-tp4-mtp2/service-final.log",
    "source_log_sha256": "e464acca61263b85d9bb1d2e0d4fc1b70c5fe21e6d87bd4397926c2df8ec1b38",
    "sealed_index_sha256": "9151a652ad0691e375d482b3671dc605a5eef3a2700cb1f444c23f037089644d",
    "source_lines": {
      "model_load": {
        "line": 178,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 17:49:39 [gpu_model_runner.py:4584] Model loading took 14.35 GiB memory and 55.690794 seconds"
      },
      "automatic_KV_budget": {
        "line": 265,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 17:54:52 [gpu_worker.py:453] Available KV cache memory: 14.41 GiB"
      },
      "grouped_KV_tokens": {
        "line": 269,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 17:54:52 [kv_cache_utils.py:1314] GPU KV cache size: 221,600 tokens"
      },
      "context_capacity": {
        "line": 270,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 17:54:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.42x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w4-tp4-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 22.007530212402344
}
Complete retest loaded admission
Retest KV budget: 14.410 GiB (automatic, rounded)221,600 grouped KV tokens · 6.42× reported capacityRetest sampled VRAM: 22.008 GiB max/cardRetest memory reports and limits
Evidence and limits

completed matrix cell

{
  "strict_tokens": 5882,
  "source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w4-tp4-mtp2",
  "source_revision": "016143b00c7a08e25ff85eb83dc052be148215ab",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "sampled_VRAM_after_ladder_bytes_by_rank": "[23613669376,23630442496,23630434304,23580102656]",
  "comparison_limit": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
  "recorded_configuration_details": {
    "cell": "w4-tp4-mtp2",
    "bits": 4,
    "group_size": 128,
    "TP": 4,
    "native_MTP": true,
    "num_speculative_tokens": 2,
    "strict_valid": true,
    "official_2k_eligible": true,
    "strict_returned_tokens": 5882,
    "strict_decode_TPS": 45.392808024901534,
    "strict_prefill_seconds": 23.36222447999171,
    "strict_total_seconds": 152.95514036499662,
    "fixed_2000_returned_tokens": 2000,
    "fixed_2000_decode_TPS": 63.60194464868311,
    "fixed_2000_prefill_seconds": 22.624249940010486,
    "fixed_2000_total_seconds": 54.08142457698705,
    "fixed_10000_returned_tokens": 10000,
    "fixed_10000_decode_TPS": 39.246321043123466,
    "fixed_10000_prefill_seconds": 23.475374940986512,
    "fixed_10000_total_seconds": 278.28769938999903,
    "packed_weight_bytes_per_rank": 3162718720,
    "packed_weight_bytes_total": 12650874880,
    "packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
    "render_nodes_by_rank": [
      "renderD129",
      "renderD130",
      "renderD131",
      "renderD132"
    ],
    "sampled_VRAM_before_ladder_bytes_by_rank": [
      23613669376,
      23630442496,
      23630434304,
      23580102656
    ],
    "sampled_VRAM_after_ladder_bytes_by_rank": [
      23613669376,
      23630442496,
      23630434304,
      23580102656
    ],
    "allocated_after_load_bytes_by_rank": [
      4805086208,
      4805086208,
      4805086208,
      4805086208
    ],
    "allocated_after_cache_bytes_by_rank": [
      20347904000,
      20347904000,
      20347904000,
      20347904000
    ],
    "reserved_after_cache_bytes_by_rank": [
      21648900096,
      21648900096,
      21648900096,
      21648900096
    ],
    "sampled_attention_KV_first_layer_views_by_rank": [
      {
        "rank": 0,
        "views": [
          {
            "shape": [
              1110,
              2,
              800,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 909312000
          }
        ]
      },
      {
        "rank": 1,
        "views": [
          {
            "shape": [
              1110,
              2,
              800,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 909312000
          }
        ]
      },
      {
        "rank": 2,
        "views": [
          {
            "shape": [
              1110,
              2,
              800,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 909312000
          }
        ]
      },
      {
        "rank": 3,
        "views": [
          {
            "shape": [
              1110,
              2,
              800,
              1,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 909312000
          }
        ]
      }
    ],
    "KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
    "KV_budget": "automatic; actual allocation receipts retained",
    "context": 131072,
    "greedy": true,
    "concurrency": 1,
    "activation_dtype": "FP16",
    "attention_KV_dtype": "FP16",
    "SSM_dtype": "float32",
    "native_MTP_head_dtype": "FP16",
    "native_MTP_unshared_parameter_bytes_by_rank": [
      212388864,
      212388864,
      212388864,
      212388864
    ],
    "model_repository": "Qwen/Qwen3.8-27B",
    "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "serving_source": "016143b00c7a08e25ff85eb83dc052be148215ab",
    "image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
    "native_MTP_model_source_sha256": "46f57b9c511c9199aa83421fa7fb167b08d5d80c14377948690cdf67d75453c3",
    "MTP_request_accounting_by_tier": {
      "c1_128_strict": {
        "drafts": 2286.0,
        "proposed": 4572.0,
        "accepted": 3595.0,
        "adjusted_unaccepted": 977.0,
        "generation": 5882.0,
        "prompt": 473.0,
        "histogram_count": 2287.0,
        "histogram_sum": 6355.0,
        "means": {
          "prompt_plus_generation_per_observed_iteration": 2.7787494534324444,
          "generation_per_all_observed_iterations": 2.5719282903366856,
          "sum_equals_prompt_plus_generation": true
        }
      },
      "c1_2000": {
        "drafts": 756.0,
        "proposed": 1512.0,
        "accepted": 1243.0,
        "adjusted_unaccepted": 269.0,
        "generation": 2000.0,
        "prompt": 473.0,
        "histogram_count": 757.0,
        "histogram_sum": 2473.0,
        "means": {
          "prompt_plus_generation_per_observed_iteration": 3.2668428005284014,
          "generation_per_all_observed_iterations": 2.642007926023778,
          "sum_equals_prompt_plus_generation": true
        }
      },
      "c1_10000": {
        "drafts": 3433.0,
        "proposed": 6866.0,
        "accepted": 6567.0,
        "adjusted_unaccepted": 299.0,
        "generation": 10000.0,
        "prompt": 473.0,
        "histogram_count": 3434.0,
        "histogram_sum": 10473.0,
        "means": {
          "prompt_plus_generation_per_observed_iteration": 3.049796156086197,
          "generation_per_all_observed_iterations": 2.9120559114735003,
          "sum_equals_prompt_plus_generation": true
        }
      }
    },
    "MTP_accounting_scope": "P-A is adjusted unaccepted, not scheduler rollback rejection; iteration means include prefill, not exact decode-only emissions.",
    "runtime_difference": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
    "MTP_scheduler_rollback_rejection_count": "unobservable",
    "host_exclusivity": "unproven",
    "serving_CPUs": "0-7,16-23",
    "cache_placement": "fresh private admitted rootFS",
    "compile_cache_disabled": true,
    "official_2k_samples": 1,
    "repeatability_established": false,
    "FP16_goal_qualifying_ladders": 0,
    "evidence": "../../tuning/qwen38-rtn-mi50/w4-tp4-mtp-result-20261022.json"
  },
  "MTP_accounting_by_tier": {
    "c1_128_strict": {
      "drafts": 2286.0,
      "proposed": 4572.0,
      "accepted": 3595.0,
      "adjusted_unaccepted": 977.0,
      "generation": 5882.0,
      "prompt": 473.0,
      "histogram_count": 2287.0,
      "histogram_sum": 6355.0,
      "means": {
        "prompt_plus_generation_per_observed_iteration": 2.7787494534324444,
        "generation_per_all_observed_iterations": 2.5719282903366856,
        "sum_equals_prompt_plus_generation": true
      }
    },
    "c1_2000": {
      "drafts": 756.0,
      "proposed": 1512.0,
      "accepted": 1243.0,
      "adjusted_unaccepted": 269.0,
      "generation": 2000.0,
      "prompt": 473.0,
      "histogram_count": 757.0,
      "histogram_sum": 2473.0,
      "means": {
        "prompt_plus_generation_per_observed_iteration": 3.2668428005284014,
        "generation_per_all_observed_iterations": 2.642007926023778,
        "sum_equals_prompt_plus_generation": true
      }
    },
    "c1_10000": {
      "drafts": 3433.0,
      "proposed": 6866.0,
      "accepted": 6567.0,
      "adjusted_unaccepted": 299.0,
      "generation": 10000.0,
      "prompt": 473.0,
      "histogram_count": 3434.0,
      "histogram_sum": 10473.0,
      "means": {
        "prompt_plus_generation_per_observed_iteration": 3.049796156086197,
        "generation_per_all_observed_iterations": 2.9120559114735003,
        "sum_equals_prompt_plus_generation": true
      }
    }
  },
  "MTP_2K_draft_batches": 756.0,
  "MTP_2K_proposed": 1512.0,
  "MTP_2K_accepted": 1243.0,
  "MTP_2K_acceptance_percent": 82.20899470899471,
  "MTP_2K_committed": 2000.0,
  "detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
  "detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
  "detail_source_pointer": "w4-tp4-mtp2",
  "retest_strict_tokens": 7228,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 4373.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:generation_tokens_total": 7228.0,
        "vllm:request_generation_tokens_sum": 7228.0,
        "vllm:spec_decode_num_accepted_tokens_total": 4373.0,
        "vllm:spec_decode_num_drafts_total": 2856.0,
        "vllm:spec_decode_num_draft_tokens_total": 5712.0
      },
      "accepted_per_position": {
        "0": 2402.0,
        "1": 1971.0
      },
      "accepted_fraction": 0.7655812324929971,
      "committed_emitted_tokens": 7228.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "1411456f9ac2de0cf4abed25becd1de45e5fcb81ad379267079da1f541901438",
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "1f754390fbaad7155236e4c71b33c173c3b158180135b13c88b7fe6ec6a3b504"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 1107.0,
        "vllm:spec_decode_num_drafts_total": 893.0,
        "vllm:spec_decode_num_draft_tokens_total": 1786.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1107.0
      },
      "accepted_per_position": {
        "0": 649.0,
        "1": 458.0
      },
      "accepted_fraction": 0.6198208286674132,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "f08b3a808c79b818598e5ffcea671b6909a8135f90f44bbe34f705ece68ef125",
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "9e4e97ae4f7e360b146b2d710dff319fd670df1e82436d97242a34bcd33a8b1c"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_accepted_tokens_total": 5990.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 5990.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 8022.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:spec_decode_num_drafts_total": 4011.0
      },
      "accepted_per_position": {
        "0": 3306.0,
        "1": 2684.0
      },
      "accepted_fraction": 0.7466965843929195,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "d96d24c021bd72fd6ce7030da03d9a08a49067fb54eacf17fbcc1063e7042ab0",
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "5eb6bf177d61d9131f899ccbf23344e6b01ad427f1062215d80cef034553981e"
      }
    }
  },
  "retest_MTP_2K_accepted": 1107.0,
  "retest_MTP_2K_proposed": 1786.0,
  "retest_MTP_2K_draft_batches": 893.0,
  "retest_MTP_2K_acceptance_percent": 61.982082866741315,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 23613620224,
    "1": 23630397440,
    "2": 23630405632,
    "3": 23580045312
  },
  "retest_memory_observation_count": 29,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w4-tp4-mtp2/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 14.41,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 14.41,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 221600,
  "retest_memory_log_report": {
    "profile": "mi50-w4-tp4-mtp2",
    "image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
    "container": "4d74b43dece8d4d95c3aa6b916fe0d831886909ec0266e72762306371a15892c",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 14.35,
    "retest_logged_model_load_seconds": 55.690794,
    "retest_logged_available_KV_GiB": 14.41,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 14.41,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 221600,
    "retest_logged_context_capacity_x": 6.42,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w4-tp4-mtp2/service-final.log",
    "source_log_sha256": "e464acca61263b85d9bb1d2e0d4fc1b70c5fe21e6d87bd4397926c2df8ec1b38",
    "sealed_index_sha256": "9151a652ad0691e375d482b3671dc605a5eef3a2700cb1f444c23f037089644d",
    "source_lines": {
      "model_load": {
        "line": 178,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 17:49:39 [gpu_model_runner.py:4584] Model loading took 14.35 GiB memory and 55.690794 seconds"
      },
      "automatic_KV_budget": {
        "line": 265,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 17:54:52 [gpu_worker.py:453] Available KV cache memory: 14.41 GiB"
      },
      "grouped_KV_tokens": {
        "line": 269,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 17:54:52 [kv_cache_utils.py:1314] GPU KV cache size: 221,600 tokens"
      },
      "context_capacity": {
        "line": 270,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 17:54:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.42x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w4-tp4-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 22.007530212402344,
  "sampled_peak_VRAM_GiB": 22.007564544677734
}
MI50Current Qwen3.84-bit RTNQwen/Qwen3.8-27BTP2MTP22K accepted 1092 / 1816 (60.13%) PASS13252 tokens 46.96632.753 15.057 / 57.65310K: 15.036 / 320.362Strict: 15.042 / 472.320 5.89 / 18.28packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:1ee2a65021b0a54e0f8784ea5e7487504cb48741de9af1bad226b5427c4fa96b
Runtime image config: sha256:b1450af604a4f69af53a3ac83f783d9572fcfafb3705c44d5730d50ba99baaad
Registry verificationDerived host-portability image; recorded benchmark parent: sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7
Exact compatibility change
Replaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately.
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp2-mtp2
GPU_RENDER_NODES=renderD128,renderD129

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-w4-tp2-mtp2-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \
  -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \
  -e MI50_RTN_BITS=4 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e HF_HUB_OFFLINE=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \
  -e MI50_RTN_TP=2 \
  -e HIP_VISIBLE_DEVICES=0,1 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  -e MAX_JOBS=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp2/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp2/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp2/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp2,dst=/runwork \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-mi50-r3.tar.gz.sha256
mkdir -p launch-tools-mi50-r3
tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp2-mtp2
GPU_RENDER_NODES=renderD128,renderD129

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-mi50-r3.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 49.476 · eligible10K 38.086 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 9027,
    "content_chars": 6853,
    "content_has_think_close": false,
    "elapsed_seconds": 263.3950856635347,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T21:22:05Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.008886664174497128,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 6853,
    "qwen_answer_text_sha256": "7b8ffe2141548b492b29edb8f8d019c0adb81ca7b4dc50bcdf65fd6064a2378f",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 6853,
    "qwen_visible_text_sha256": "7b8ffe2141548b492b29edb8f8d019c0adb81ca7b4dc50bcdf65fd6064a2378f",
    "raw_has_think_close": false,
    "reasoning_chars": 33708,
    "text_chars": 40562,
    "text_sha256": "4c56847c8958e8315b5f613ed3fe5cd07c7e1a6f515ca50e3e6120e539871a60",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 252.45595039613545,
    "vllm_decode_tps": 35.75673295018593,
    "vllm_generation_tokens_delta": 9027.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 10.92989128921181,
    "vllm_prefill_tps": 43.27581926335057,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 34.27171003308407
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 51.36650285869837,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T21:22:57Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.00833950936794281,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 8988,
    "text_chars": 8988,
    "text_sha256": "4f354560f326ce7fa34f1b9058c95277a45fee0768e60c45fdbb2871fad810ce",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 40.42383814975619,
    "vllm_decode_tps": 49.47575716562834,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 10.934961513616145,
    "vllm_prefill_tps": 43.25575352149374,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 38.93588016886615
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 273.5147092798725,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T21:27:30Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.007087661884725094,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 14636,
    "text_chars": 14636,
    "text_sha256": "81403098ce625505c2198d17f999d526fbd010f3a8de362f435c4e4952fd0add",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 262.5658295126632,
    "vllm_decode_tps": 38.08568700108677,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 10.941470579244196,
    "vllm_prefill_tps": 43.230020733892374,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 36.56110498162478
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestMTP counter replayPASS14,111 natural tokens2K 47.78710K 37.319 · diagnostic2K MTP accepted 1109 / 1784 (62.16%)
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T18:18:15.13601314Z",
      "FinishedAt": "2026-10-26T18:56:11.307013625Z"
    },
    "devices_after": [
      {
        "card": 1,
        "render": 129,
        "pci": "0000:06:00.0",
        "actual_card": "card0",
        "busy_percent": 0,
        "vram_bytes": 12862812160,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:0a:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 12777545728,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T18:56:11.450407+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 47.78708170874797,
  "retest_10K_TPS": 37.319231865004355,
  "retest_strict_tokens": 14111,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 14111,
      "elapsed_seconds": 516.6818805149524,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 501.4409740610281,
      "vllm_decode_tps": 28.140899387856194,
      "vllm_generation_tokens_delta": 14111.0,
      "vllm_prefill_seconds_delta": 15.226414316915907,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 57.152446987107396,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 41.8523150710389,
      "vllm_decode_tps": 47.78708170874797,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 15.288337494013831,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 283.10880334395915,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 267.95835552492645,
      "vllm_decode_tps": 37.319231865004355,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 15.136290078051388,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 15.288337494013831,
  "retest_2K_total_seconds": 57.152446987107396,
  "retest_10K_prefill_seconds": 15.136290078051388,
  "retest_10K_total_seconds": 283.10880334395915,
  "retest_strict_prefill_seconds": 15.226414316915907,
  "retest_strict_total_seconds": 516.6818805149524,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_sum": 14111.0,
        "vllm:spec_decode_num_accepted_tokens_total": 8690.0,
        "vllm:spec_decode_num_draft_tokens_total": 10842.0,
        "vllm:spec_decode_num_drafts_total": 5421.0,
        "vllm:generation_tokens_total": 14111.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 8690.0
      },
      "accepted_per_position": {
        "0": 4679.0,
        "1": 4011.0
      },
      "accepted_fraction": 0.8015126360450101,
      "committed_emitted_tokens": 14111.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "ff1459ea939cf973977e7de0e4877ef1a7f3b7ed5a7381611bb0583fbd95b814",
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "f4f791c56c4a24e726d34177403687315b177e29ec4638a4e3c350792aa44b99"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 1784.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1109.0,
        "vllm:spec_decode_num_drafts_total": 892.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1109.0
      },
      "accepted_per_position": {
        "0": 660.0,
        "1": 449.0
      },
      "accepted_fraction": 0.6216367713004485,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "6147a60b6fc55ffe8bd73699a6e21cdb051e35aa4381a03802c8ea7cfb9f84b9",
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "ec311d29ffa90c0544ecfc89a16dc2853c2850a5cb19fa86fdb0c4c4b7dacf7b"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 6880.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6561.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_drafts_total": 3440.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6561.0
      },
      "accepted_per_position": {
        "0": 3329.0,
        "1": 3232.0
      },
      "accepted_fraction": 0.9536337209302326,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "b7ec2069b7cb5c29289e1dc5bf3d7d6e4d61f7ab90f67de5c7bb74de89fc14ea",
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "4669652226a155c0001a247d6d68276dff4f6419210d5cd06f367b4ddf6bbc4b"
      }
    }
  },
  "retest_MTP_2K_accepted": 1109.0,
  "retest_MTP_2K_proposed": 1784.0,
  "retest_MTP_2K_draft_batches": 892.0,
  "retest_MTP_2K_acceptance_percent": 62.163677130044846,
  "retest_loaded_admission": "Complete record: chart-details/row-11-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30614081536,
    "1": 30599421952
  },
  "retest_memory_observation_count": 36,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w4-tp2-mtp2/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 27.94,
  "retest_logged_model_load_seconds": 175.217768,
  "retest_logged_available_KV_GiB": null,
  "retest_explicit_KV_budget_bytes": 6442450944,
  "retest_KV_budget_GiB": 6.0,
  "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
  "retest_logged_GPU_KV_tokens": 45600,
  "retest_logged_context_capacity_x": 1.34,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-w4-tp2-mtp2",
    "image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
    "container": "e2fbe3010395831aa4c2972d9c94dbc1ccae461c86819aa28afb76f3f9e3ebd6",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 27.94,
    "retest_logged_model_load_seconds": 175.217768,
    "retest_logged_available_KV_GiB": null,
    "retest_explicit_KV_budget_bytes": 6442450944,
    "retest_KV_budget_GiB": 6.0,
    "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
    "retest_logged_GPU_KV_tokens": 45600,
    "retest_logged_context_capacity_x": 1.34,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w4-tp2-mtp2/service-final.log",
    "source_log_sha256": "59412ba4a8cd81636f00030fe647d14f836027e24663bf27f338e3a657c67e2f",
    "sealed_index_sha256": "a258b5d8f94c32526dc0aa359b7bde709b0e9177620b565088dad25c4b98c5a7",
    "source_lines": {
      "model_load": {
        "line": 144,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 18:24:01 [gpu_model_runner.py:4584] Model loading took 27.94 GiB memory and 175.217768 seconds"
      },
      "explicit_KV_budget": {
        "line": 7,
        "text": "(APIServer pid=1) INFO 10-26 18:18:53 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 4, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 2}, 'optimization_level': '3'}"
      },
      "explicit_budget_worker_reports": [
        {
          "line": 157,
          "text": "(Worker pid=217) (Worker_TP1 pid=217) INFO 10-26 18:25:14 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        },
        {
          "line": 158,
          "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 18:25:14 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        }
      ],
      "grouped_KV_tokens": {
        "line": 160,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 18:25:14 [kv_cache_utils.py:1314] GPU KV cache size: 45,600 tokens"
      },
      "context_capacity": {
        "line": 161,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 18:25:14 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.34x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w4-tp2-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.511585235595703
}
Complete retest loaded admission
Retest KV budget: 6.000 GiB (explicit)45,600 grouped KV tokens · 1.34× reported capacityRetest sampled VRAM: 28.512 GiB max/cardRetest memory reports and limits
Evidence and limits

completed matrix cell

{
  "strict_tokens": 13252,
  "source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w4-tp2-mtp2",
  "source_revision": "016143b00c7a08e25ff85eb83dc052be148215ab",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "sampled_VRAM_after_ladder_bytes_by_rank": "[19633012736,19633037312]",
  "comparison_limit": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
  "recorded_configuration_details": {
    "cell": "w4-tp2-mtp2",
    "bits": 4,
    "group_size": 128,
    "TP": 2,
    "native_MTP": true,
    "num_speculative_tokens": 2,
    "strict_valid": true,
    "official_2k_eligible": true,
    "strict_returned_tokens": 13252,
    "strict_decode_TPS": 28.981154832388903,
    "strict_prefill_seconds": 15.042166554019786,
    "strict_total_seconds": 472.3203631230281,
    "fixed_2000_returned_tokens": 2000,
    "fixed_2000_decode_TPS": 46.966211495697614,
    "fixed_2000_prefill_seconds": 15.056974492967129,
    "fixed_2000_total_seconds": 57.652922383043915,
    "fixed_10000_returned_tokens": 10000,
    "fixed_10000_decode_TPS": 32.75325418710531,
    "fixed_10000_prefill_seconds": 15.036388729000464,
    "fixed_10000_total_seconds": 320.3623473200132,
    "packed_weight_bytes_per_rank": 6325437440,
    "packed_weight_bytes_total": 12650874880,
    "packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
    "render_nodes_by_rank": [
      "renderD129",
      "renderD130"
    ],
    "sampled_VRAM_before_ladder_bytes_by_rank": [
      19633012736,
      19633037312
    ],
    "sampled_VRAM_after_ladder_bytes_by_rank": [
      19633012736,
      19633037312
    ],
    "allocated_after_load_bytes_by_rank": [
      9456126976,
      9456126976
    ],
    "allocated_after_cache_bytes_by_rank": [
      15928451072,
      15928451072
    ],
    "reserved_after_cache_bytes_by_rank": [
      17576230912,
      17576230912
    ],
    "sampled_attention_KV_first_layer_views_by_rank": [
      {
        "rank": 0,
        "views": [
          {
            "shape": [
              231,
              2,
              800,
              2,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 378470400
          }
        ]
      },
      {
        "rank": 1,
        "views": [
          {
            "shape": [
              231,
              2,
              800,
              2,
              256
            ],
            "dtype": "torch.float16",
            "logical_bytes": 378470400
          }
        ]
      }
    ],
    "KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
    "KV_budget": "6442450944 bytes explicit",
    "context": 131072,
    "greedy": true,
    "concurrency": 1,
    "activation_dtype": "FP16",
    "attention_KV_dtype": "FP16",
    "SSM_dtype": "float32",
    "native_MTP_head_dtype": "FP16",
    "native_MTP_unshared_parameter_bytes_by_rank": [
      424725504,
      424725504
    ],
    "model_repository": "Qwen/Qwen3.8-27B",
    "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "serving_source": "016143b00c7a08e25ff85eb83dc052be148215ab",
    "image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
    "native_MTP_model_source_sha256": "46f57b9c511c9199aa83421fa7fb167b08d5d80c14377948690cdf67d75453c3",
    "MTP_request_accounting_by_tier": {
      "c1_128_strict": {
        "drafts": 5135.0,
        "proposed": 10270.0,
        "accepted": 8117.0,
        "adjusted_unaccepted": 2153.0,
        "generation": 13252.0,
        "prompt": 473.0,
        "histogram_count": 5136.0,
        "histogram_sum": 13725.0,
        "means": {
          "prompt_plus_generation_per_observed_iteration": 2.6723130841121496,
          "generation_per_all_observed_iterations": 2.5802180685358254,
          "sum_equals_prompt_plus_generation": true
        }
      },
      "c1_2000": {
        "drafts": 908.0,
        "proposed": 1816.0,
        "accepted": 1092.0,
        "adjusted_unaccepted": 724.0,
        "generation": 2000.0,
        "prompt": 473.0,
        "histogram_count": 909.0,
        "histogram_sum": 2473.0,
        "means": {
          "prompt_plus_generation_per_observed_iteration": 2.7205720572057204,
          "generation_per_all_observed_iterations": 2.2002200220022003,
          "sum_equals_prompt_plus_generation": true
        }
      },
      "c1_10000": {
        "drafts": 3808.0,
        "proposed": 7616.0,
        "accepted": 6191.0,
        "adjusted_unaccepted": 1425.0,
        "generation": 10000.0,
        "prompt": 473.0,
        "histogram_count": 3809.0,
        "histogram_sum": 10473.0,
        "means": {
          "prompt_plus_generation_per_observed_iteration": 2.749540561827251,
          "generation_per_all_observed_iterations": 2.625360987135731,
          "sum_equals_prompt_plus_generation": true
        }
      }
    },
    "MTP_accounting_scope": "P-A is adjusted unaccepted, not scheduler rollback rejection; iteration means include prefill, not exact decode-only emissions.",
    "runtime_difference": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
    "MTP_scheduler_rollback_rejection_count": "unobservable",
    "host_exclusivity": "unproven",
    "serving_CPUs": "0-7,16-23",
    "cache_placement": "fresh private admitted rootFS",
    "compile_cache_disabled": true,
    "official_2k_samples": 1,
    "repeatability_established": false,
    "FP16_goal_qualifying_ladders": 0,
    "evidence": "../../tuning/qwen38-rtn-mi50/w4-tp2-mtp-result-20261023.json"
  },
  "MTP_accounting_by_tier": {
    "c1_128_strict": {
      "drafts": 5135.0,
      "proposed": 10270.0,
      "accepted": 8117.0,
      "adjusted_unaccepted": 2153.0,
      "generation": 13252.0,
      "prompt": 473.0,
      "histogram_count": 5136.0,
      "histogram_sum": 13725.0,
      "means": {
        "prompt_plus_generation_per_observed_iteration": 2.6723130841121496,
        "generation_per_all_observed_iterations": 2.5802180685358254,
        "sum_equals_prompt_plus_generation": true
      }
    },
    "c1_2000": {
      "drafts": 908.0,
      "proposed": 1816.0,
      "accepted": 1092.0,
      "adjusted_unaccepted": 724.0,
      "generation": 2000.0,
      "prompt": 473.0,
      "histogram_count": 909.0,
      "histogram_sum": 2473.0,
      "means": {
        "prompt_plus_generation_per_observed_iteration": 2.7205720572057204,
        "generation_per_all_observed_iterations": 2.2002200220022003,
        "sum_equals_prompt_plus_generation": true
      }
    },
    "c1_10000": {
      "drafts": 3808.0,
      "proposed": 7616.0,
      "accepted": 6191.0,
      "adjusted_unaccepted": 1425.0,
      "generation": 10000.0,
      "prompt": 473.0,
      "histogram_count": 3809.0,
      "histogram_sum": 10473.0,
      "means": {
        "prompt_plus_generation_per_observed_iteration": 2.749540561827251,
        "generation_per_all_observed_iterations": 2.625360987135731,
        "sum_equals_prompt_plus_generation": true
      }
    }
  },
  "MTP_2K_draft_batches": 908.0,
  "MTP_2K_proposed": 1816.0,
  "MTP_2K_accepted": 1092.0,
  "MTP_2K_acceptance_percent": 60.13215859030837,
  "MTP_2K_committed": 2000.0,
  "detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
  "detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
  "detail_source_pointer": "w4-tp2-mtp2",
  "retest_strict_tokens": 14111,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_sum": 14111.0,
        "vllm:spec_decode_num_accepted_tokens_total": 8690.0,
        "vllm:spec_decode_num_draft_tokens_total": 10842.0,
        "vllm:spec_decode_num_drafts_total": 5421.0,
        "vllm:generation_tokens_total": 14111.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 8690.0
      },
      "accepted_per_position": {
        "0": 4679.0,
        "1": 4011.0
      },
      "accepted_fraction": 0.8015126360450101,
      "committed_emitted_tokens": 14111.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "ff1459ea939cf973977e7de0e4877ef1a7f3b7ed5a7381611bb0583fbd95b814",
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "f4f791c56c4a24e726d34177403687315b177e29ec4638a4e3c350792aa44b99"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 1784.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1109.0,
        "vllm:spec_decode_num_drafts_total": 892.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1109.0
      },
      "accepted_per_position": {
        "0": 660.0,
        "1": 449.0
      },
      "accepted_fraction": 0.6216367713004485,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "6147a60b6fc55ffe8bd73699a6e21cdb051e35aa4381a03802c8ea7cfb9f84b9",
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "ec311d29ffa90c0544ecfc89a16dc2853c2850a5cb19fa86fdb0c4c4b7dacf7b"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 6880.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6561.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_drafts_total": 3440.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6561.0
      },
      "accepted_per_position": {
        "0": 3329.0,
        "1": 3232.0
      },
      "accepted_fraction": 0.9536337209302326,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "b7ec2069b7cb5c29289e1dc5bf3d7d6e4d61f7ab90f67de5c7bb74de89fc14ea",
        "/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "4669652226a155c0001a247d6d68276dff4f6419210d5cd06f367b4ddf6bbc4b"
      }
    }
  },
  "retest_MTP_2K_accepted": 1109.0,
  "retest_MTP_2K_proposed": 1784.0,
  "retest_MTP_2K_draft_batches": 892.0,
  "retest_MTP_2K_acceptance_percent": 62.163677130044846,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30614081536,
    "1": 30599421952
  },
  "retest_memory_observation_count": 36,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w4-tp2-mtp2/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": null,
  "retest_explicit_KV_budget_bytes": 6442450944,
  "retest_KV_budget_GiB": 6.0,
  "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
  "retest_logged_GPU_KV_tokens": 45600,
  "retest_memory_log_report": {
    "profile": "mi50-w4-tp2-mtp2",
    "image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
    "container": "e2fbe3010395831aa4c2972d9c94dbc1ccae461c86819aa28afb76f3f9e3ebd6",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 27.94,
    "retest_logged_model_load_seconds": 175.217768,
    "retest_logged_available_KV_GiB": null,
    "retest_explicit_KV_budget_bytes": 6442450944,
    "retest_KV_budget_GiB": 6.0,
    "retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
    "retest_logged_GPU_KV_tokens": 45600,
    "retest_logged_context_capacity_x": 1.34,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w4-tp2-mtp2/service-final.log",
    "source_log_sha256": "59412ba4a8cd81636f00030fe647d14f836027e24663bf27f338e3a657c67e2f",
    "sealed_index_sha256": "a258b5d8f94c32526dc0aa359b7bde709b0e9177620b565088dad25c4b98c5a7",
    "source_lines": {
      "model_load": {
        "line": 144,
        "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 18:24:01 [gpu_model_runner.py:4584] Model loading took 27.94 GiB memory and 175.217768 seconds"
      },
      "explicit_KV_budget": {
        "line": 7,
        "text": "(APIServer pid=1) INFO 10-26 18:18:53 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 4, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 2}, 'optimization_level': '3'}"
      },
      "explicit_budget_worker_reports": [
        {
          "line": 157,
          "text": "(Worker pid=217) (Worker_TP1 pid=217) INFO 10-26 18:25:14 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        },
        {
          "line": 158,
          "text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 18:25:14 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
        }
      ],
      "grouped_KV_tokens": {
        "line": 160,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 18:25:14 [kv_cache_utils.py:1314] GPU KV cache size: 45,600 tokens"
      },
      "context_capacity": {
        "line": 161,
        "text": "(EngineCore_DP0 pid=198) INFO 10-26 18:25:14 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.34x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w4-tp2-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.511585235595703,
  "sampled_peak_VRAM_GiB": 18.284690856933594
}
V620Current Qwen3.8FP16Qwen/Qwen3.8-27BTP4MTP off PASS9102 tokens 33.45132.283 3.692 / 63.49410K: 3.717 / 313.490Strict: — / — 1.19 / 28.88retained FP16 parameter extents; projection buffers separate / VRAM GiBSeparate packed-buffer extents: 17.509 GiB/rank; includes alternate views, not additive physical storage. Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:7045fec47d4265247bd61a45ec50e73d13d2def440121f047e9aaa4b530e24d4
Runtime image config: sha256:71f0e460e6327c9b3c5edcc90ff104e43ac14089883164c8bad20e9b141d1aeb
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256
mkdir -p launch-tools-v620-serve-r5
tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp4-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-fp16-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Preflight cleanup compatibility
After its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged.
Retained lab Docker command · not portable
docker run -d \
  --name serve-v620-fp16-tp4-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --cpuset-mems 0 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/card5:/dev/dri/card5:rwm \
  --device /dev/dri/card6:/dev/dri/card6:rwm \
  --device /dev/dri/card7:/dev/dri/card7:rwm \
  --device /dev/dri/card8:/dev/dri/card8:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
  --device /dev/dri/renderD136:/dev/dri/renderD136:rwm \
  --user root \
  --workdir /tmp \
  -e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
  -e ROCM_VERSION=7.14.0 \
  -e TORCH_VERSION=2.12.0+rocm7.14.0 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
  -e PYTORCH_ROCM_ARCH=gfx1030 \
  -e GPU_ARCHS=gfx1030 \
  -e AMDGPU_TARGETS=gfx1030 \
  -e CMAKE_HIP_ARCHITECTURES=gfx1030 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp0/cache/torch_extensions,dst=/root/.cache/torch_extensions \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp0/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
  --entrypoint /usr/bin/env \
  sha256:71f0e460e6327c9b3c5edcc90ff104e43ac14089883164c8bad20e9b141d1aeb \
  V756F94_BOUNDARY_MODE=posted \
  DO_NOT_TRACK=1 \
  GPU_DEVICE_ORDINAL=0,1,2,3 \
  GPU_MAX_HW_QUEUES=1 \
  HSA_OVERRIDE_GFX_VERSION=10.3.0 \
  OMP_NUM_THREADS=4 \
  TEMP=/tmp/torchinductor_root \
  TMPDIR=/tmp/torchinductor_root \
  TMP=/tmp/torchinductor_root \
  TORCH_BLAS_PREFER_HIPBLASLT=0 \
  TRITON_CACHE_DIR=/root/.triton \
  VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  VLLM_GPU_MEMORY_UTILIZATION=0.95 \
  VLLM_LOGGING_LEVEL=INFO \
  VLLM_PORT=9002 \
  VLLM_RDNA_FORCE_FP16=1 \
  VLLM_WORKER_MULTIPROC_METHOD=spawn \
  NCCL_P2P_DISABLE=0 \
  NCCL_ALGO=Tree \
  NCCL_PROTO=LL \
  NCCL_MIN_NCHANNELS=4 \
  NCCL_MAX_NCHANNELS=4 \
  NCCL_NTHREADS=256 \
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
  HIP_VISIBLE_DEVICES=0,1,2,3 \
  CUDA_VISIBLE_DEVICES=0,1,2,3 \
  ROCR_VISIBLE_DEVICES=0,1,2,3 \
  VLLM_CACHE_ROOT=/root/.cache/vllm \
  PYTORCH_ROCM_ARCH=gfx1030 \
  PYTHONHASHSEED=0 \
  /opt/venv/bin/vllm \
  serve \
  /opt/local-models/qwen38-27b-hf \
  --served-model-name \
  Qwen/Qwen3.8-27B \
  --enable-auto-tool-choice \
  --tool-call-parser \
  qwen3_coder \
  --dtype \
  half \
  --host \
  0.0.0.0 \
  --port \
  9002 \
  --tensor-parallel-size \
  4 \
  --max-model-len \
  131072 \
  --gpu-memory-utilization \
  0.95 \
  --trust-remote-code \
  --generation-config \
  vllm \
  -O=3 \
  --reasoning-parser \
  qwen3 \
  --async-scheduling \
  --hf-overrides \
  '{"architectures":["Qwen3_5ForCausalLM"]}' \
  --language-model-only \
  --max-num-seqs \
  2 \
  --max-num-batched-tokens \
  2048 \
  --mamba-ssm-cache-dtype \
  float32 \
  --no-enable-prefix-caching \
  --no-enforce-eager \
  --attention-backend \
  TRITON_ATTN \
  --disable-custom-all-reduce \
  --seed \
  0 \
  --worker-cls \
  vllm.vnext_qwen38_worker.Worker
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp4-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-r2/portable/launch.py --config v620-fp16-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-v620-serve-r5.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 33.089 · eligible10K 31.895 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 8403,
    "content_chars": 18748,
    "content_has_think_close": false,
    "elapsed_seconds": 265.0742276192177,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T14:51:04Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.010981950908899307,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 18748,
    "qwen_answer_text_sha256": "9442102c75db56b3781c1291bf8560af55e51ce1e430bb25972fec0512ee9988",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 18748,
    "qwen_visible_text_sha256": "9442102c75db56b3781c1291bf8560af55e51ce1e430bb25972fec0512ee9988",
    "raw_has_think_close": false,
    "reasoning_chars": 24559,
    "text_chars": 43308,
    "text_sha256": "5ae0a353a6f8f0bb582929b4301027c8eea4f03635e43e8c1310d52e1b571406",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 261.28218361292966,
    "vllm_decode_tps": 32.16063140550152,
    "vllm_generation_tokens_delta": 8403.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 3.763594701187685,
    "vllm_prefill_tps": 125.67771972118423,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 31.700554503061724
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 64.21472008898854,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T14:52:08Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.010160903912037611,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 10359,
    "text_chars": 10359,
    "text_sha256": "ff423fe2afad7ed6d76abe61f2c7d0294d6c883fc7e7549ae7b4039b5aadd9cb",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 60.44229609705508,
    "vllm_decode_tps": 33.08941137491707,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 3.7466725350823253,
    "vllm_prefill_tps": 126.24535386293289,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 31.145506781442116
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 27196,
    "content_has_think_close": false,
    "elapsed_seconds": 317.32863993081264,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T14:57:26Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.009598379023373127,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 27196,
    "qwen_answer_text_sha256": "1d6586c92055be911c8df924fd3b36800958433ffca444602a0e3b8d2fc11a91",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 27196,
    "qwen_visible_text_sha256": "1d6586c92055be911c8df924fd3b36800958433ffca444602a0e3b8d2fc11a91",
    "raw_has_think_close": false,
    "reasoning_chars": 24559,
    "text_chars": 51756,
    "text_sha256": "4c8b6331b15aaf31132fce62e6e77116c74c3caf46315b9ab0c7b10807c345ad",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 313.5314573459327,
    "vllm_decode_tps": 31.894726241030963,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 3.777682553976774,
    "vllm_prefill_tps": 125.2090384095593,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 31.513071124561293
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS5,674 natural tokens2K 33.44610K 32.252 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T21:35:28.639248975Z",
      "FinishedAt": "2026-10-26T22:21:26.85186675Z"
    },
    "devices_after": [
      {
        "card": 5,
        "render": 133,
        "pci": "0000:16:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:19:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 7,
        "render": 135,
        "pci": "0000:1c:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 8,
        "render": 136,
        "pci": "0000:1f:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T22:21:26.998315+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 33.44616375717536,
  "retest_10K_TPS": 32.25177356684413,
  "retest_strict_tokens": 5674,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 5674,
      "elapsed_seconds": 176.00342907500453,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 172.4922093800269,
      "vllm_decode_tps": 32.89423922618617,
      "vllm_generation_tokens_delta": 5674.0,
      "vllm_prefill_seconds_delta": 3.498497008928098,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 63.30487280106172,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 59.79759037599433,
      "vllm_decode_tps": 33.44616375717536,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 3.495661307941191,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 313.58174517401494,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 310.06046781502664,
      "vllm_decode_tps": 32.25177356684413,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 3.50971574394498,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 3.495661307941191,
  "retest_2K_total_seconds": 63.30487280106172,
  "retest_10K_prefill_seconds": 3.50971574394498,
  "retest_10K_total_seconds": 313.58174517401494,
  "retest_strict_prefill_seconds": 3.498497008928098,
  "retest_strict_total_seconds": 176.00342907500453,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": null,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30992297984,
    "1": 31004942336,
    "2": 31005011968,
    "3": 30950563840
  },
  "retest_memory_observation_count": 44,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-fp16-tp4-mtp0/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 19.34,
  "retest_logged_model_load_seconds": 102.409229,
  "retest_logged_available_KV_GiB": 7.92,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 7.92,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 129360,
  "retest_logged_context_capacity_x": 3.87,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "v620-fp16-tp4-mtp0",
    "image": "sha256:71f0e460e6327c9b3c5edcc90ff104e43ac14089883164c8bad20e9b141d1aeb",
    "container": "7d983ace8802d4bb9a7baa53f5380cf5063ccfa287413bae5200b7410067fd7b",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 19.34,
    "retest_logged_model_load_seconds": 102.409229,
    "retest_logged_available_KV_GiB": 7.92,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 7.92,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 129360,
    "retest_logged_context_capacity_x": 3.87,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-fp16-tp4-mtp0/service-final.log",
    "source_log_sha256": "2105e2e2c71b1f1c486467df44c1e3bb7f21fcf1728f97336882c12f7f9fca2f",
    "sealed_index_sha256": "0e7d2dc4efd36ed79cc775fbbcb19e362de1cd83ef6d074fe33e79bd4bcd296a",
    "source_lines": {
      "model_load": {
        "line": 851,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 21:39:20 [gpu_model_runner.py:4579] Model loading took 19.34 GiB memory and 102.409229 seconds"
      },
      "automatic_KV_budget": {
        "line": 938,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 21:43:38 [gpu_worker.py:453] Available KV cache memory: 7.92 GiB"
      },
      "grouped_KV_tokens": {
        "line": 943,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 21:43:39 [kv_cache_utils.py:1314] GPU KV cache size: 129,360 tokens"
      },
      "context_capacity": {
        "line": 944,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 21:43:39 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 3.87x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-fp16-tp4-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.875667572021484
}
Retest KV budget: 7.920 GiB (automatic, rounded)129,360 grouped KV tokens · 3.87× reported capacityRetest sampled VRAM: 28.876 GiB max/cardRetest memory reports and limits
Evidence and limits

selected FP16 reference

{
  "strict_tokens": 9102,
  "source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-control-result.json#$.measurements",
  "source_revision": "8ae761a65953ca037494e287f20e03b680503fb4",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "latency_sources": "[{\"path\": \"/home/joe/PhoneSync/Documents/gfx1030/.work/qwen38-migration/retained-control-r0/evidence/run/ladder1/c1_2000_proxy/summary.json\", \"sha256\": \"a32602f1ab06bb87ac7957e783b6bfbaff4deb08c29dca645847ad7cb21241fa\"}, {\"path\": \"/home/joe/PhoneSync/Documents/gfx1030/.work/qwen38-migration/retained-control-r0/evidence/run/ladder1/c1_10000_proxy/summary.json\", \"sha256\": \"2d98720ac31ea42ada1d2b70bd6f4dc25dbde1e9164a62658664d2c0122a5032\"}]",
  "sampled_peak_VRAM_bytes_by_PCI": {
    "0000:16:00.0": 30992314368,
    "0000:19:00.0": 31004827648,
    "0000:1c:00.0": 31004913664,
    "0000:1f:00.0": 30950211584
  },
  "memory_accounting_limit": "All retained integer-buffer logical extents include alternate representations and are not unique physical allocations or measured memory traffic. VRAM is the maximum of retained startup samples.",
  "memory_source": "/home/joe/PhoneSync/Documents/gfx1030/.work/qwen38-migration/retained-control-r0/evidence.tar.gz",
  "memory_source_sha256": "10368e0004f8558f53363a614f289d6e6695e5c26daa2def9a008e2dcb0463ca",
  "retest_strict_tokens": 5674,
  "retest_MTP_accounting_by_tier": {},
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30992297984,
    "1": 31004942336,
    "2": 31005011968,
    "3": 30950563840
  },
  "retest_memory_observation_count": 44,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-fp16-tp4-mtp0/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 7.92,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 7.92,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 129360,
  "retest_memory_log_report": {
    "profile": "v620-fp16-tp4-mtp0",
    "image": "sha256:71f0e460e6327c9b3c5edcc90ff104e43ac14089883164c8bad20e9b141d1aeb",
    "container": "7d983ace8802d4bb9a7baa53f5380cf5063ccfa287413bae5200b7410067fd7b",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 19.34,
    "retest_logged_model_load_seconds": 102.409229,
    "retest_logged_available_KV_GiB": 7.92,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 7.92,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 129360,
    "retest_logged_context_capacity_x": 3.87,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-fp16-tp4-mtp0/service-final.log",
    "source_log_sha256": "2105e2e2c71b1f1c486467df44c1e3bb7f21fcf1728f97336882c12f7f9fca2f",
    "sealed_index_sha256": "0e7d2dc4efd36ed79cc775fbbcb19e362de1cd83ef6d074fe33e79bd4bcd296a",
    "source_lines": {
      "model_load": {
        "line": 851,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 21:39:20 [gpu_model_runner.py:4579] Model loading took 19.34 GiB memory and 102.409229 seconds"
      },
      "automatic_KV_budget": {
        "line": 938,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 21:43:38 [gpu_worker.py:453] Available KV cache memory: 7.92 GiB"
      },
      "grouped_KV_tokens": {
        "line": 943,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 21:43:39 [kv_cache_utils.py:1314] GPU KV cache size: 129,360 tokens"
      },
      "context_capacity": {
        "line": 944,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 21:43:39 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 3.87x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-fp16-tp4-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.875667572021484,
  "sampled_peak_VRAM_GiB": 28.87557601928711
}
V620Current Qwen3.8FP16Qwen/Qwen3.8-27BTP2MTP off PASS8355 tokens 13.06812.694 0.798 / 153.85910K: 0.799 / 788.600Strict: 0.792 / 655.029 25.17 / 28.32FP16 parameter extents / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:afc254a3459d27e4d4fb5b350cbd8c081074da3b696305a7cc8ef7e81bfe476b
Runtime image config: sha256:83a139552fe2e33ba48844e32e69b83ac6d10c592cd0547dfdd1643eaad2408f
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256
mkdir -p launch-tools-v620-serve-r5
tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp2-mtp0
GPU_RENDER_NODES=renderD128,renderD129

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-fp16-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Preflight cleanup compatibility
After its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged.
Retained lab Docker command · not portable
docker run -d \
  --name serve-v620-fp16-tp2-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --cpuset-mems 0 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/card5:/dev/dri/card5:rwm \
  --device /dev/dri/card6:/dev/dri/card6:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
  -e ROCM_VERSION=7.14.0 \
  -e TORCH_VERSION=2.12.0+rocm7.14.0 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
  -e PYTORCH_ROCM_ARCH=gfx1030 \
  -e GPU_ARCHS=gfx1030 \
  -e AMDGPU_TARGETS=gfx1030 \
  -e CMAKE_HIP_ARCHITECTURES=gfx1030 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp0/cache/torch_extensions,dst=/root/.cache/torch_extensions \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp0/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
  --entrypoint /usr/bin/env \
  sha256:83a139552fe2e33ba48844e32e69b83ac6d10c592cd0547dfdd1643eaad2408f \
  V756F94_BOUNDARY_MODE=posted \
  DO_NOT_TRACK=1 \
  GPU_DEVICE_ORDINAL=0,1 \
  GPU_MAX_HW_QUEUES=1 \
  HSA_OVERRIDE_GFX_VERSION=10.3.0 \
  OMP_NUM_THREADS=4 \
  TEMP=/tmp/torchinductor_root \
  TMPDIR=/tmp/torchinductor_root \
  TMP=/tmp/torchinductor_root \
  TORCH_BLAS_PREFER_HIPBLASLT=0 \
  TRITON_CACHE_DIR=/root/.triton \
  VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  VLLM_GPU_MEMORY_UTILIZATION=0.95 \
  VLLM_LOGGING_LEVEL=INFO \
  VLLM_PORT=9002 \
  VLLM_RDNA_FORCE_FP16=1 \
  VLLM_WORKER_MULTIPROC_METHOD=spawn \
  NCCL_P2P_DISABLE=0 \
  NCCL_ALGO=Tree \
  NCCL_PROTO=LL \
  NCCL_MIN_NCHANNELS=4 \
  NCCL_MAX_NCHANNELS=4 \
  NCCL_NTHREADS=256 \
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
  HIP_VISIBLE_DEVICES=0,1 \
  CUDA_VISIBLE_DEVICES=0,1 \
  ROCR_VISIBLE_DEVICES=0,1 \
  VLLM_CACHE_ROOT=/root/.cache/vllm \
  PYTORCH_ROCM_ARCH=gfx1030 \
  PYTHONHASHSEED=0 \
  /opt/venv/bin/vllm \
  serve \
  /opt/local-models/qwen38-27b-hf \
  --served-model-name \
  Qwen/Qwen3.8-27B \
  --enable-auto-tool-choice \
  --tool-call-parser \
  qwen3_coder \
  --dtype \
  half \
  --host \
  0.0.0.0 \
  --port \
  9002 \
  --tensor-parallel-size \
  2 \
  --max-model-len \
  131072 \
  --gpu-memory-utilization \
  0.95 \
  --trust-remote-code \
  --generation-config \
  vllm \
  -O=3 \
  --reasoning-parser \
  qwen3 \
  --async-scheduling \
  --hf-overrides \
  '{"architectures":["Qwen3_5ForCausalLM"]}' \
  --language-model-only \
  --max-num-seqs \
  2 \
  --max-num-batched-tokens \
  2048 \
  --mamba-ssm-cache-dtype \
  float32 \
  --no-enable-prefix-caching \
  --no-enforce-eager \
  --attention-backend \
  TRITON_ATTN \
  --disable-custom-all-reduce \
  --seed \
  0 \
  --worker-cls \
  vllm.vnext_qwen38_worker.Worker
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp2-mtp0
GPU_RENDER_NODES=renderD128,renderD129

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-r2/portable/launch.py --config v620-fp16-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-v620-serve-r5.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 13.092 · eligible10K 12.714 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 5745,
    "content_chars": 12238,
    "content_has_think_close": false,
    "elapsed_seconds": 445.8737674509175,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T16:22:26Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.019420526921749115,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 12238,
    "qwen_answer_text_sha256": "3a79027ad7fed9ab4a8ecdf167ff2ee2cbecab9c7326e7bec78ffa1b0c124d5c",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 12238,
    "qwen_visible_text_sha256": "3a79027ad7fed9ab4a8ecdf167ff2ee2cbecab9c7326e7bec78ffa1b0c124d5c",
    "raw_has_think_close": false,
    "reasoning_chars": 17602,
    "text_chars": 29841,
    "text_sha256": "b0bde25f1140d16eebe07f1b6aa3ec0c00c2d286d99f4a858096c553b6213840",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 445.05560585297644,
    "vllm_decode_tps": 12.908499352545743,
    "vllm_generation_tokens_delta": 5745.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.7985437789466232,
    "vllm_prefill_tps": 592.3282009959989,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 12.884812741607227
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 153.58387053897604,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T16:24:59Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.017818843945860863,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 10439,
    "text_chars": 10439,
    "text_sha256": "ac4d2fe9d9291b42059c546ee942aa7a4e1e96c2f2aa522b5db331457e9aa1ce",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 152.76036863587797,
    "vllm_decode_tps": 13.092400979780507,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8026171950623393,
    "vllm_prefill_tps": 589.3220365946211,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 13.022200788281646
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 35044,
    "content_has_think_close": false,
    "elapsed_seconds": 787.3251471030526,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T16:38:07Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.014115805039182305,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 35044,
    "qwen_answer_text_sha256": "02b0dc02851bd7baf8e137f4e7b9e4c308948ffc73da1c1da4d5a61edd167ead",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 35044,
    "qwen_visible_text_sha256": "02b0dc02851bd7baf8e137f4e7b9e4c308948ffc73da1c1da4d5a61edd167ead",
    "raw_has_think_close": false,
    "reasoning_chars": 17602,
    "text_chars": 52647,
    "text_sha256": "fb7a87c2a39821a2b73c9834aa37be8ce25942b70a75dfa28ce7dcaf0fc12673",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 786.5056542060338,
    "vllm_decode_tps": 12.714466763871973,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.7967424760572612,
    "vllm_prefill_tps": 593.6673570369629,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 12.701232822036491
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS11,009 natural tokens2K 13.12510K 12.752 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T22:23:38.017976016Z",
      "FinishedAt": "2026-10-26T23:40:00.366118153Z"
    },
    "devices_after": [
      {
        "card": 5,
        "render": 133,
        "pci": "0000:16:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:19:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T23:40:00.522295+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 13.125058049552255,
  "retest_10K_TPS": 12.75176962944575,
  "retest_strict_tokens": 11009,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 11009,
      "elapsed_seconds": 867.0248347370652,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 866.2204551490722,
      "vllm_decode_tps": 12.709235777750603,
      "vllm_generation_tokens_delta": 11009.0,
      "vllm_prefill_seconds_delta": 0.7914496649755165,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 153.18691095500253,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 152.3802784299478,
      "vllm_decode_tps": 13.125058049552255,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 0.7956740560475737,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 785.0079319690121,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 784.2048821920762,
      "vllm_decode_tps": 12.75176962944575,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 0.790560997906141,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 0.7956740560475737,
  "retest_2K_total_seconds": 153.18691095500253,
  "retest_10K_prefill_seconds": 0.790560997906141,
  "retest_10K_total_seconds": 785.0079319690121,
  "retest_strict_prefill_seconds": 0.7914496649755165,
  "retest_strict_total_seconds": 867.0248347370652,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": null,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30409326592,
    "1": 30409351168
  },
  "retest_memory_observation_count": 72,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-fp16-tp2-mtp0/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 25.39,
  "retest_logged_model_load_seconds": 15.697194,
  "retest_logged_available_KV_GiB": 1.24,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 1.24,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 9408,
  "retest_logged_context_capacity_x": 0.3,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "v620-fp16-tp2-mtp0",
    "image": "sha256:83a139552fe2e33ba48844e32e69b83ac6d10c592cd0547dfdd1643eaad2408f",
    "container": "15339a206d6624e7abc9caab614f44169f6673d584ba35e9b55bb74a045adbff",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 25.39,
    "retest_logged_model_load_seconds": 15.697194,
    "retest_logged_available_KV_GiB": 1.24,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 1.24,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 9408,
    "retest_logged_context_capacity_x": 0.3,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-fp16-tp2-mtp0/service-final.log",
    "source_log_sha256": "f341d1fc02158cf2480d0571daa0c0d94ea5331083628a0a5fbf9f801b2669a9",
    "sealed_index_sha256": "cf2eb4f5955d369cad6634fb9728c17c9d89e4376c3fad6427c6d195add6bad7",
    "source_lines": {
      "model_load": {
        "line": 91,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 22:25:31 [gpu_model_runner.py:4579] Model loading took 25.39 GiB memory and 15.697194 seconds"
      },
      "automatic_KV_budget": {
        "line": 102,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 22:26:47 [gpu_worker.py:453] Available KV cache memory: 1.24 GiB"
      },
      "grouped_KV_tokens": {
        "line": 104,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 22:26:47 [kv_cache_utils.py:1314] GPU KV cache size: 9,408 tokens"
      },
      "context_capacity": {
        "line": 105,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 22:26:47 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 0.30x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-fp16-tp2-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.32091522216797
}
Retest KV budget: 1.240 GiB (automatic, rounded)9,408 grouped KV tokens · 0.30× reported capacityRetest sampled VRAM: 28.321 GiB max/cardRetest memory reports and limits
Evidence and limits

Completed, independently reviewed FP16 TP2 matrix cell; no FP16 release qualification or promotion.

{
  "strict_tokens": 8355,
  "source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.FP16_W8_TP_MTP_matrix.cells[0].result",
  "source_revision": "1f00a8f7e264ad38c8997d33807e2395bd8e0dd4",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "attention_KV_dtype": "FP16",
  "sampled_peak_VRAM_bytes_by_PCI": "{\"0000:16:00.0\": 30409351168, \"0000:19:00.0\": 30409719808}",
  "KV_capacity": "{\"available_GiB\": 1.24, \"concurrency_at131072\": 0.3, \"tokens\": 9408}",
  "attention_KV_logical_view_bytes_per_rank": 1310195712,
  "post_KV_allocator_allocated_bytes_per_rank": 28462785536,
  "post_KV_allocator_reserved_bytes_per_rank": 28672262144,
  "comparison_limit": "TP2 ordinary inherited FP16 decoder, linear/allreduce, finalnorm and LMhead replace TP4 lossless-packed/posted paths; not an isolated TP change. | Fresh rootSSD caches; selected TP4 reference storage placement retained separately. | Runtime reports 9408 GPU KV tokens and0.30x concurrency at configured131072; full-context residency was not demonstrated. | One sample/cell, no repeatability or FP16qualification; official2K below44. | TP2 ordinary FP16 decoder/fallback and dense loaded storage differ materially from selected TP4 packed control; no causal attribution of speed to TP alone. | Logical tensor/KV views may alias. PyTorchallocated/reserved, sampledcardVRAM and runtimeavailableKV are distinct observations; no fullcontext residency proof. | Inherited closure mode saysquantized-complete but actualbaseline_kindFP16_TP2/checks/image identify thisFP16cell; no quantized result relabel. | W8card7 diagnostic overlapped excludedwarmups only, not official samples.",
  "recorded_configuration_details": {
    "source": "1f00a8f7e264ad38c8997d33807e2395bd8e0dd4",
    "image": "sha256:83a139552fe2e33ba48844e32e69b83ac6d10c592cd0547dfdd1643eaad2408f",
    "strict_valid": true,
    "strict_tokens": 8355,
    "strict_diagnostic_TPS": 12.770910773696434,
    "official_fixed2K_TPS": 13.067638634805824,
    "diagnostic_fixed10K_TPS": 12.693775456156407,
    "metrics": {
      "strict": {
        "completion_tokens": 8355,
        "finish_reason": "stop",
        "elapsed_seconds": 655.0289586309809,
        "vllm_prefill_seconds_delta": 0.7922419030219316,
        "vllm_decode_seconds_delta": 654.2211552529479,
        "vllm_decode_tps": 12.770910773696434,
        "vllm_generation_tokens_delta": 8355.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "f1c67a3a47e860b7bba9f8ff7be2b0266d50b2fff6cb0bb0307c4c8430c39059"
      },
      "fixed2K": {
        "completion_tokens": 2000,
        "finish_reason": "length",
        "elapsed_seconds": 153.8593644410139,
        "vllm_prefill_seconds_delta": 0.7977076760143973,
        "vllm_decode_seconds_delta": 153.0498398289783,
        "vllm_decode_tps": 13.067638634805824,
        "vllm_generation_tokens_delta": 2000.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "7910f673846d91314be1e19fef6cf2b0e8e8b2dc3b25d877365eaf7de2dbb31b"
      },
      "fixed10K": {
        "completion_tokens": 10000,
        "finish_reason": "length",
        "elapsed_seconds": 788.5997820449993,
        "vllm_prefill_seconds_delta": 0.79924124001991,
        "vllm_decode_seconds_delta": 787.7876865349826,
        "vllm_decode_tps": 12.693775456156407,
        "vllm_generation_tokens_delta": 10000.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "40e6f56b389ce2d9681ea939d167f6bf89ed490bd10ba821cac09bf4b49c47cc"
      }
    },
    "weight_dtype": "native FP16; no quantization",
    "activation_dtype": "FP16",
    "attention_KV_dtype": "FP16",
    "TP": 2,
    "MTP_tokens": 0,
    "packed_weight_bytes_per_rank": 0,
    "FP16_logical_tensor_bytes_per_rank": 27030891008,
    "sampled_peak_VRAM_bytes": {
      "0000:16:00.0": 30409351168,
      "0000:19:00.0": 30409719808
    },
    "KV_capacity": {
      "available_GiB": 1.24,
      "tokens": 9408,
      "concurrency_at131072": 0.3
    },
    "configured_context": 131072,
    "complete_ladders": 1,
    "sample_count": 1,
    "repeatability_established": false,
    "FP16_release_qualification": false,
    "archive": {
      "path": "/home/ai/vnext-v620-qwen38-fp16-tp2-service-r0-20261023/raw-evidence.tar.gz",
      "bytes": 381935,
      "sha256": "86afe4421e7be303028b88b503517b15399cdde69a8831b27ce4b1582a11e260",
      "files": 231,
      "index_sha256": "fea34562414c93ead3ef2db5fa65d85b3163d9cb7cd950701fa402b9973efaec"
    },
    "local_primary_record": ".work/fp16-w8-tp-mtp-matrix/fp16-tp2-service-r0/sealed-mirror/retained/evidence/measurement-closure.json",
    "review": {
      "path": ".work/fp16-w8-tp-mtp-matrix/reviews/FP16-TP2-final-cell-review.json",
      "sha256": "43a550344eb2b3059ef139536e3740ab33e7cf4be39d2cd9770f5669ffd5a7e8"
    },
    "closure": {
      "container_id": "eeec72e25a4ac42331400e4525ee45261cc72371a50606f7f6bd94d6b4640d3c",
      "image_id": "sha256:83a139552fe2e33ba48844e32e69b83ac6d10c592cd0547dfdd1643eaad2408f",
      "exit_code": 0,
      "OOMKilled": false,
      "restart_count": 0,
      "bank_idle": true,
      "devices": [
        {
          "pci": "0000:16:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        },
        {
          "pci": "0000:19:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        }
      ],
      "started_at": "2026-10-23T10:34:57.477000767Z",
      "finished_at": "2026-10-23T11:48:28.181764449Z",
      "closed_at": "2026-10-23T11:48:28.337427+00:00",
      "kernel_fault_matches": [],
      "startup_requests_completed_and_drained": 3,
      "official_phase_samples": 11,
      "complete_ladders": 1,
      "strict_valid": true,
      "official_fixed_2000_tps": 13.067638634805824,
      "qualifying_ladders": 0,
      "FP16_qualification": false,
      "baseline_kind": "FP16_TP2"
    },
    "final_counters": {
      "vllm:prompt_tokens_total": 6622,
      "vllm:generation_tokens_total": 42355,
      "vllm:request_decode_time_seconds_count": 14,
      "vllm:request_prefill_time_seconds_count": 14,
      "vllm:num_requests_running": 0,
      "vllm:num_requests_waiting": 0
    },
    "runtime_differences": [
      "TP2 ordinary inherited FP16 decoder, linear/allreduce, finalnorm and LMhead replace TP4 lossless-packed/posted paths; not an isolated TP change.",
      "Fresh rootSSD caches; selected TP4 reference storage placement retained separately.",
      "Runtime reports 9408 GPU KV tokens and0.30x concurrency at configured131072; full-context residency was not demonstrated."
    ],
    "limits": [
      "One sample/cell, no repeatability or FP16qualification; official2K below44.",
      "TP2 ordinary FP16 decoder/fallback and dense loaded storage differ materially from selected TP4 packed control; no causal attribution of speed to TP alone.",
      "Logical tensor/KV views may alias. PyTorchallocated/reserved, sampledcardVRAM and runtimeavailableKV are distinct observations; no fullcontext residency proof.",
      "Inherited closure mode saysquantized-complete but actualbaseline_kindFP16_TP2/checks/image identify thisFP16cell; no quantized result relabel.",
      "W8card7 diagnostic overlapped excludedwarmups only, not official samples."
    ]
  },
  "detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
  "detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
  "detail_source_pointer": "FP16_W8_TP_MTP_matrix.cells[0]",
  "retest_strict_tokens": 11009,
  "retest_MTP_accounting_by_tier": {},
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30409326592,
    "1": 30409351168
  },
  "retest_memory_observation_count": 72,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-fp16-tp2-mtp0/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 1.24,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 1.24,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 9408,
  "retest_memory_log_report": {
    "profile": "v620-fp16-tp2-mtp0",
    "image": "sha256:83a139552fe2e33ba48844e32e69b83ac6d10c592cd0547dfdd1643eaad2408f",
    "container": "15339a206d6624e7abc9caab614f44169f6673d584ba35e9b55bb74a045adbff",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 25.39,
    "retest_logged_model_load_seconds": 15.697194,
    "retest_logged_available_KV_GiB": 1.24,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 1.24,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 9408,
    "retest_logged_context_capacity_x": 0.3,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-fp16-tp2-mtp0/service-final.log",
    "source_log_sha256": "f341d1fc02158cf2480d0571daa0c0d94ea5331083628a0a5fbf9f801b2669a9",
    "sealed_index_sha256": "cf2eb4f5955d369cad6634fb9728c17c9d89e4376c3fad6427c6d195add6bad7",
    "source_lines": {
      "model_load": {
        "line": 91,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 22:25:31 [gpu_model_runner.py:4579] Model loading took 25.39 GiB memory and 15.697194 seconds"
      },
      "automatic_KV_budget": {
        "line": 102,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 22:26:47 [gpu_worker.py:453] Available KV cache memory: 1.24 GiB"
      },
      "grouped_KV_tokens": {
        "line": 104,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 22:26:47 [kv_cache_utils.py:1314] GPU KV cache size: 9,408 tokens"
      },
      "context_capacity": {
        "line": 105,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 22:26:47 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 0.30x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-fp16-tp2-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.32091522216797,
  "sampled_peak_VRAM_GiB": 28.321258544921875
}
V620Current Qwen3.8FP16Qwen/Qwen3.8-27BTP4MTP22K accepted 1266 / 1468 (86.24%) FAIL4212 tokens 39.586*28.223 3.188 / 53.72310K: 3.243 / 357.569Strict: 3.436 / 118.761 1.19 / 27.56retained FP16 parameter extents; projection buffers separate / VRAM GiBSeparate packed-buffer extents: 17.509 GiB/rank; includes alternate views, not additive physical storage. Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:ad86e1b5181d4a8b4a45d24bce14b96f691c0927ca1409e4d43c6eda40764e0d
Runtime image config: sha256:470486980ed01b78334c8e0b3ae36c9396d0ac9d5dba1183d581a857fe95674d
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256
mkdir -p launch-tools-v620-serve-r5
tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp4-mtp2
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-fp16-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Preflight cleanup compatibility
After its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged.
Retained lab Docker command · not portable
docker run -d \
  --name serve-v620-fp16-tp4-mtp2-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --cpuset-mems 0 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/card5:/dev/dri/card5:rwm \
  --device /dev/dri/card6:/dev/dri/card6:rwm \
  --device /dev/dri/card7:/dev/dri/card7:rwm \
  --device /dev/dri/card8:/dev/dri/card8:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
  --device /dev/dri/renderD136:/dev/dri/renderD136:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
  -e ROCM_VERSION=7.14.0 \
  -e TORCH_VERSION=2.12.0+rocm7.14.0 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
  -e PYTORCH_ROCM_ARCH=gfx1030 \
  -e GPU_ARCHS=gfx1030 \
  -e AMDGPU_TARGETS=gfx1030 \
  -e CMAKE_HIP_ARCHITECTURES=gfx1030 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp2/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp2/cache/torch_extensions,dst=/root/.cache/torch_extensions \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp2/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
  --entrypoint /usr/bin/env \
  sha256:470486980ed01b78334c8e0b3ae36c9396d0ac9d5dba1183d581a857fe95674d \
  V756F94_BOUNDARY_MODE=posted \
  DO_NOT_TRACK=1 \
  GPU_DEVICE_ORDINAL=0,1,2,3 \
  GPU_MAX_HW_QUEUES=1 \
  HSA_OVERRIDE_GFX_VERSION=10.3.0 \
  OMP_NUM_THREADS=4 \
  TEMP=/tmp/torchinductor_root \
  TMPDIR=/tmp/torchinductor_root \
  TMP=/tmp/torchinductor_root \
  TORCH_BLAS_PREFER_HIPBLASLT=0 \
  TRITON_CACHE_DIR=/root/.triton \
  VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  VLLM_GPU_MEMORY_UTILIZATION=0.95 \
  VLLM_LOGGING_LEVEL=INFO \
  VLLM_PORT=9002 \
  VLLM_RDNA_FORCE_FP16=1 \
  VLLM_WORKER_MULTIPROC_METHOD=spawn \
  NCCL_P2P_DISABLE=0 \
  NCCL_ALGO=Tree \
  NCCL_PROTO=LL \
  NCCL_MIN_NCHANNELS=4 \
  NCCL_MAX_NCHANNELS=4 \
  NCCL_NTHREADS=256 \
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
  HIP_VISIBLE_DEVICES=0,1,2,3 \
  CUDA_VISIBLE_DEVICES=0,1,2,3 \
  ROCR_VISIBLE_DEVICES=0,1,2,3 \
  VLLM_CACHE_ROOT=/root/.cache/vllm \
  PYTORCH_ROCM_ARCH=gfx1030 \
  PYTHONHASHSEED=0 \
  /opt/venv/bin/vllm \
  serve \
  /opt/local-models/qwen38-27b-hf \
  --served-model-name \
  Qwen/Qwen3.8-27B \
  --enable-auto-tool-choice \
  --tool-call-parser \
  qwen3_coder \
  --dtype \
  half \
  --host \
  0.0.0.0 \
  --port \
  9002 \
  --tensor-parallel-size \
  4 \
  --max-model-len \
  131072 \
  --gpu-memory-utilization \
  0.95 \
  --trust-remote-code \
  --generation-config \
  vllm \
  -O=3 \
  --reasoning-parser \
  qwen3 \
  --async-scheduling \
  --hf-overrides \
  '{"architectures":["Qwen3_5ForCausalLM"]}' \
  --language-model-only \
  --max-num-seqs \
  2 \
  --max-num-batched-tokens \
  2048 \
  --mamba-ssm-cache-dtype \
  float32 \
  --no-enable-prefix-caching \
  --no-enforce-eager \
  --attention-backend \
  TRITON_ATTN \
  --disable-custom-all-reduce \
  --seed \
  0 \
  --worker-cls \
  vllm.vnext_qwen38_worker.Worker \
  --speculative-config \
  '{"method":"mtp","num_speculative_tokens":2}'
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp4-mtp2
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-r2/portable/launch.py --config v620-fp16-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-v620-serve-r5.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 32.783 · eligible10K 25.989 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 8503,
    "content_chars": 18777,
    "content_has_think_close": false,
    "elapsed_seconds": 313.99245239887387,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T17:52:11Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.013237901031970978,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 18777,
    "qwen_answer_text_sha256": "0efa285a05756d4fa88517b1757968c2239a589403da2f8f23b3765d95c9d4e2",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 18777,
    "qwen_visible_text_sha256": "0efa285a05756d4fa88517b1757968c2239a589403da2f8f23b3765d95c9d4e2",
    "raw_has_think_close": false,
    "reasoning_chars": 25187,
    "text_chars": 43965,
    "text_sha256": "2faa6a681bead4a90b43fa84dc20b179f7bb62ae038ed66bf007960f41c5e720",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 310.6009135739878,
    "vllm_decode_tps": 27.375965840404756,
    "vllm_generation_tokens_delta": 8503.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 3.370070411125198,
    "vllm_prefill_tps": 140.35315061624334,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 27.08026876135987
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 64.3155171200633,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T17:53:16Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.014873459935188293,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 10388,
    "text_chars": 10388,
    "text_sha256": "814c78cff59e5fe898f376561d8a7d6c88fe49407d365e6dfa2b9968ccd3efa1",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 61.00775268115103,
    "vllm_decode_tps": 32.78271878744225,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 3.2730508069507778,
    "vllm_prefill_tps": 144.51349150936454,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 31.09669469447673
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 26696,
    "content_has_think_close": false,
    "elapsed_seconds": 388.10585302114487,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T17:59:44Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.0162037571426481,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 26696,
    "qwen_answer_text_sha256": "1395c5317617a065b003c9214deb93a08142c96d81125b0968c7a5c22c77dd41",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 26696,
    "qwen_visible_text_sha256": "1395c5317617a065b003c9214deb93a08142c96d81125b0968c7a5c22c77dd41",
    "raw_has_think_close": false,
    "reasoning_chars": 25187,
    "text_chars": 51884,
    "text_sha256": "22fb1bdfc8331e84906af8185b7249afaff80ccd51cc9a6bcb763122a674d672",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 384.77785909199156,
    "vllm_decode_tps": 25.9890213631269,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 3.313091953052208,
    "vllm_prefill_tps": 142.7669399770947,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 25.7661664263929
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestMTP counter replayPASS9,359 natural tokens2K 32.92610K 25.631 · diagnostic2K MTP accepted 1108 / 1782 (62.18%)
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T23:42:11.23673073Z",
      "FinishedAt": "2026-10-27T00:29:51.078728116Z"
    },
    "devices_after": [
      {
        "card": 5,
        "render": 133,
        "pci": "0000:16:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:19:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 7,
        "render": 135,
        "pci": "0000:1c:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 8,
        "render": 136,
        "pci": "0000:1f:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-27T00:29:51.231343+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 32.925748283050446,
  "retest_10K_TPS": 25.630758150290323,
  "retest_strict_tokens": 9359,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 9359,
      "elapsed_seconds": 357.20782125892583,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 353.9098056239309,
      "vllm_decode_tps": 26.44459082872938,
      "vllm_generation_tokens_delta": 9359.0,
      "vllm_prefill_seconds_delta": 3.2840392630314454,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 64.00938890292309,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 60.74273492000066,
      "vllm_decode_tps": 32.925748283050446,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 3.2527504609897733,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 393.44662721501663,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 390.1562310940353,
      "vllm_decode_tps": 25.630758150290323,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 3.2780255859252065,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 3.2527504609897733,
  "retest_2K_total_seconds": 64.00938890292309,
  "retest_10K_prefill_seconds": 3.2780255859252065,
  "retest_10K_total_seconds": 393.44662721501663,
  "retest_strict_prefill_seconds": 3.2840392630314454,
  "retest_strict_total_seconds": 357.20782125892583,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 7194.0,
        "vllm:spec_decode_num_drafts_total": 3597.0,
        "vllm:spec_decode_num_accepted_tokens_total": 5761.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:generation_tokens_total": 9359.0,
        "vllm:request_generation_tokens_sum": 9359.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 5761.0
      },
      "accepted_per_position": {
        "0": 3109.0,
        "1": 2652.0
      },
      "accepted_fraction": 0.800806227411732,
      "committed_emitted_tokens": 9359.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "2276ca23df0bed94b682de122f7bd0c2b00260cb47bab2037e2f43969ea70a4d",
        "/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "e556fe48dbe5c75807870feb3d7fa9d231f189e83256bd5b0205ac4e3a6c6bb8"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 1108.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_drafts_total": 891.0,
        "vllm:spec_decode_num_draft_tokens_total": 1782.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1108.0
      },
      "accepted_per_position": {
        "0": 658.0,
        "1": 450.0
      },
      "accepted_fraction": 0.621773288439955,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "beff10249012b5c21c4b4f08a002d9d1b1942567e62040c29fc81c3c7f18886f",
        "/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "9edb860863c37ec9015da0b8a76e971b6c6b6591fbc394f34c7a05d88e4b116f"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 7690.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_drafts_total": 3845.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6155.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6155.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_generation_tokens_count": 1.0
      },
      "accepted_per_position": {
        "0": 3324.0,
        "1": 2831.0
      },
      "accepted_fraction": 0.8003901170351105,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "d7b34f54ae7309059cd34e1d1e1c711375863431290ef5ec2011db877332c33a",
        "/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "b7472565372751ecbb75b3d7944f94907595198288e1e00dce240ad47304310b"
      }
    }
  },
  "retest_MTP_2K_accepted": 1108.0,
  "retest_MTP_2K_proposed": 1782.0,
  "retest_MTP_2K_draft_batches": 891.0,
  "retest_MTP_2K_acceptance_percent": 62.177328843995504,
  "retest_loaded_admission": null,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 29542203392,
    "1": 29554749440,
    "2": 29552521216,
    "3": 29499723776
  },
  "retest_memory_observation_count": 46,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-fp16-tp4-mtp2/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 20.64,
  "retest_logged_model_load_seconds": 68.668489,
  "retest_logged_available_KV_GiB": 6.11,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 6.11,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 93600,
  "retest_logged_context_capacity_x": 2.72,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "v620-fp16-tp4-mtp2",
    "image": "sha256:470486980ed01b78334c8e0b3ae36c9396d0ac9d5dba1183d581a857fe95674d",
    "container": "ecc08d8c02c333ce0a0cbdc6b46ac5a15fc230cb1189ca255b88a68f07f03fd9",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 20.64,
    "retest_logged_model_load_seconds": 68.668489,
    "retest_logged_available_KV_GiB": 6.11,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 6.11,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 93600,
    "retest_logged_context_capacity_x": 2.72,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-fp16-tp4-mtp2/service-final.log",
    "source_log_sha256": "7b07a9c59e06465fab4a524d823a544a019ffd6d1da77b716d1da525392aaa3c",
    "sealed_index_sha256": "77ccd32707a78a1145f1917f797db402a47ad46148bc20fb5ff6f1af0a7821aa",
    "source_lines": {
      "model_load": {
        "line": 893,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 23:45:18 [gpu_model_runner.py:4579] Model loading took 20.64 GiB memory and 68.668489 seconds"
      },
      "automatic_KV_budget": {
        "line": 986,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-27 00:02:07 [gpu_worker.py:453] Available KV cache memory: 6.11 GiB"
      },
      "grouped_KV_tokens": {
        "line": 991,
        "text": "(EngineCore_DP0 pid=85) INFO 10-27 00:02:07 [kv_cache_utils.py:1314] GPU KV cache size: 93,600 tokens"
      },
      "context_capacity": {
        "line": 992,
        "text": "(EngineCore_DP0 pid=85) INFO 10-27 00:02:07 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 2.72x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-fp16-tp4-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 27.525005340576172
}
Retest KV budget: 6.110 GiB (automatic, rounded)93,600 grouped KV tokens · 2.72× reported capacityRetest sampled VRAM: 27.525 GiB max/cardRetest memory reports and limits
Evidence and limits

Completed strict-failed: no final answer. Fixed 2K and 10K speeds are diagnostic and ineligible; no strict rerun.

{
  "strict_tokens": 4212,
  "source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.FP16_W8_TP_MTP_matrix.cells[1].result",
  "source_revision": "5af8a8e02d310edaf1fbb289bcf430ea2fe75791",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "attention_KV_dtype": "FP16",
  "memory_accounting_limit": "Retained FP16 parameter extents are not all model weights: TP4 projection tensors use lossless integer buffers. All packed-buffer extents include alternate representations. Draft tensors can share target embeddings/head. None of these logical sums is a unique physical allocation or measured memory traffic.",
  "load_and_KV_allocator_bytes_per_rank": "[{\"KV_allocated_bytes\": 27199667712, \"KV_reserved_bytes\": 27770486784, \"draft_KV_logical_view_sum_bytes\": 387481600, \"load_allocated_bytes\": 20569802752, \"load_reserved_bytes\": 22196256768, \"rank\": 0, \"target_KV_logical_view_sum_bytes\": 6199705600}, {\"KV_allocated_bytes\": 27201705984, \"KV_reserved_bytes\": 27766292480, \"draft_KV_logical_view_sum_bytes\": 387481600, \"load_allocated_bytes\": 20571841024, \"load_reserved_bytes\": 22192062464, \"rank\": 1, \"target_KV_logical_view_sum_bytes\": 6199705600}, {\"KV_allocated_bytes\": 27201809408, \"KV_reserved_bytes\": 27764195328, \"draft_KV_logical_view_sum_bytes\": 387481600, \"load_allocated_bytes\": 20571944448, \"load_reserved_bytes\": 22189965312, \"rank\": 2, \"target_KV_logical_view_sum_bytes\": 6199705600}, {\"KV_allocated_bytes\": 27202460672, \"KV_reserved_bytes\": 27762098176, \"draft_KV_logical_view_sum_bytes\": 387481600, \"load_allocated_bytes\": 20572595712, \"load_reserved_bytes\": 22187868160, \"rank\": 3, \"target_KV_logical_view_sum_bytes\": 6199705600}]",
  "sampled_peak_VRAM_bytes_by_PCI": "{\"0000:16:00.0\": 29575184384, \"0000:19:00.0\": 29587857408, \"0000:1c:00.0\": 29585731584, \"0000:1f:00.0\": 29533675520}",
  "KV_capacity": "{\"GPU_KV_tokens\": 94400, \"actual_KV_shape\": [473, 2, 800, 1, 256], \"available_KV_GiB\": 6.15, \"graph_actual_GiB_each_rank\": 0.27, \"max_concurrency_at131072\": 2.73, \"service_ready_log_sha256\": \"881258f0123ae8f7314137c015c2fb15188bc00616dd64750de5cdc7b713a771\"}",
  "attention_KV_logical_view_shape": "[473, 2, 800, 1, 256]",
  "post_KV_allocator_allocated_bytes_by_rank": "{\"0\": 27199667712, \"1\": 27201705984, \"2\": 27201809408, \"3\": 27202460672}",
  "post_KV_allocator_reserved_bytes_by_rank": "{\"0\": 27770486784, \"1\": 27766292480, \"2\": 27764195328, \"3\": 27762098176}",
  "MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_sum": 4212.0,
        "vllm:generation_tokens_total": 4212.0,
        "vllm:spec_decode_num_drafts_total": 1471.0,
        "vllm:spec_decode_num_draft_tokens_total": 2942.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_total": 2740.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 2740.0
      },
      "accepted_per_position": {
        "0": 1403.0,
        "1": 1337.0
      },
      "accepted_fraction": 0.9313392250169953,
      "committed_emitted_tokens": 4212.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-before.txt": "f56d64dad72696f5f4697a23288e544e9fdce350bae5d41cb590eced112a31b2",
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-after.txt": "6c6096b79e7766569d8f8b3a85d93317babe945918808cd1a82a042f605cee86"
      }
    },
    "fixed2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:generation_tokens_total": 2000.0,
        "vllm:spec_decode_num_drafts_total": 734.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 1468.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1266.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1266.0
      },
      "accepted_per_position": {
        "0": 666.0,
        "1": 600.0
      },
      "accepted_fraction": 0.8623978201634878,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-before.txt": "9412ba3b6d627fce6606a9d9b2ecbc8ede1018cdf4187ced3bb1f9c4c0b9ce36",
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-after.txt": "267c4ee5121579bc10250e527b2f98a8bfa68eab7f732aede2e1218ab9355c26"
      }
    },
    "fixed10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 6802.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6600.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6600.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_drafts_total": 3401.0
      },
      "accepted_per_position": {
        "0": 3333.0,
        "1": 3267.0
      },
      "accepted_fraction": 0.9703028521023228,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-before.txt": "b25e84e356c2d6c146b0b48a94b21be683ff3b00fda0db153f4d0c8e1077f853",
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-after.txt": "617f37fd48e4cf04df1ec067066442637258348978916934fe3477d6b600e3d1"
      }
    }
  },
  "MTP_accounting_limit": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
  "source_evidence_commit": "7221c11",
  "comparison_limit": "Native MTP requires inherited ordinary target decoder/residual/norm/collectives instead of the unsupported TP4 posted boundary. The r0 pre-HTTP initialization failure remains retained. | Actual800-block attentionKV uses inherited fallback; historical non-MTP uses784 native route. No isolated MTP gain is established. | Same pinned official checkpoint, FP16 activation/attentionKV, greedy profile/context131072 and fresh private rootSSD caches. | Original ladder exits2 on strict failure. Only the remaining fixed2K/10K were continued with original proxy/settings on unchanged loaded model; idle gap and actual parent exits retained. | StrictFAIL remainssolecorrectnessverdict. No repeatstrict, passfishing, officialspeedeligibility orFP16qualification. | MTPaccepted counts precede finalstoptrimming; committedtokens only speednumerator. E/draftbatches is descriptive, not measuredtokensperGPUstep. | Targetintegerpackcounts include retainedmultipleformats; targetplainFP16 sums are not totaluniquephysicalmemory. Draftsharedembedding/head overlaptarget; KVviews mayalias; allocator/cardVRAM/runtimecapacity are distinct. | Onecohort; ordinarytargetcollective/norm and800attentionfallback materially differ fromTP4offreference. No isolatedMTPgain orcausalclaim.",
  "recorded_configuration_details": {
    "source": "5af8a8e02d310edaf1fbb289bcf430ea2fe75791",
    "image": "sha256:470486980ed01b78334c8e0b3ae36c9396d0ac9d5dba1183d581a857fe95674d",
    "manifest": "f3db159511bad243ff7939343846e5febc89327f5774cae9419cbafcde6c1b59",
    "strict_valid": false,
    "strict_tokens": 4212,
    "strict_failure": "4212 naturalstop;missing_think_close_or_parser_answer;no finalanswer",
    "strict_diagnostic_TPS": 36.52689113332412,
    "official_fixed2K_TPS": null,
    "official_speed_eligible": false,
    "diagnostic_fixed2K_TPS": 39.58600405759037,
    "diagnostic_fixed10K_TPS": 28.223489817950945,
    "metrics": {
      "strict": {
        "completion_tokens": 4212,
        "finish_reason": "stop",
        "elapsed_seconds": 118.7605559109943,
        "vllm_prefill_seconds_delta": 3.4359108269563876,
        "vllm_decode_seconds_delta": 115.31230469699949,
        "vllm_decode_tps": 36.52689113332412,
        "vllm_generation_tokens_delta": 4212.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "88d2ed6ee05e0b1aa5ba8823cbc24b5a88d80c8266b6d5fc74dc55902d3d6f23",
        "server_e2e_seconds": 118.75620102882385,
        "MTP": {
          "status": "observed",
          "actual_MTP_work_observed": true,
          "counter_deltas": {
            "vllm:request_generation_tokens_sum": 4212.0,
            "vllm:generation_tokens_total": 4212.0,
            "vllm:spec_decode_num_drafts_total": 1471.0,
            "vllm:spec_decode_num_draft_tokens_total": 2942.0,
            "vllm:request_generation_tokens_count": 1.0,
            "vllm:spec_decode_num_accepted_tokens_total": 2740.0,
            "vllm:spec_decode_num_accepted_tokens_per_pos_total": 2740.0
          },
          "accepted_per_position": {
            "0": 1403.0,
            "1": 1337.0
          },
          "accepted_fraction": 0.9313392250169953,
          "committed_emitted_tokens": 4212.0,
          "request_tokens_agree": true,
          "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
          "sources": {
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-before.txt": "f56d64dad72696f5f4697a23288e544e9fdce350bae5d41cb590eced112a31b2",
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-after.txt": "6c6096b79e7766569d8f8b3a85d93317babe945918808cd1a82a042f605cee86"
          }
        }
      },
      "fixed2K": {
        "completion_tokens": 2000,
        "finish_reason": "length",
        "elapsed_seconds": 53.72339402302168,
        "vllm_prefill_seconds_delta": 3.1883261629845947,
        "vllm_decode_seconds_delta": 50.522906962025445,
        "vllm_decode_tps": 39.58600405759037,
        "vllm_generation_tokens_delta": 2000.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "8194a75ebf601b18347c55e6d90a806ab947b2498a3c2b33df8928ead3e02028",
        "server_e2e_seconds": 53.71901798248291,
        "MTP": {
          "status": "observed",
          "actual_MTP_work_observed": true,
          "counter_deltas": {
            "vllm:generation_tokens_total": 2000.0,
            "vllm:spec_decode_num_drafts_total": 734.0,
            "vllm:request_generation_tokens_sum": 2000.0,
            "vllm:request_generation_tokens_count": 1.0,
            "vllm:spec_decode_num_draft_tokens_total": 1468.0,
            "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1266.0,
            "vllm:spec_decode_num_accepted_tokens_total": 1266.0
          },
          "accepted_per_position": {
            "0": 666.0,
            "1": 600.0
          },
          "accepted_fraction": 0.8623978201634878,
          "committed_emitted_tokens": 2000.0,
          "request_tokens_agree": true,
          "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
          "sources": {
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-before.txt": "9412ba3b6d627fce6606a9d9b2ecbc8ede1018cdf4187ced3bb1f9c4c0b9ce36",
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-after.txt": "267c4ee5121579bc10250e527b2f98a8bfa68eab7f732aede2e1218ab9355c26"
          }
        }
      },
      "fixed10K": {
        "completion_tokens": 10000,
        "finish_reason": "length",
        "elapsed_seconds": 357.5690232060151,
        "vllm_prefill_seconds_delta": 3.242719638044946,
        "vllm_decode_seconds_delta": 354.31479467998724,
        "vllm_decode_tps": 28.223489817950945,
        "vllm_generation_tokens_delta": 10000.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "4257fd9708b76c87e6eee072f2051550b27816ce3500f4812bc0e797317f66d9",
        "server_e2e_seconds": 357.5645079612732,
        "MTP": {
          "status": "observed",
          "actual_MTP_work_observed": true,
          "counter_deltas": {
            "vllm:spec_decode_num_draft_tokens_total": 6802.0,
            "vllm:generation_tokens_total": 10000.0,
            "vllm:spec_decode_num_accepted_tokens_total": 6600.0,
            "vllm:request_generation_tokens_count": 1.0,
            "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6600.0,
            "vllm:request_generation_tokens_sum": 10000.0,
            "vllm:spec_decode_num_drafts_total": 3401.0
          },
          "accepted_per_position": {
            "0": 3333.0,
            "1": 3267.0
          },
          "accepted_fraction": 0.9703028521023228,
          "committed_emitted_tokens": 10000.0,
          "request_tokens_agree": true,
          "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
          "sources": {
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-before.txt": "b25e84e356c2d6c146b0b48a94b21be683ff3b00fda0db153f4d0c8e1077f853",
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-after.txt": "617f37fd48e4cf04df1ec067066442637258348978916934fe3477d6b600e3d1"
          }
        }
      }
    },
    "weight_dtype": "Native FP16 with selected lossless integer packs; no quantization",
    "activation_dtype": "FP16",
    "attention_KV_dtype": "FP16",
    "TP": 4,
    "MTP_tokens": 2,
    "MTP_precision": "Five native Qwen MTP projections FP16, same official revision; embedding/head shared with target",
    "packed_logical_integer_bytes_per_rank": [
      18799031148,
      18800533083,
      18800415543,
      18799381732
    ],
    "packed_bytes_limit": "Logical tensor extents include multiple retained lossless representations; not unique physical allocations or quantized weights. Draft shared embedding/head must not be added twice.",
    "draft_five_projection_FP16_bytes_per_rank": 212336640,
    "sampled_peak_VRAM_bytes": {
      "0000:16:00.0": 29575184384,
      "0000:19:00.0": 29587857408,
      "0000:1c:00.0": 29585731584,
      "0000:1f:00.0": 29533675520
    },
    "KV_capacity": {
      "service_ready_log_sha256": "881258f0123ae8f7314137c015c2fb15188bc00616dd64750de5cdc7b713a771",
      "available_KV_GiB": 6.15,
      "GPU_KV_tokens": 94400,
      "max_concurrency_at131072": 2.73,
      "graph_actual_GiB_each_rank": 0.27,
      "actual_KV_shape": [
        473,
        2,
        800,
        1,
        256
      ]
    },
    "load_and_KV_allocator_bytes_per_rank": [
      {
        "rank": 0,
        "load_allocated_bytes": 20569802752,
        "load_reserved_bytes": 22196256768,
        "KV_allocated_bytes": 27199667712,
        "KV_reserved_bytes": 27770486784,
        "target_KV_logical_view_sum_bytes": 6199705600,
        "draft_KV_logical_view_sum_bytes": 387481600
      },
      {
        "rank": 1,
        "load_allocated_bytes": 20571841024,
        "load_reserved_bytes": 22192062464,
        "KV_allocated_bytes": 27201705984,
        "KV_reserved_bytes": 27766292480,
        "target_KV_logical_view_sum_bytes": 6199705600,
        "draft_KV_logical_view_sum_bytes": 387481600
      },
      {
        "rank": 2,
        "load_allocated_bytes": 20571944448,
        "load_reserved_bytes": 22189965312,
        "KV_allocated_bytes": 27201809408,
        "KV_reserved_bytes": 27764195328,
        "target_KV_logical_view_sum_bytes": 6199705600,
        "draft_KV_logical_view_sum_bytes": 387481600
      },
      {
        "rank": 3,
        "load_allocated_bytes": 20572595712,
        "load_reserved_bytes": 22187868160,
        "KV_allocated_bytes": 27202460672,
        "KV_reserved_bytes": 27762098176,
        "target_KV_logical_view_sum_bytes": 6199705600,
        "draft_KV_logical_view_sum_bytes": 387481600
      }
    ],
    "configured_context": 131072,
    "complete_ladders": 1,
    "sample_count": 1,
    "repeatability_established": false,
    "FP16_release_qualification": false,
    "archive": {
      "path": "/home/ai/vnext-v620-qwen38-fp16-tp4-mtp2-service-r1-20261023/raw-evidence.tar.gz",
      "bytes": 447218,
      "sha256": "002994242cea527782ecaf7bbc3da98b6f15755f28f38338bc3c213fb50ab02d",
      "files": 249,
      "index_sha256": "c707c0df10917b1a3a4a5a328cf66ac149f5e67622d686b5997edce0c1f6e89a"
    },
    "local_primary_record": ".work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/measurement-closure.json",
    "review": {
      "path": ".work/fp16-w8-tp-mtp-matrix/reviews/FP16-TP4-MTP2-r1-final-cell-review.json",
      "sha256": "7590389df32af23bad89dfcb365a3947f274f148ec871c47ce91c0465596c987"
    },
    "closure": {
      "container_id": "b5d65c39ad36036f9d62f935335427adf830338159429b9789b2f31ca2459029",
      "image_id": "sha256:470486980ed01b78334c8e0b3ae36c9396d0ac9d5dba1183d581a857fe95674d",
      "exit_code": 0,
      "OOMKilled": false,
      "restart_count": 0,
      "bank_idle": true,
      "devices": [
        {
          "pci": "0000:16:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        },
        {
          "pci": "0000:19:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        },
        {
          "pci": "0000:1c:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        },
        {
          "pci": "0000:1f:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        }
      ],
      "started_at": "2026-10-23T12:53:46.083011206Z",
      "finished_at": "2026-10-23T13:42:15.417744232Z",
      "closed_at": "2026-10-23T13:42:15.583708+00:00",
      "kernel_fault_matches": [],
      "startup_requests_completed_and_drained": 3,
      "official_phase_samples": 11,
      "complete_ladders": 1,
      "strict_valid": false,
      "official_fixed_2000_tps": null,
      "measured_fixed_2000_tps": 39.58600405759037,
      "valid_comparator": false,
      "official_speed_eligible": false,
      "promotion_eligible": false,
      "qualifying_ladders": 0,
      "FP16_qualification": false,
      "baseline_kind": "FP16_TP4_MTP2"
    },
    "final_counters": {
      "vllm:prompt_tokens_total": 6622,
      "vllm:generation_tokens_total": 38212,
      "vllm:request_decode_time_seconds_count": 14,
      "vllm:request_prefill_time_seconds_count": 14,
      "vllm:num_requests_running": 0,
      "vllm:num_requests_waiting": 0
    },
    "runtime_differences": [
      "Native MTP requires inherited ordinary target decoder/residual/norm/collectives instead of the unsupported TP4 posted boundary. The r0 pre-HTTP initialization failure remains retained.",
      "Actual800-block attentionKV uses inherited fallback; historical non-MTP uses784 native route. No isolated MTP gain is established.",
      "Same pinned official checkpoint, FP16 activation/attentionKV, greedy profile/context131072 and fresh private rootSSD caches.",
      "Original ladder exits2 on strict failure. Only the remaining fixed2K/10K were continued with original proxy/settings on unchanged loaded model; idle gap and actual parent exits retained."
    ],
    "limits": [
      "StrictFAIL remainssolecorrectnessverdict. No repeatstrict, passfishing, officialspeedeligibility orFP16qualification.",
      "MTPaccepted counts precede finalstoptrimming; committedtokens only speednumerator. E/draftbatches is descriptive, not measuredtokensperGPUstep.",
      "Targetintegerpackcounts include retainedmultipleformats; targetplainFP16 sums are not totaluniquephysicalmemory. Draftsharedembedding/head overlaptarget; KVviews mayalias; allocator/cardVRAM/runtimecapacity are distinct.",
      "Onecohort; ordinarytargetcollective/norm and800attentionfallback materially differ fromTP4offreference. No isolatedMTPgain orcausalclaim."
    ]
  },
  "MTP_2K_draft_batches": 734.0,
  "MTP_2K_proposed": 1468.0,
  "MTP_2K_accepted": 1266.0,
  "MTP_2K_acceptance_percent": 86.23978201634877,
  "MTP_2K_committed": 2000.0,
  "detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
  "detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
  "detail_source_pointer": "FP16_W8_TP_MTP_matrix.cells[1]",
  "loaded_metadata_sources_by_rank": {
    "0": {
      "path": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-0-pid-103.json",
      "sha256": "928ef5f45f3514b3c1299bb216e29ccb1e7e68a1e3313e3265d40333654df8b5"
    },
    "1": {
      "path": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-1-pid-104.json",
      "sha256": "b42567a2c1f8817818b7794e37a5be515d7806e6504792b0933be1208f469df5"
    },
    "2": {
      "path": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-2-pid-105.json",
      "sha256": "a8646e8e4ef3ebba23f68645457131c031de8179741969d2292527f81de54c93"
    },
    "3": {
      "path": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-3-pid-106.json",
      "sha256": "67ad2947d99cb0464fdcbc70550ea2c6d111db32980b2f6edd6e2ed563e62d0a"
    }
  },
  "memory_source": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-3-pid-106.json",
  "memory_source_sha256": "67ad2947d99cb0464fdcbc70550ea2c6d111db32980b2f6edd6e2ed563e62d0a",
  "retest_strict_tokens": 9359,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 7194.0,
        "vllm:spec_decode_num_drafts_total": 3597.0,
        "vllm:spec_decode_num_accepted_tokens_total": 5761.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:generation_tokens_total": 9359.0,
        "vllm:request_generation_tokens_sum": 9359.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 5761.0
      },
      "accepted_per_position": {
        "0": 3109.0,
        "1": 2652.0
      },
      "accepted_fraction": 0.800806227411732,
      "committed_emitted_tokens": 9359.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "2276ca23df0bed94b682de122f7bd0c2b00260cb47bab2037e2f43969ea70a4d",
        "/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "e556fe48dbe5c75807870feb3d7fa9d231f189e83256bd5b0205ac4e3a6c6bb8"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 1108.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_drafts_total": 891.0,
        "vllm:spec_decode_num_draft_tokens_total": 1782.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1108.0
      },
      "accepted_per_position": {
        "0": 658.0,
        "1": 450.0
      },
      "accepted_fraction": 0.621773288439955,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "beff10249012b5c21c4b4f08a002d9d1b1942567e62040c29fc81c3c7f18886f",
        "/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "9edb860863c37ec9015da0b8a76e971b6c6b6591fbc394f34c7a05d88e4b116f"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 7690.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_drafts_total": 3845.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6155.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6155.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_generation_tokens_count": 1.0
      },
      "accepted_per_position": {
        "0": 3324.0,
        "1": 2831.0
      },
      "accepted_fraction": 0.8003901170351105,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "d7b34f54ae7309059cd34e1d1e1c711375863431290ef5ec2011db877332c33a",
        "/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "b7472565372751ecbb75b3d7944f94907595198288e1e00dce240ad47304310b"
      }
    }
  },
  "retest_MTP_2K_accepted": 1108.0,
  "retest_MTP_2K_proposed": 1782.0,
  "retest_MTP_2K_draft_batches": 891.0,
  "retest_MTP_2K_acceptance_percent": 62.177328843995504,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 29542203392,
    "1": 29554749440,
    "2": 29552521216,
    "3": 29499723776
  },
  "retest_memory_observation_count": 46,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-fp16-tp4-mtp2/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 6.11,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 6.11,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 93600,
  "retest_memory_log_report": {
    "profile": "v620-fp16-tp4-mtp2",
    "image": "sha256:470486980ed01b78334c8e0b3ae36c9396d0ac9d5dba1183d581a857fe95674d",
    "container": "ecc08d8c02c333ce0a0cbdc6b46ac5a15fc230cb1189ca255b88a68f07f03fd9",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 20.64,
    "retest_logged_model_load_seconds": 68.668489,
    "retest_logged_available_KV_GiB": 6.11,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 6.11,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 93600,
    "retest_logged_context_capacity_x": 2.72,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-fp16-tp4-mtp2/service-final.log",
    "source_log_sha256": "7b07a9c59e06465fab4a524d823a544a019ffd6d1da77b716d1da525392aaa3c",
    "sealed_index_sha256": "77ccd32707a78a1145f1917f797db402a47ad46148bc20fb5ff6f1af0a7821aa",
    "source_lines": {
      "model_load": {
        "line": 893,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 23:45:18 [gpu_model_runner.py:4579] Model loading took 20.64 GiB memory and 68.668489 seconds"
      },
      "automatic_KV_budget": {
        "line": 986,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-27 00:02:07 [gpu_worker.py:453] Available KV cache memory: 6.11 GiB"
      },
      "grouped_KV_tokens": {
        "line": 991,
        "text": "(EngineCore_DP0 pid=85) INFO 10-27 00:02:07 [kv_cache_utils.py:1314] GPU KV cache size: 93,600 tokens"
      },
      "context_capacity": {
        "line": 992,
        "text": "(EngineCore_DP0 pid=85) INFO 10-27 00:02:07 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 2.72x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-fp16-tp4-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 27.525005340576172,
  "sampled_peak_VRAM_GiB": 27.55583953857422
}
V620Current Qwen3.8FP16Qwen/Qwen3.8-27BTP2MTP2 NOT RUNNot reportedCapacity blocked; no measured ladder —— — / —10K: — / —Strict: — / — 25.05 / 28.55FP16 parameter extents / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:0a8ae2ddb11ddae09242459cbfdbb56b028bdd1c1a9e02bdd621a76c8a3d9e08
Runtime image config: sha256:a98060dd75e701abc267843ccd7d8513c7baf7b6abb168c6c289bd2dbad5d399
Registry verification
Capacity-limited diagnostic; full benchmark unsupported.
Short-request diagnostic · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256
mkdir -p launch-tools-v620-serve-r5
tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp2-mtp2
GPU_RENDER_NODES=renderD128,renderD129

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-fp16-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Preflight cleanup compatibility
After its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged.
Retained lab Docker command · not portable
docker run -d \
  --name serve-v620-fp16-tp2-mtp2-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --cpuset-mems 0 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/card5:/dev/dri/card5:rwm \
  --device /dev/dri/card6:/dev/dri/card6:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
  -e ROCM_VERSION=7.14.0 \
  -e TORCH_VERSION=2.12.0+rocm7.14.0 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
  -e PYTORCH_ROCM_ARCH=gfx1030 \
  -e GPU_ARCHS=gfx1030 \
  -e AMDGPU_TARGETS=gfx1030 \
  -e CMAKE_HIP_ARCHITECTURES=gfx1030 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp2/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp2/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp2/cache/torch_extensions,dst=/root/.cache/torch_extensions \
  --entrypoint /usr/bin/env \
  sha256:a98060dd75e701abc267843ccd7d8513c7baf7b6abb168c6c289bd2dbad5d399 \
  V756F94_BOUNDARY_MODE=posted \
  DO_NOT_TRACK=1 \
  GPU_DEVICE_ORDINAL=0,1 \
  GPU_MAX_HW_QUEUES=1 \
  HSA_OVERRIDE_GFX_VERSION=10.3.0 \
  OMP_NUM_THREADS=4 \
  TEMP=/tmp/torchinductor_root \
  TMPDIR=/tmp/torchinductor_root \
  TMP=/tmp/torchinductor_root \
  TORCH_BLAS_PREFER_HIPBLASLT=0 \
  TRITON_CACHE_DIR=/root/.triton \
  VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  VLLM_GPU_MEMORY_UTILIZATION=0.95 \
  VLLM_LOGGING_LEVEL=INFO \
  VLLM_PORT=9002 \
  VLLM_RDNA_FORCE_FP16=1 \
  VLLM_WORKER_MULTIPROC_METHOD=spawn \
  NCCL_P2P_DISABLE=0 \
  NCCL_ALGO=Tree \
  NCCL_PROTO=LL \
  NCCL_MIN_NCHANNELS=4 \
  NCCL_MAX_NCHANNELS=4 \
  NCCL_NTHREADS=256 \
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
  HIP_VISIBLE_DEVICES=0,1 \
  CUDA_VISIBLE_DEVICES=0,1 \
  ROCR_VISIBLE_DEVICES=0,1 \
  VLLM_CACHE_ROOT=/root/.cache/vllm \
  PYTORCH_ROCM_ARCH=gfx1030 \
  PYTHONHASHSEED=0 \
  /opt/venv/bin/vllm \
  serve \
  /opt/local-models/qwen38-27b-hf \
  --served-model-name \
  Qwen/Qwen3.8-27B \
  --enable-auto-tool-choice \
  --tool-call-parser \
  qwen3_coder \
  --dtype \
  half \
  --host \
  0.0.0.0 \
  --port \
  9002 \
  --tensor-parallel-size \
  2 \
  --max-model-len \
  131072 \
  --gpu-memory-utilization \
  0.95 \
  --trust-remote-code \
  --generation-config \
  vllm \
  -O=3 \
  --reasoning-parser \
  qwen3 \
  --async-scheduling \
  --hf-overrides \
  '{"architectures":["Qwen3_5ForCausalLM"]}' \
  --language-model-only \
  --max-num-seqs \
  2 \
  --max-num-batched-tokens \
  2048 \
  --mamba-ssm-cache-dtype \
  float32 \
  --no-enable-prefix-caching \
  --no-enforce-eager \
  --attention-backend \
  TRITON_ATTN \
  --disable-custom-all-reduce \
  --seed \
  0 \
  --worker-cls \
  vllm.vnext_qwen38_worker.Worker \
  --speculative-config \
  '{"method":"mtp","num_speculative_tokens":2}'
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Reproduce the capacity failure · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp2-mtp2
GPU_RENDER_NODES=renderD128,renderD129

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-r2/portable/launch.py --config v620-fp16-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-v620-serve-r5.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
Capacity limitation reproducedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: capacity-blocked-reviewed

The unchanged FP16 TP2 MTP2 profile served a short request but could not complete the prescribed 2,000-token startup requests. KV saturation and zero generation were logged before the original 30-minute request timeout. No strict or speed sample exists.

Reviewed capacity outcome
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Startup timeout; KV cache full; no measured ladderVerification · Raw retestEarlier setup attempt retainedAdmission observer repair and replayObserved startup limit and closureNO LADDER
Retained failure
TimeoutError('timed out')
Retest KV budget: 0.360 GiB (automatic, rounded)2,400 grouped KV tokens · 0.08× reported capacityRetest sampled VRAM: 28.554 GiB max/cardRetest memory reports and limits
Evidence and limits

Capacity-blocked under unchanged context131072, FP16 KV and utilization0.95 after bounded reserve-accounting repair. HTTP and 0.40GiB KV reached, but all three startup requests failed to complete; cache100% and preemptions observed. No strict verdict, TPS or measured latency. Exact cohort stopped and evidence retained; no further retry selected.

{
  "source": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/closure.json#$",
  "source_revision": "bd4bcaf2a85f5f2e7620a1383eb258d2429bb28c",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "strict_tokens": null,
  "attention_KV_dtype": "FP16",
  "sampled_VRAM_startup_max_bytes_by_PCI": "{\"0000:16:00.0\": 30658633728, \"0000:19:00.0\": 30656995328}",
  "KV_capacity": "{\"interpretation\": \"Exact captured kv_cache_utils.py1297-1300 logs floor(num_blocks/group_count)*min_block_size; max concurrency uses a different per-group max-memory calculation799-818. Thus 2400 is a grouped capacity indicator, not a separately proven exact single-request ceiling. Earlier473+1926=2399 observation is corroborative, not a complete proof of allocator exhaustion arithmetic.\", \"reported_GiB\": 0.4, \"reported_grouped_tokens\": 2400}",
  "source_evidence_commit": "68f943b",
  "comparison_limit": "Scoped post-draft-sharing reserve cleanup before memory profiling; actual FP16 target/draft tensor identities unchanged. Original 0.95 utilization, 131072 context, FP16 attention KV and 800-block fallback retained. No throughput, strict outcome or completed startup inferred. No number inferred. Originalprofile0.95preserved; no context/KV precision reduction. r0preHTTPfailure andr1failedstartup retained; no further speculative FP16tuning for this cell.",
  "loaded_metadata_sources_by_rank": {
    "0": {
      "path": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-0-pid-103.json",
      "sha256": "1574ea88a3fe9200c54eb968d1a210900afa83e8e7dc4fcbeba0b58e129e95ed"
    },
    "1": {
      "path": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-1-pid-104.json",
      "sha256": "4a31d3561be3db61a8df5134ea6669ef019e85afa15f4076db010f149aae5d06"
    }
  },
  "memory_accounting_limit": "Retained FP16 parameter extents are not all model weights: TP4 projection tensors use lossless integer buffers. All packed-buffer extents include alternate representations. Draft tensors can share target embeddings/head. None of these logical sums is a unique physical allocation or measured memory traffic.",
  "memory_source": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-1-pid-104.json",
  "memory_source_sha256": "4a31d3561be3db61a8df5134ea6669ef019e85afa15f4076db010f149aae5d06",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30659104768,
    "1": 30657032192
  },
  "retest_memory_observation_count": 46,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-fp16-tp2-mtp2/runtime-observations.jsonl",
  "retest_outcome_source": "retest-evidence/v620-fp16-tp2-mtp2/outcome.json",
  "retest_logged_available_KV_GiB": 0.36,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 0.36,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 2400,
  "retest_memory_log_report": {
    "profile": "v620-fp16-tp2-mtp2",
    "image": "sha256:a98060dd75e701abc267843ccd7d8513c7baf7b6abb168c6c289bd2dbad5d399",
    "container": "7da02e7c6d52440d7406b6ae5c7c29a90753fa6e5d16e0fe3656656a0ab9b989",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 25.76,
    "retest_logged_model_load_seconds": 15.149719,
    "retest_logged_available_KV_GiB": 0.36,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 0.36,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 2400,
    "retest_logged_context_capacity_x": 0.08,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-fp16-tp2-mtp2/service-final.log",
    "source_log_sha256": "3b8d72488fed8ca3839b7289454b1ea79619e1f6c0abe3566f286e7a5231fe0a",
    "sealed_index_sha256": "ec69fada416a475bb7eb71d8a77a656e3a1b45000a1dec03b45a57191a86dc0b",
    "source_lines": {
      "model_load": {
        "line": 117,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-27 02:20:55 [gpu_model_runner.py:4604] Model loading took 25.76 GiB memory and 15.149719 seconds"
      },
      "automatic_KV_budget": {
        "line": 167,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-27 02:36:10 [gpu_worker.py:453] Available KV cache memory: 0.36 GiB"
      },
      "grouped_KV_tokens": {
        "line": 170,
        "text": "(EngineCore_DP0 pid=85) INFO 10-27 02:36:10 [kv_cache_utils.py:1314] GPU KV cache size: 2,400 tokens"
      },
      "context_capacity": {
        "line": 171,
        "text": "(EngineCore_DP0 pid=85) INFO 10-27 02:36:10 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 0.08x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-fp16-tp2-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.553516387939453,
  "sampled_peak_VRAM_GiB": 28.553077697753906
}
V620Current Qwen3.88-bit RTNQwen/Qwen3.8-27BTP4MTP off PASS7563 tokens 29.78728.871 1.105 / 68.26310K: 1.107 / 347.487Strict: — / — 5.80 / 23.40packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:74b0db52bb4f4aacf390ec2d18c6d8ff9d6ed403ca3c942fb55bc0bda09cfa3f
Runtime image config: sha256:4f2d919ceed2b8b21cfd4d13661d83d197fbdddcc82e5a799dc4689c8c407794
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256
mkdir -p launch-tools-v620-serve-r5
tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp4-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w8-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Preflight cleanup compatibility
After its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged.
Retained lab Docker command · not portable
docker run -d \
  --name serve-v620-w8-tp4-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --cpuset-mems 0 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/card5:/dev/dri/card5:rwm \
  --device /dev/dri/card6:/dev/dri/card6:rwm \
  --device /dev/dri/card7:/dev/dri/card7:rwm \
  --device /dev/dri/card8:/dev/dri/card8:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
  --device /dev/dri/renderD136:/dev/dri/renderD136:rwm \
  --user root \
  --workdir /tmp \
  -e VNEXT_RTN_W8_FUSED_CTA=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
  -e ROCM_VERSION=7.14.0 \
  -e TORCH_VERSION=2.12.0+rocm7.14.0 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
  -e PYTORCH_ROCM_ARCH=gfx1030 \
  -e GPU_ARCHS=gfx1030 \
  -e AMDGPU_TARGETS=gfx1030 \
  -e CMAKE_HIP_ARCHITECTURES=gfx1030 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e VNEXT_GDN_HEAD32=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp0/cache/torch_extensions,dst=/root/.cache/torch_extensions \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp0/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --entrypoint /usr/bin/env \
  sha256:4f2d919ceed2b8b21cfd4d13661d83d197fbdddcc82e5a799dc4689c8c407794 \
  V756F94_BOUNDARY_MODE=posted \
  DO_NOT_TRACK=1 \
  GPU_DEVICE_ORDINAL=0,1,2,3 \
  GPU_MAX_HW_QUEUES=1 \
  HSA_OVERRIDE_GFX_VERSION=10.3.0 \
  OMP_NUM_THREADS=4 \
  TEMP=/tmp/torchinductor_root \
  TMPDIR=/tmp/torchinductor_root \
  TMP=/tmp/torchinductor_root \
  TORCH_BLAS_PREFER_HIPBLASLT=0 \
  TRITON_CACHE_DIR=/root/.triton \
  VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  VLLM_GPU_MEMORY_UTILIZATION=0.95 \
  VLLM_LOGGING_LEVEL=INFO \
  VLLM_PORT=9002 \
  VLLM_RDNA_FORCE_FP16=1 \
  VLLM_WORKER_MULTIPROC_METHOD=spawn \
  NCCL_P2P_DISABLE=0 \
  NCCL_ALGO=Tree \
  NCCL_PROTO=LL \
  NCCL_MIN_NCHANNELS=4 \
  NCCL_MAX_NCHANNELS=4 \
  NCCL_NTHREADS=256 \
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
  HIP_VISIBLE_DEVICES=0,1,2,3 \
  CUDA_VISIBLE_DEVICES=0,1,2,3 \
  ROCR_VISIBLE_DEVICES=0,1,2,3 \
  VLLM_CACHE_ROOT=/root/.cache/vllm \
  PYTORCH_ROCM_ARCH=gfx1030 \
  PYTHONHASHSEED=0 \
  /opt/venv/bin/python \
  -m \
  vllm.vnext_rtn_entry \
  serve \
  /opt/local-models/qwen38-27b-hf \
  --served-model-name \
  Qwen/Qwen3.8-27B \
  --enable-auto-tool-choice \
  --tool-call-parser \
  qwen3_coder \
  --dtype \
  half \
  --host \
  0.0.0.0 \
  --port \
  9002 \
  --tensor-parallel-size \
  4 \
  --max-model-len \
  131072 \
  --gpu-memory-utilization \
  0.95 \
  --trust-remote-code \
  --generation-config \
  vllm \
  -O=3 \
  --reasoning-parser \
  qwen3 \
  --async-scheduling \
  --hf-overrides \
  '{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":8,"group_size":128}}' \
  --language-model-only \
  --max-num-seqs \
  2 \
  --max-num-batched-tokens \
  2048 \
  --mamba-ssm-cache-dtype \
  float32 \
  --no-enable-prefix-caching \
  --no-enforce-eager \
  --attention-backend \
  TRITON_ATTN \
  --disable-custom-all-reduce \
  --seed \
  0 \
  --worker-cls \
  vllm.vnext_rtn_worker.Worker \
  --quantization \
  vnext_rtn
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-r4.tar.gz.sha256
mkdir -p launch-tools-v620-r4
tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp4-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-v620-r4/portable/launch.py --config v620-w8-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-v620-serve-r5.tar.gz · SHA-256
launch-tools-v620-r4.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
Portable observer compatibility
The evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged.
All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 30.220 · eligible10K 29.262 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 18422,
    "content_chars": 16775,
    "content_has_think_close": false,
    "elapsed_seconds": 651.1220705858432,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T20:38:30Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.017328426940366626,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 16775,
    "qwen_answer_text_sha256": "39f0a17de9fb54affb4a939aedfb42dae34a7ed41e547fe64896061e5023ed9b",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 16775,
    "qwen_visible_text_sha256": "39f0a17de9fb54affb4a939aedfb42dae34a7ed41e547fe64896061e5023ed9b",
    "raw_has_think_close": false,
    "reasoning_chars": 58239,
    "text_chars": 75015,
    "text_sha256": "65708cbbfc593557091707304990a3d2abc03c001ff0e9e842c59998f12be5b7",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 649.9961330560036,
    "vllm_decode_tps": 28.341707070452312,
    "vllm_generation_tokens_delta": 18422.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.1034011258743703,
    "vllm_prefill_tps": 428.67456712551353,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 28.292697839937933
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 67.31212564720772,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T20:39:37Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008899811888113618,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 5641,
    "text_chars": 5641,
    "text_sha256": "58897a6606b49de4b7c6d917b21cdd4091c9547ccfe4d8a805378ae57fafc003",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 66.1812575191725,
    "vllm_decode_tps": 30.220036230357305,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.1045551309362054,
    "vllm_prefill_tps": 428.2267011870126,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 29.712328659508987
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 342.86440511001274,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T20:45:20Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.018551325891166925,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 34008,
    "text_chars": 34008,
    "text_sha256": "8ec2c8505bb25ab3e4c7075862729060467b5a59839beada37ac51d9975fc583",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 341.7400923699606,
    "vllm_decode_tps": 29.262004146631444,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.1042267300654203,
    "vllm_prefill_tps": 428.354057297614,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 29.16604888393522
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS7,745 natural tokens2K 30.46210K 29.500 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T17:48:33.669083559Z",
      "FinishedAt": "2026-10-26T18:39:14.260880813Z"
    },
    "devices_after": [
      {
        "card": 5,
        "render": 133,
        "pci": "0000:16:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:19:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 7,
        "render": 135,
        "pci": "0000:1c:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 8,
        "render": 136,
        "pci": "0000:1f:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T18:39:14.398926+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 30.461981881258904,
  "retest_10K_TPS": 29.50011363293815,
  "retest_strict_tokens": 7745,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 7745,
      "elapsed_seconds": 261.14458458300214,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 260.0247855489142,
      "vllm_decode_tps": 29.785622103870786,
      "vllm_generation_tokens_delta": 7745.0,
      "vllm_prefill_seconds_delta": 1.1050167970824987,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 66.77428013109602,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 65.65560992702376,
      "vllm_decode_tps": 30.461981881258904,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 1.1085248430026695,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 340.1001898489194,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 338.98174510197714,
      "vllm_decode_tps": 29.50011363293815,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 1.1070090770954266,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 1.1085248430026695,
  "retest_2K_total_seconds": 66.77428013109602,
  "retest_10K_prefill_seconds": 1.1070090770954266,
  "retest_10K_total_seconds": 340.1001898489194,
  "retest_strict_prefill_seconds": 1.1050167970824987,
  "retest_strict_total_seconds": 261.14458458300214,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": "Complete record: chart-details/row-16-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 25104838656,
    "1": 25121308672,
    "2": 25121206272,
    "3": 25071071232
  },
  "retest_memory_observation_count": 49,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-w8-tp4-mtp0/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 13.55,
  "retest_logged_model_load_seconds": 69.213373,
  "retest_logged_available_KV_GiB": 13.67,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 13.67,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 223440,
  "retest_logged_context_capacity_x": 6.68,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "v620-w8-tp4-mtp0",
    "image": "sha256:4f2d919ceed2b8b21cfd4d13661d83d197fbdddcc82e5a799dc4689c8c407794",
    "container": "86d58c7ae420738f87e752aacaf340b66c6c2aaf59dc74b10e8688076ccb84d4",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 13.55,
    "retest_logged_model_load_seconds": 69.213373,
    "retest_logged_available_KV_GiB": 13.67,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 13.67,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 223440,
    "retest_logged_context_capacity_x": 6.68,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w8-tp4-mtp0/service-final.log",
    "source_log_sha256": "726feb8d07ee3d1196f5ae64c801549b862215683eec091a52bf1c101766150e",
    "sealed_index_sha256": "24a7a4434d6a3c8345a672f0f4aba71a5b4211d8aaa88c3563efe7c930366fab",
    "source_lines": {
      "model_load": {
        "line": 120,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 17:51:30 [gpu_model_runner.py:4579] Model loading took 13.55 GiB memory and 69.213373 seconds"
      },
      "automatic_KV_budget": {
        "line": 139,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 17:55:28 [gpu_worker.py:453] Available KV cache memory: 13.67 GiB"
      },
      "grouped_KV_tokens": {
        "line": 145,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 17:55:29 [kv_cache_utils.py:1314] GPU KV cache size: 223,440 tokens"
      },
      "context_capacity": {
        "line": 146,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 17:55:29 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.68x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w8-tp4-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 23.396041870117188
}
Complete retest loaded admission
Retest KV budget: 13.670 GiB (automatic, rounded)223,440 grouped KV tokens · 6.68× reported capacityRetest sampled VRAM: 23.396 GiB max/cardRetest memory reports and limits
Evidence and limits

strict-valid W8 ordered-FP32 baseline

{
  "strict_tokens": 7563,
  "source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.full_model_results.RTN8_fused_CTA",
  "source_revision": "8668c2fc8e377c2990edd57988a85a03b87750bd",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "comparison_limit": "One admitted sample, with quantized projection/TP routes, Head32/compiler mode and root-SSD caches explicitly different from the preserved FP16 reference. No FP16 qualification or repeatability claim.",
  "latency_sources": "[{\"path\": \"/home/joe/PhoneSync/Documents/gfx1030/.work/quantized-baselines/rtn8-fused-service-r0/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/summary.json\", \"sha256\": \"9564dacafa3c6f496df67ca9700658dc9f4e956124f24a361206b69ecc1dd12e\"}, {\"path\": \"/home/joe/PhoneSync/Documents/gfx1030/.work/quantized-baselines/rtn8-fused-service-r0/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/summary.json\", \"sha256\": \"56da2ba97b816e39f869fb067046b95ddc4e39c86cca2603988c4c7c9b7d3560\"}]",
  "sampled_peak_VRAM_bytes_by_PCI": {
    "0000:16:00.0": 25104637952,
    "0000:19:00.0": 25121427456,
    "0000:1c:00.0": 25121427456,
    "0000:1f:00.0": 25071353856
  },
  "memory_accounting_limit": "Packed target matrices exclude FP16 embeddings/head/norm/recurrent weights. Sampled physical VRAM is not a guaranteed continuous peak.",
  "memory_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#full_model_results.RTN8_fused_CTA.memory",
  "memory_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
  "retest_strict_tokens": 7745,
  "retest_MTP_accounting_by_tier": {},
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 25104838656,
    "1": 25121308672,
    "2": 25121206272,
    "3": 25071071232
  },
  "retest_memory_observation_count": 49,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-w8-tp4-mtp0/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 13.67,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 13.67,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 223440,
  "retest_memory_log_report": {
    "profile": "v620-w8-tp4-mtp0",
    "image": "sha256:4f2d919ceed2b8b21cfd4d13661d83d197fbdddcc82e5a799dc4689c8c407794",
    "container": "86d58c7ae420738f87e752aacaf340b66c6c2aaf59dc74b10e8688076ccb84d4",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 13.55,
    "retest_logged_model_load_seconds": 69.213373,
    "retest_logged_available_KV_GiB": 13.67,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 13.67,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 223440,
    "retest_logged_context_capacity_x": 6.68,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w8-tp4-mtp0/service-final.log",
    "source_log_sha256": "726feb8d07ee3d1196f5ae64c801549b862215683eec091a52bf1c101766150e",
    "sealed_index_sha256": "24a7a4434d6a3c8345a672f0f4aba71a5b4211d8aaa88c3563efe7c930366fab",
    "source_lines": {
      "model_load": {
        "line": 120,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 17:51:30 [gpu_model_runner.py:4579] Model loading took 13.55 GiB memory and 69.213373 seconds"
      },
      "automatic_KV_budget": {
        "line": 139,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 17:55:28 [gpu_worker.py:453] Available KV cache memory: 13.67 GiB"
      },
      "grouped_KV_tokens": {
        "line": 145,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 17:55:29 [kv_cache_utils.py:1314] GPU KV cache size: 223,440 tokens"
      },
      "context_capacity": {
        "line": 146,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 17:55:29 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.68x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w8-tp4-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 23.396041870117188,
  "sampled_peak_VRAM_GiB": 23.39615249633789
}
V620Current Qwen3.88-bit RTNQwen/Qwen3.8-27BTP2MTP off PASS8188 tokens 20.93119.987 0.773 / 96.33810K: 0.774 / 501.120Strict: 0.774 / 406.285 11.60 / 17.69packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:254536b65a31bbd16a2a864818b58581b4a6e0480d431c990dc7a51be38e016c
Runtime image config: sha256:95626d8b1e5b583a5348b5e7c9a0cd91840877eb5e3407e916f53298f80cf9f6
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256
mkdir -p launch-tools-v620-serve-r5
tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp2-mtp0
GPU_RENDER_NODES=renderD128,renderD129

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w8-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Preflight cleanup compatibility
After its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged.
Retained lab Docker command · not portable
docker run -d \
  --name serve-v620-w8-tp2-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --cpuset-mems 0 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/card5:/dev/dri/card5:rwm \
  --device /dev/dri/card6:/dev/dri/card6:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --user root \
  --workdir /tmp \
  -e VNEXT_RTN_W8_FUSED_CTA=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
  -e ROCM_VERSION=7.14.0 \
  -e TORCH_VERSION=2.12.0+rocm7.14.0 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
  -e PYTORCH_ROCM_ARCH=gfx1030 \
  -e GPU_ARCHS=gfx1030 \
  -e AMDGPU_TARGETS=gfx1030 \
  -e CMAKE_HIP_ARCHITECTURES=gfx1030 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e VNEXT_GDN_HEAD32=1 \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp0/cache/torch_extensions,dst=/root/.cache/torch_extensions \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp0/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --entrypoint /usr/bin/env \
  sha256:95626d8b1e5b583a5348b5e7c9a0cd91840877eb5e3407e916f53298f80cf9f6 \
  V756F94_BOUNDARY_MODE=posted \
  DO_NOT_TRACK=1 \
  GPU_DEVICE_ORDINAL=0,1 \
  GPU_MAX_HW_QUEUES=1 \
  HSA_OVERRIDE_GFX_VERSION=10.3.0 \
  OMP_NUM_THREADS=4 \
  TEMP=/tmp/torchinductor_root \
  TMPDIR=/tmp/torchinductor_root \
  TMP=/tmp/torchinductor_root \
  TORCH_BLAS_PREFER_HIPBLASLT=0 \
  TRITON_CACHE_DIR=/root/.triton \
  VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  VLLM_GPU_MEMORY_UTILIZATION=0.95 \
  VLLM_LOGGING_LEVEL=INFO \
  VLLM_PORT=9002 \
  VLLM_RDNA_FORCE_FP16=1 \
  VLLM_WORKER_MULTIPROC_METHOD=spawn \
  NCCL_P2P_DISABLE=0 \
  NCCL_ALGO=Tree \
  NCCL_PROTO=LL \
  NCCL_MIN_NCHANNELS=4 \
  NCCL_MAX_NCHANNELS=4 \
  NCCL_NTHREADS=256 \
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
  HIP_VISIBLE_DEVICES=0,1 \
  CUDA_VISIBLE_DEVICES=0,1 \
  ROCR_VISIBLE_DEVICES=0,1 \
  VLLM_CACHE_ROOT=/root/.cache/vllm \
  PYTORCH_ROCM_ARCH=gfx1030 \
  PYTHONHASHSEED=0 \
  /opt/venv/bin/python \
  -m \
  vllm.vnext_rtn_entry \
  serve \
  /opt/local-models/qwen38-27b-hf \
  --served-model-name \
  Qwen/Qwen3.8-27B \
  --enable-auto-tool-choice \
  --tool-call-parser \
  qwen3_coder \
  --dtype \
  half \
  --host \
  0.0.0.0 \
  --port \
  9002 \
  --tensor-parallel-size \
  2 \
  --max-model-len \
  131072 \
  --gpu-memory-utilization \
  0.95 \
  --trust-remote-code \
  --generation-config \
  vllm \
  -O=3 \
  --reasoning-parser \
  qwen3 \
  --async-scheduling \
  --hf-overrides \
  '{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":8,"group_size":128}}' \
  --language-model-only \
  --max-num-seqs \
  2 \
  --max-num-batched-tokens \
  2048 \
  --mamba-ssm-cache-dtype \
  float32 \
  --no-enable-prefix-caching \
  --no-enforce-eager \
  --attention-backend \
  TRITON_ATTN \
  --disable-custom-all-reduce \
  --seed \
  0 \
  --worker-cls \
  vllm.vnext_rtn_worker.Worker \
  --quantization \
  vnext_rtn
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-r4.tar.gz.sha256
mkdir -p launch-tools-v620-r4
tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp2-mtp0
GPU_RENDER_NODES=renderD128,renderD129

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-v620-r4/portable/launch.py --config v620-w8-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-v620-serve-r5.tar.gz · SHA-256
launch-tools-v620-r4.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
Portable observer compatibility
The evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged.
All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict FAIL2K 20.633 · diagnostic / ineligible10K 19.706 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 16516,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 868.9049477491062,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T22:14:21Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.018124318914487958,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 31279,
    "text_chars": 31279,
    "text_sha256": "67a2ad5a629db0dc2d1d7f6cac288c0ffd40df34d177b02d981357ccb3f1696d",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 868.1123461599927,
    "vllm_decode_tps": 19.025187319425715,
    "vllm_generation_tokens_delta": 16516.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.7737751340027899,
    "vllm_prefill_tps": 611.2887054836458,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 19.00783283923588
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 97.72851502406411,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T22:16:01Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.018401601118966937,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 8011,
    "text_chars": 8011,
    "text_sha256": "c36609287d3c6bbbd2f917533763502b96d9181ac5a028ad6ceaf0ab8978787f",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 96.9333495539613,
    "vllm_decode_tps": 20.632733823838727,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.7728608581237495,
    "vllm_prefill_tps": 612.0118453770418,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 20.464856132394228
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 508.2517163429875,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T22:24:30Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.020981378853321075,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 21006,
    "text_chars": 21006,
    "text_sha256": "1bd4ca5ff1900fcd998b29a09350f8aa5cdbc7fb321d5b4e835c123254180511",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 507.4579654929694,
    "vllm_decode_tps": 19.70606568424936,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.7743516750633717,
    "vllm_prefill_tps": 610.8335724350186,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 19.675290173052012
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS9,474 natural tokens2K 20.89810K 19.958 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T18:41:25.776124549Z",
      "FinishedAt": "2026-10-26T19:39:36.891683362Z"
    },
    "devices_after": [
      {
        "card": 5,
        "render": 133,
        "pci": "0000:16:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:19:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T19:39:37.029335+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 20.897708334429417,
  "retest_10K_TPS": 19.95841850159981,
  "retest_strict_tokens": 9474,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 9474,
      "elapsed_seconds": 473.81853826798033,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 473.03272304392885,
      "vllm_decode_tps": 20.028212718637192,
      "vllm_generation_tokens_delta": 9474.0,
      "vllm_prefill_seconds_delta": 0.7731824229704216,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 96.49031426000874,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 95.70427378895693,
      "vllm_decode_tps": 20.897708334429417,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 0.771934352000244,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 501.828497888986,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 501.0417032390833,
      "vllm_decode_tps": 19.95841850159981,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 0.774901254917495,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 0.771934352000244,
  "retest_2K_total_seconds": 96.49031426000874,
  "retest_10K_prefill_seconds": 0.774901254917495,
  "retest_10K_total_seconds": 501.828497888986,
  "retest_strict_prefill_seconds": 0.7731824229704216,
  "retest_strict_total_seconds": 473.81853826798033,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": "Complete record: chart-details/row-17-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 28108984320,
    "1": 28109066240
  },
  "retest_memory_observation_count": 56,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-w8-tp2-mtp0/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 25.74,
  "retest_logged_model_load_seconds": 120.96945,
  "retest_logged_available_KV_GiB": 0.89,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 0.89,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 7056,
  "retest_logged_context_capacity_x": 0.22,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "v620-w8-tp2-mtp0",
    "image": "sha256:95626d8b1e5b583a5348b5e7c9a0cd91840877eb5e3407e916f53298f80cf9f6",
    "container": "c3d8022dd7ce5fa8e2e24216effa6f2440fdfff5db6759f71cebf69280de0bc9",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 25.74,
    "retest_logged_model_load_seconds": 120.96945,
    "retest_logged_available_KV_GiB": 0.89,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 0.89,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 7056,
    "retest_logged_context_capacity_x": 0.22,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w8-tp2-mtp0/service-final.log",
    "source_log_sha256": "76f146dedd1427413c72a2fddf5bb88dc8157ba19bbd406d8e4f3a661b4da0eb",
    "sealed_index_sha256": "b39dd2de284da65749c1d6bee3a8772a4ccc714135b8ce6dc83ff8ec1f7d42e1",
    "source_lines": {
      "model_load": {
        "line": 92,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 18:45:06 [gpu_model_runner.py:4598] Model loading took 25.74 GiB memory and 120.969450 seconds"
      },
      "automatic_KV_budget": {
        "line": 105,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 18:48:53 [gpu_worker.py:453] Available KV cache memory: 0.89 GiB"
      },
      "grouped_KV_tokens": {
        "line": 107,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 18:48:53 [kv_cache_utils.py:1314] GPU KV cache size: 7,056 tokens"
      },
      "context_capacity": {
        "line": 108,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 18:48:53 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 0.22x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w8-tp2-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 26.178607940673828
}
Complete retest loaded admission
Retest KV budget: 0.890 GiB (automatic, rounded)7,056 grouped KV tokens · 0.22× reported capacityRetest sampled VRAM: 26.179 GiB max/cardRetest memory reports and limits
Evidence and limits

Completed, independently reviewed ordered-FP32 W8 TP2 MTP-off matrix cell; one eligible sample, no promotion or FP16 qualification.

{
  "strict_tokens": 8188,
  "source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.FP16_W8_TP_MTP_matrix.cells[3].result",
  "source_revision": "1970f7c0fcbce8645012e1aa24493e3abb4fc243",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "attention_KV_dtype": "FP16",
  "memory_accounting_limit": "Packed weight bytes exclude retained FP16 embeddings/head/norm/recurrent weights. Logical KV views may alias; allocator metrics, runtime capacity and sampled VRAM are distinct.",
  "sampled_peak_VRAM_bytes_by_PCI": "{\"0000:16:00.0\": 18992091136, \"0000:19:00.0\": 18992078848}",
  "KV_capacity": "{\"available_GiB\": 0.89, \"concurrency_at131072\": 0.22, \"tokens\": 7056}",
  "attention_KV_logical_view_bytes_per_rank": 950534144,
  "attention_KV_logical_view_shape": "[37, 2, 784, 2, 256]",
  "post_KV_allocator_allocated_bytes_per_rank": 16209667072,
  "post_KV_allocator_reserved_bytes_per_rank": 17171480576,
  "source_evidence_commit": "85aa7a3",
  "comparison_limit": "Selected U ordered-FP32 runtime extended only for TP2/MTP shapes; old U arithmetic preserved, M>32 retains original reconstruction. Native candidate45281e2c and source1970f7c are admitted. | TP2 reshards and ordinary decoder/fallback differ from historical TP4 U; no isolated TP or kernel-gain attribution. | Fresh private rootSSD caches, FP16 activation/attentionKV, context131072 and memoryutil0.95 retained. | Runtime reports7056 GPUKVtokens/0.22x at configured131072; full-context residency is not demonstrated. | One unchangedcohort/oneofficial2Ksample, no repeatability orFP16promotion. Strict alone correctness;engineeringreplay not an additionalgate. | NewTP2shards and ordinarydecoder/fallback differ fromTP4Ureference; no isolatedTPorUgain claim. | Packed12,460,636,160B/rank excludes retainedFP16embedding/head/norm/recurrentweights. LogicalKVviews950,534,144B/rank mayalias; allocator and observedVRAM are distinct. | Runtime0.89GiBavailableKV/7056tokens/0.22x atconfigured131072; no residentfullcontextcapacity demonstrated. | Generated source linkage and mapped DSO identities support route admission; no per-projection dynamic kernel trace was captured.",
  "recorded_configuration_details": {
    "source": "1970f7c0fcbce8645012e1aa24493e3abb4fc243",
    "image": "sha256:95626d8b1e5b583a5348b5e7c9a0cd91840877eb5e3407e916f53298f80cf9f6",
    "strict_valid": true,
    "strict_tokens": 8188,
    "strict_diagnostic_TPS": 20.1923964270491,
    "official_fixed2K_TPS": 20.930590077305382,
    "diagnostic_fixed10K_TPS": 19.986683546664402,
    "metrics": {
      "strict": {
        "completion_tokens": 8188,
        "finish_reason": "stop",
        "elapsed_seconds": 406.28512426803354,
        "vllm_prefill_seconds_delta": 0.7738519569975324,
        "vllm_decode_seconds_delta": 405.49917042197194,
        "vllm_decode_tps": 20.1923964270491,
        "vllm_generation_tokens_delta": 8188.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "ce8f2a7ca5918c1cbdcc0df5a649d11b9df57dfd8a6319a8ba7852a040770d55",
        "server_e2e_seconds": 406.28012442588806
      },
      "fixed2K": {
        "completion_tokens": 2000,
        "finish_reason": "length",
        "elapsed_seconds": 96.33843383798376,
        "vllm_prefill_seconds_delta": 0.7734992449986748,
        "vllm_decode_seconds_delta": 95.55392335396027,
        "vllm_decode_tps": 20.930590077305382,
        "vllm_generation_tokens_delta": 2000.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "e608af506987d81b2a428e2d52738da1d72794e39cc6ea0df9504708305cf522",
        "server_e2e_seconds": 96.3344099521637
      },
      "fixed10K": {
        "completion_tokens": 10000,
        "finish_reason": "length",
        "elapsed_seconds": 501.12039932300104,
        "vllm_prefill_seconds_delta": 0.7739897519932128,
        "vllm_decode_seconds_delta": 500.3331331409863,
        "vllm_decode_tps": 19.986683546664402,
        "vllm_generation_tokens_delta": 10000.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "2d126341314c81e523f5197c8113887a1e7e43437821c325905f0ca620bd4002",
        "server_e2e_seconds": 501.1148111820221
      }
    },
    "weight_dtype": "symmetric RTN W8 in GPTQ-v2 layout; group128, ties-to-even, zero128",
    "activation_dtype": "FP16",
    "attention_KV_dtype": "FP16",
    "TP": 2,
    "MTP_tokens": 0,
    "quantized_modules_per_rank": 256,
    "packed_component_bytes_per_rank": {
      "qweight": 12175278080,
      "qzeros": 95119360,
      "scales": 190238720,
      "g_idx": 0
    },
    "packed_weight_bytes_per_rank": 12460636160,
    "sampled_peak_VRAM_bytes": {
      "0000:16:00.0": 18992091136,
      "0000:19:00.0": 18992078848
    },
    "KV_capacity": {
      "available_GiB": 0.89,
      "tokens": 7056,
      "concurrency_at131072": 0.22
    },
    "configured_context": 131072,
    "complete_ladders": 1,
    "sample_count": 1,
    "repeatability_established": false,
    "FP16_release_qualification": false,
    "archive": {
      "path": "/home/ai/vnext-v620-qwen38-rtn8-tp2-service-r0-20261023/raw-evidence.tar.gz",
      "bytes": 983825,
      "sha256": "c55b028de5f3d82086171c113b2b999cb779437ae2268f85761e0c0d0e124f7e",
      "files": 238,
      "index_sha256": "bcf623f7dda6961239239f8a905e5bbe745f060c694213f0509f126d2e5b8ffa"
    },
    "local_primary_record": ".work/fp16-w8-tp-mtp-matrix/w8-tp2-service-r0/sealed-mirror/retained/evidence/measurement-closure.json",
    "review": {
      "path": ".work/fp16-w8-tp-mtp-matrix/reviews/W8-TP2-final-cell-review.json",
      "sha256": "3311aa6971d18b9b8b150197d05918113dc1c8b7d5f1ad03c9319363391cc8fd"
    },
    "closure": {
      "container_id": "f7c859ecdb830c729b8d58601c0e285645014069254c8dc49aba1ec0e0d98f0d",
      "image_id": "sha256:95626d8b1e5b583a5348b5e7c9a0cd91840877eb5e3407e916f53298f80cf9f6",
      "exit_code": 0,
      "OOMKilled": false,
      "restart_count": 0,
      "bank_idle": true,
      "devices": [
        {
          "pci": "0000:16:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        },
        {
          "pci": "0000:19:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        }
      ],
      "started_at": "2026-10-23T11:54:59.718906406Z",
      "finished_at": "2026-10-23T12:52:52.033932533Z",
      "closed_at": "2026-10-23T12:52:52.186527+00:00",
      "kernel_fault_matches": [],
      "startup_requests_completed_and_drained": 3,
      "official_phase_samples": 11,
      "complete_ladders": 1,
      "strict_valid": true,
      "official_fixed_2000_tps": 20.930590077305382,
      "qualifying_ladders": 0,
      "FP16_qualification": false,
      "baseline_kind": "RTN8"
    },
    "final_counters": {
      "vllm:prompt_tokens_total": 6622,
      "vllm:generation_tokens_total": 42188,
      "vllm:request_decode_time_seconds_count": 14,
      "vllm:request_prefill_time_seconds_count": 14,
      "vllm:num_requests_running": 0,
      "vllm:num_requests_waiting": 0
    },
    "runtime_differences": [
      "Selected U ordered-FP32 runtime extended only for TP2/MTP shapes; old U arithmetic preserved, M>32 retains original reconstruction. Native candidate45281e2c and source1970f7c are admitted.",
      "TP2 reshards and ordinary decoder/fallback differ from historical TP4 U; no isolated TP or kernel-gain attribution.",
      "Fresh private rootSSD caches, FP16 activation/attentionKV, context131072 and memoryutil0.95 retained.",
      "Runtime reports7056 GPUKVtokens/0.22x at configured131072; full-context residency is not demonstrated."
    ],
    "limits": [
      "One unchangedcohort/oneofficial2Ksample, no repeatability orFP16promotion. Strict alone correctness;engineeringreplay not an additionalgate.",
      "NewTP2shards and ordinarydecoder/fallback differ fromTP4Ureference; no isolatedTPorUgain claim.",
      "Packed12,460,636,160B/rank excludes retainedFP16embedding/head/norm/recurrentweights. LogicalKVviews950,534,144B/rank mayalias; allocator and observedVRAM are distinct.",
      "Runtime0.89GiBavailableKV/7056tokens/0.22x atconfigured131072; no residentfullcontextcapacity demonstrated.",
      "Generated source linkage and mapped DSO identities support route admission; no per-projection dynamic kernel trace was captured."
    ]
  },
  "detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
  "detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
  "detail_source_pointer": "FP16_W8_TP_MTP_matrix.cells[3]",
  "retest_strict_tokens": 9474,
  "retest_MTP_accounting_by_tier": {},
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 28108984320,
    "1": 28109066240
  },
  "retest_memory_observation_count": 56,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-w8-tp2-mtp0/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 0.89,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 0.89,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 7056,
  "retest_memory_log_report": {
    "profile": "v620-w8-tp2-mtp0",
    "image": "sha256:95626d8b1e5b583a5348b5e7c9a0cd91840877eb5e3407e916f53298f80cf9f6",
    "container": "c3d8022dd7ce5fa8e2e24216effa6f2440fdfff5db6759f71cebf69280de0bc9",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 25.74,
    "retest_logged_model_load_seconds": 120.96945,
    "retest_logged_available_KV_GiB": 0.89,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 0.89,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 7056,
    "retest_logged_context_capacity_x": 0.22,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w8-tp2-mtp0/service-final.log",
    "source_log_sha256": "76f146dedd1427413c72a2fddf5bb88dc8157ba19bbd406d8e4f3a661b4da0eb",
    "sealed_index_sha256": "b39dd2de284da65749c1d6bee3a8772a4ccc714135b8ce6dc83ff8ec1f7d42e1",
    "source_lines": {
      "model_load": {
        "line": 92,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 18:45:06 [gpu_model_runner.py:4598] Model loading took 25.74 GiB memory and 120.969450 seconds"
      },
      "automatic_KV_budget": {
        "line": 105,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 18:48:53 [gpu_worker.py:453] Available KV cache memory: 0.89 GiB"
      },
      "grouped_KV_tokens": {
        "line": 107,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 18:48:53 [kv_cache_utils.py:1314] GPU KV cache size: 7,056 tokens"
      },
      "context_capacity": {
        "line": 108,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 18:48:53 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 0.22x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w8-tp2-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 26.178607940673828,
  "sampled_peak_VRAM_GiB": 17.687763214111328
}
V620Current Qwen3.88-bit RTNQwen/Qwen3.8-27BTP4MTP22K accepted 1094 / 1810 (60.44%) PASS8037 tokens 32.93025.672 1.133 / 61.88010K: 1.148 / 390.687Strict: 1.133 / 293.352 5.90 / 28.43packed target/draft projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:e3ec7a663ee96460173b72b3be0fb6e5f4c77ab7a32c93b4824d60dcead4bde1
Runtime image config: sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256
mkdir -p launch-tools-v620-serve-r5
tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp4-mtp2
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w8-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Preflight cleanup compatibility
After its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged.
Retained lab Docker command · not portable
docker run -d \
  --name serve-v620-w8-tp4-mtp2-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --cpuset-mems 0 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/card5:/dev/dri/card5:rwm \
  --device /dev/dri/card6:/dev/dri/card6:rwm \
  --device /dev/dri/card7:/dev/dri/card7:rwm \
  --device /dev/dri/card8:/dev/dri/card8:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
  --device /dev/dri/renderD136:/dev/dri/renderD136:rwm \
  --user root \
  --workdir /tmp \
  -e VNEXT_RTN_W8_NATIVE_DISPATCH=1 \
  -e VNEXT_RTN_W8_FUSED_CTA=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
  -e ROCM_VERSION=7.14.0 \
  -e TORCH_VERSION=2.12.0+rocm7.14.0 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
  -e PYTORCH_ROCM_ARCH=gfx1030 \
  -e GPU_ARCHS=gfx1030 \
  -e AMDGPU_TARGETS=gfx1030 \
  -e CMAKE_HIP_ARCHITECTURES=gfx1030 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e VNEXT_GDN_HEAD32=1 \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp2/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp2/cache/torch_extensions,dst=/root/.cache/torch_extensions \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp2/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
  --entrypoint /usr/bin/env \
  sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567 \
  V756F94_BOUNDARY_MODE=posted \
  DO_NOT_TRACK=1 \
  GPU_DEVICE_ORDINAL=0,1,2,3 \
  GPU_MAX_HW_QUEUES=1 \
  HSA_OVERRIDE_GFX_VERSION=10.3.0 \
  OMP_NUM_THREADS=4 \
  TEMP=/tmp/torchinductor_root \
  TMPDIR=/tmp/torchinductor_root \
  TMP=/tmp/torchinductor_root \
  TORCH_BLAS_PREFER_HIPBLASLT=0 \
  TRITON_CACHE_DIR=/root/.triton \
  VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  VLLM_GPU_MEMORY_UTILIZATION=0.95 \
  VLLM_LOGGING_LEVEL=INFO \
  VLLM_PORT=9002 \
  VLLM_RDNA_FORCE_FP16=1 \
  VLLM_WORKER_MULTIPROC_METHOD=spawn \
  NCCL_P2P_DISABLE=0 \
  NCCL_ALGO=Tree \
  NCCL_PROTO=LL \
  NCCL_MIN_NCHANNELS=4 \
  NCCL_MAX_NCHANNELS=4 \
  NCCL_NTHREADS=256 \
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
  HIP_VISIBLE_DEVICES=0,1,2,3 \
  CUDA_VISIBLE_DEVICES=0,1,2,3 \
  ROCR_VISIBLE_DEVICES=0,1,2,3 \
  VLLM_CACHE_ROOT=/root/.cache/vllm \
  PYTORCH_ROCM_ARCH=gfx1030 \
  PYTHONHASHSEED=0 \
  /opt/venv/bin/python \
  -m \
  vllm.vnext_rtn_entry \
  serve \
  /opt/local-models/qwen38-27b-hf \
  --served-model-name \
  Qwen/Qwen3.8-27B \
  --enable-auto-tool-choice \
  --tool-call-parser \
  qwen3_coder \
  --dtype \
  half \
  --host \
  0.0.0.0 \
  --port \
  9002 \
  --tensor-parallel-size \
  4 \
  --speculative-config \
  '{"method":"mtp","num_speculative_tokens":2}' \
  --max-model-len \
  131072 \
  --gpu-memory-utilization \
  0.95 \
  --trust-remote-code \
  --generation-config \
  vllm \
  -O=3 \
  --reasoning-parser \
  qwen3 \
  --async-scheduling \
  --hf-overrides \
  '{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":8,"group_size":128}}' \
  --language-model-only \
  --max-num-seqs \
  2 \
  --max-num-batched-tokens \
  2048 \
  --mamba-ssm-cache-dtype \
  float32 \
  --no-enable-prefix-caching \
  --no-enforce-eager \
  --attention-backend \
  TRITON_ATTN \
  --disable-custom-all-reduce \
  --seed \
  0 \
  --worker-cls \
  vllm.vnext_rtn_worker.Worker \
  --quantization \
  vnext_rtn
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-r4.tar.gz.sha256
mkdir -p launch-tools-v620-r4
tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp4-mtp2
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-v620-r4/portable/launch.py --config v620-w8-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-v620-serve-r5.tar.gz · SHA-256
launch-tools-v620-r4.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
Portable observer compatibility
The evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged.
All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict FAIL2K 41.076 · diagnostic / ineligible10K 28.673 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 4246,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 114.82530898298137,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T23:31:49Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.013302841922268271,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 7640,
    "text_chars": 7640,
    "text_sha256": "d4c7e10745a0e1515bdc4ea22c6f5f88dfe322b09830ee5f0129ea9ac13e4379",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 113.66740340087563,
    "vllm_decode_tps": 37.354596594640704,
    "vllm_generation_tokens_delta": 4246.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.142697302158922,
    "vllm_prefill_tps": 413.93289290729155,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 36.977910511255956
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 49.847530151950195,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T23:32:41Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.012120805215090513,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 4415,
    "text_chars": 4415,
    "text_sha256": "70fbcad763bc75d1617c2fd34f2e0871229bddeeea35839eac3f6b3207b66ad3",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 48.690047662938014,
    "vllm_decode_tps": 41.07615613451872,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.1360376980155706,
    "vllm_prefill_tps": 416.359422602116,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 40.12234896901413
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 349.91630028304644,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T23:38:31Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.017869479954242706,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 15915,
    "text_chars": 15915,
    "text_sha256": "ee7c5fc867eca6b5f4b62a627614b132f2a803ab6b18a5268cb2396bbe0ca109",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 348.76385564310476,
    "vllm_decode_tps": 28.67269597521926,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.1339959390461445,
    "vllm_prefill_tps": 417.10907747858585,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 28.57826283574393
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestMTP counter replayPASS10,158 natural tokens2K 33.38410K 24.895 · diagnostic2K MTP accepted 1122 / 1758 (63.82%)
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T19:41:48.215231213Z",
      "FinishedAt": "2026-10-26T20:30:51.421619496Z"
    },
    "devices_after": [
      {
        "card": 5,
        "render": 133,
        "pci": "0000:16:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:19:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 7,
        "render": 135,
        "pci": "0000:1c:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 8,
        "render": 136,
        "pci": "0000:1f:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T20:30:51.545320+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 33.384149922977,
  "retest_10K_TPS": 24.895468336506017,
  "retest_strict_tokens": 10158,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 10158,
      "elapsed_seconds": 412.2303533559898,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 411.0824744999409,
      "vllm_decode_tps": 24.710369889537727,
      "vllm_generation_tokens_delta": 10158.0,
      "vllm_prefill_seconds_delta": 1.1355813550762832,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 61.05597344099078,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 59.90866937197279,
      "vllm_decode_tps": 33.384149922977,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 1.136036392999813,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 402.8300866619684,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 401.6795291750459,
      "vllm_decode_tps": 24.895468336506017,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 1.1382164129754528,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 1.136036392999813,
  "retest_2K_total_seconds": 61.05597344099078,
  "retest_10K_prefill_seconds": 1.1382164129754528,
  "retest_10K_total_seconds": 402.8300866619684,
  "retest_strict_prefill_seconds": 1.1355813550762832,
  "retest_strict_total_seconds": 412.2303533559898,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:generation_tokens_total": 10158.0,
        "vllm:spec_decode_num_draft_tokens_total": 8090.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6113.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:request_generation_tokens_sum": 10158.0,
        "vllm:spec_decode_num_drafts_total": 4045.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6113.0
      },
      "accepted_per_position": {
        "0": 3356.0,
        "1": 2757.0
      },
      "accepted_fraction": 0.7556242274412855,
      "committed_emitted_tokens": 10158.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "b0c7fc7c40a28cc81666d74d656deded0faa1edf58d638d792732d22f635a0a3",
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "4266e9362ae653507a568738210298328ae118dae456893caf38eac6b5e259c2"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 1758.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1122.0,
        "vllm:spec_decode_num_drafts_total": 879.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1122.0,
        "vllm:request_generation_tokens_sum": 2000.0
      },
      "accepted_per_position": {
        "0": 652.0,
        "1": 470.0
      },
      "accepted_fraction": 0.6382252559726962,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "b96c35f259b8cd18c589eb921758bf9ce6b7e775968528fc12d8e68656e4d816",
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "46b1bbec57fb4711473fc1074c580eeb32642138f4dda7915b8a8b898d7b7b55"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 7984.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6007.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_drafts_total": 3992.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6007.0
      },
      "accepted_per_position": {
        "0": 3303.0,
        "1": 2704.0
      },
      "accepted_fraction": 0.7523797595190381,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "f1cd06ff0d404e30521e2e57fd27914f1d1f3c059744359565eaba6cf8eac279",
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "6393ab308a1f1c437c10bdb67e948cc15e6d647d1a94cc644bb7a5cc99b3840f"
      }
    }
  },
  "retest_MTP_2K_accepted": 1122.0,
  "retest_MTP_2K_proposed": 1758.0,
  "retest_MTP_2K_draft_batches": 879.0,
  "retest_MTP_2K_acceptance_percent": 63.82252559726962,
  "retest_loaded_admission": "Complete record: chart-details/row-18-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30542385152,
    "1": 30557487104,
    "2": 30557450240,
    "3": 30507130880
  },
  "retest_memory_observation_count": 46,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-w8-tp4-mtp2/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 8.25,
  "retest_logged_model_load_seconds": 69.672332,
  "retest_logged_available_KV_GiB": 18.53,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 18.53,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 284800,
  "retest_logged_context_capacity_x": 8.25,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "v620-w8-tp4-mtp2",
    "image": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
    "container": "5457bd41fa75f10e8266c62e7d825106f9d861c3a122df35c891d462aff0ef09",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 8.25,
    "retest_logged_model_load_seconds": 69.672332,
    "retest_logged_available_KV_GiB": 18.53,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 18.53,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 284800,
    "retest_logged_context_capacity_x": 8.25,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w8-tp4-mtp2/service-final.log",
    "source_log_sha256": "196a6879d46ceafafa4327fa2e08df4d40d490907a2174e6abe1f3d38e50358f",
    "sealed_index_sha256": "02fc37701b599060b58b02a78ac7165d21c59bda1893085577c5b7228fd3431e",
    "source_lines": {
      "model_load": {
        "line": 154,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 19:44:59 [gpu_model_runner.py:4598] Model loading took 8.25 GiB memory and 69.672332 seconds"
      },
      "automatic_KV_budget": {
        "line": 225,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 20:02:31 [gpu_worker.py:453] Available KV cache memory: 18.53 GiB"
      },
      "grouped_KV_tokens": {
        "line": 229,
        "text": "(EngineCore_DP0 pid=85) INFO 10-26 20:02:31 [kv_cache_utils.py:1314] GPU KV cache size: 284,800 tokens"
      },
      "context_capacity": {
        "line": 230,
        "text": "(EngineCore_DP0 pid=85) INFO 10-26 20:02:31 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 8.25x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w8-tp4-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.458877563476562
}
Complete retest loaded admission
Retest KV budget: 18.530 GiB (automatic, rounded)284,800 grouped KV tokens · 8.25× reported capacityRetest sampled VRAM: 28.459 GiB max/cardRetest memory reports and limits
Evidence and limits

Completed, independently reviewed W8 TP4 native MTP2 full ladder; strict PASS 8037. One eligible sample, no promotion or FP16 qualification.

{
  "strict_tokens": 8037,
  "source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.FP16_W8_TP_MTP_matrix.cells[4].result",
  "source_revision": "e4e131d2aa3424f1fb968a979a1e34bf5c510a16",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "attention_KV_dtype": "FP16",
  "MTP_model_class": "vllm.model_executor.models.qwen3_5_mtp.Qwen3_5MTP",
  "MTP_precision": "Five draft projections use the same W8/group128 RTN scheme; remaining draft parameters FP16, embedding/head shared with target. Same official checkpoint revision.",
  "memory_accounting_limit": "Packed totals here include separately itemized target and draft projections, excluding retained FP16 parameters/shared embeddings/head. The 16 target plus one draft logical KV views are not a disjoint physical allocation sum. Sampled VRAM, allocator counters and runtime capacity are separate.",
  "load_and_KV_allocator_bytes_per_rank": "[{\"KV_allocated\": 28207806976, \"KV_reserved\": 28718399488, \"KV_shapes\": [[1425, 2, 800, 1, 256]], \"load_allocated\": 8315061248, \"load_reserved\": 8856272896, \"rank\": 0}, {\"KV_allocated\": 28207913984, \"KV_reserved\": 28718399488, \"KV_shapes\": [[1425, 2, 800, 1, 256]], \"load_allocated\": 8315168256, \"load_reserved\": 8856272896, \"rank\": 1}, {\"KV_allocated\": 28207902720, \"KV_reserved\": 28718399488, \"KV_shapes\": [[1425, 2, 800, 1, 256]], \"load_allocated\": 8315156992, \"load_reserved\": 8856272896, \"rank\": 2}, {\"KV_allocated\": 28209225216, \"KV_reserved\": 28718399488, \"KV_shapes\": [[1425, 2, 800, 1, 256]], \"load_allocated\": 8316479488, \"load_reserved\": 8856272896, \"rank\": 3}]",
  "sampled_peak_VRAM_bytes_by_PCI": "{\"0000:16:00.0\": 30507261952, \"0000:19:00.0\": 30521655296, \"0000:1c:00.0\": 30521917440, \"0000:1f:00.0\": 30471643136}",
  "KV_capacity": "{\"available_GiB\": 18.49, \"draft_views\": 1, \"reported_concurrency_at131072\": 8.24, \"reported_grouped_tokens\": 284800, \"target_views\": 16, \"view_bytes\": 1167360000, \"view_shape\": [1425, 2, 800, 1, 256]}",
  "attention_KV_logical_view_shape": "[1425, 2, 800, 1, 256]",
  "MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 6366.0,
        "vllm:spec_decode_num_accepted_tokens_total": 4853.0,
        "vllm:request_generation_tokens_sum": 8037.0,
        "vllm:generation_tokens_total": 8037.0,
        "vllm:spec_decode_num_drafts_total": 3183.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 4853.0
      },
      "accepted_per_position": {
        "0": 2665.0,
        "1": 2188.0
      },
      "accepted_fraction": 0.7623311341501728,
      "committed_emitted_tokens": 8037.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-before.txt": "f25ea1303c7b1b799dc39b451ba96372cf5f5ce3f3e95dcce6cb298949c411ed",
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-after.txt": "6b4fc8665875fcfcb7c367f42504561e046ac6fbe9e7a66f99ec7e5e02bfe9c3"
      }
    },
    "fixed2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 1094.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_draft_tokens_total": 1810.0,
        "vllm:spec_decode_num_drafts_total": 905.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1094.0
      },
      "accepted_per_position": {
        "0": 661.0,
        "1": 433.0
      },
      "accepted_fraction": 0.6044198895027625,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-before.txt": "1d4560681bb3859f5d6973269d741310351b402726225fe1a8ddc261774ec795",
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-after.txt": "aa43719aabaf5b07691d948f97c452bc201e4bc888f252f8c02c5bdcf8b020a9"
      }
    },
    "fixed10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 6117.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6117.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_drafts_total": 3884.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 7768.0
      },
      "accepted_per_position": {
        "0": 3319.0,
        "1": 2798.0
      },
      "accepted_fraction": 0.7874613800205973,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-before.txt": "1ed44508b3ea244ca341d97f5dd473da7972d10374032b9f9af3342b91df3982",
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-after.txt": "5073090e5bc7f524f85a7bff408b5058b951839f7491f1d4b6873525e0a5aa84"
      }
    }
  },
  "MTP_accounting_limit": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
  "source_evidence_commit": "0c0751c8b79fc57442317af1898946cc4f4cfec8",
  "comparison_limit": "Opaque native selector repairs the free-s47 compiler failure; it forwards to the unchanged ordered-FP32 U operator for M<=32 and original safe reconstruction for larger batches. New dispatch flag1 and mapped DSOdf0df1a9; native U45281e2c and safef52f2e8b unchanged. | Native MTP2 target and draft, ordinary decoder/collectives and 800 attention fallback differ from historical non-MTP U profile. No isolated MTP or compiler-speed attribution. | Fresh private rootSSD caches, FP16 activations/attentionKV, memory utilization0.95, context131072, greedy concurrency1 and original requests/scorer retained. | One sample per tier; no repeatability or final qualification. | Speed numerator uses committed outputs, not draft proposals; acceptance counts precede stop trimming. | Static256 opaque routes plus loaded/mapped libraries are not dynamic per-kernel trace; MTP counters show actual speculative work. | New native opaque selector bypasses compiler symbol defect without changing U/safe numerical operators; MTP uses800 attention fallback and fresh private rootSSD caches. Historical W8/non-MTP cache/KV/backend differences prevent isolated causal attribution. | Logical packed extents and KV views are not summed into physical memory; remaining FP16 tensor bytes not fully itemized. | Earlier initialization NameError retained; inactive failed-tail unused because strict passed.",
  "recorded_configuration_details": {
    "source": "e4e131d2aa3424f1fb968a979a1e34bf5c510a16",
    "image": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
    "service_manifest_sha256": "c174581c0c18d46646363cc42d8e658ce1aa756746c803e4920aedce0ee737b6",
    "model_repository": "Qwen/Qwen3.8-27B",
    "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "strict_valid": true,
    "strict_tokens": 8037,
    "strict_diagnostic_TPS": 27.504710310538147,
    "official_fixed2K_TPS": 32.930074165374116,
    "diagnostic_fixed10K_TPS": 25.672148198446543,
    "metrics": {
      "strict": {
        "completion_tokens": 8037,
        "finish_reason": "stop",
        "elapsed_seconds": 293.35235242999624,
        "vllm_prefill_seconds_delta": 1.1333177799824625,
        "vllm_decode_seconds_delta": 292.2044954940211,
        "vllm_decode_tps": 27.504710310538147,
        "vllm_generation_tokens_delta": 8037.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "d7a1b04f9328629cb64dc6acf53a90c6048065daecaf10ae16036c7e82fa778e",
        "server_e2e_seconds": 293.3473253250122,
        "MTP": {
          "status": "observed",
          "actual_MTP_work_observed": true,
          "counter_deltas": {
            "vllm:request_generation_tokens_count": 1.0,
            "vllm:spec_decode_num_draft_tokens_total": 6366.0,
            "vllm:spec_decode_num_accepted_tokens_total": 4853.0,
            "vllm:request_generation_tokens_sum": 8037.0,
            "vllm:generation_tokens_total": 8037.0,
            "vllm:spec_decode_num_drafts_total": 3183.0,
            "vllm:spec_decode_num_accepted_tokens_per_pos_total": 4853.0
          },
          "accepted_per_position": {
            "0": 2665.0,
            "1": 2188.0
          },
          "accepted_fraction": 0.7623311341501728,
          "committed_emitted_tokens": 8037.0,
          "request_tokens_agree": true,
          "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
          "sources": {
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-before.txt": "f25ea1303c7b1b799dc39b451ba96372cf5f5ce3f3e95dcce6cb298949c411ed",
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-after.txt": "6b4fc8665875fcfcb7c367f42504561e046ac6fbe9e7a66f99ec7e5e02bfe9c3"
          }
        }
      },
      "fixed2K": {
        "completion_tokens": 2000,
        "finish_reason": "length",
        "elapsed_seconds": 61.88037476199679,
        "vllm_prefill_seconds_delta": 1.1334461470250972,
        "vllm_decode_seconds_delta": 60.73475540796062,
        "vllm_decode_tps": 32.930074165374116,
        "vllm_generation_tokens_delta": 2000.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "96e2767d5adb74a3c90c5198ea21f68211501710b8811fd1515d243c4eab27a5",
        "server_e2e_seconds": 61.87655425071716,
        "MTP": {
          "status": "observed",
          "actual_MTP_work_observed": true,
          "counter_deltas": {
            "vllm:spec_decode_num_accepted_tokens_total": 1094.0,
            "vllm:request_generation_tokens_count": 1.0,
            "vllm:generation_tokens_total": 2000.0,
            "vllm:request_generation_tokens_sum": 2000.0,
            "vllm:spec_decode_num_draft_tokens_total": 1810.0,
            "vllm:spec_decode_num_drafts_total": 905.0,
            "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1094.0
          },
          "accepted_per_position": {
            "0": 661.0,
            "1": 433.0
          },
          "accepted_fraction": 0.6044198895027625,
          "committed_emitted_tokens": 2000.0,
          "request_tokens_agree": true,
          "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
          "sources": {
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-before.txt": "1d4560681bb3859f5d6973269d741310351b402726225fe1a8ddc261774ec795",
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-after.txt": "aa43719aabaf5b07691d948f97c452bc201e4bc888f252f8c02c5bdcf8b020a9"
          }
        }
      },
      "fixed10K": {
        "completion_tokens": 10000,
        "finish_reason": "length",
        "elapsed_seconds": 390.68672217201674,
        "vllm_prefill_seconds_delta": 1.1481118430383503,
        "vllm_decode_seconds_delta": 389.5271997769596,
        "vllm_decode_tps": 25.672148198446543,
        "vllm_generation_tokens_delta": 10000.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "b63985089a787c260e30876d287e402a34a6373bf71bc3556f271b2799b61c22",
        "server_e2e_seconds": 390.6815514564514,
        "MTP": {
          "status": "observed",
          "actual_MTP_work_observed": true,
          "counter_deltas": {
            "vllm:spec_decode_num_accepted_tokens_total": 6117.0,
            "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6117.0,
            "vllm:generation_tokens_total": 10000.0,
            "vllm:request_generation_tokens_sum": 10000.0,
            "vllm:spec_decode_num_drafts_total": 3884.0,
            "vllm:request_generation_tokens_count": 1.0,
            "vllm:spec_decode_num_draft_tokens_total": 7768.0
          },
          "accepted_per_position": {
            "0": 3319.0,
            "1": 2798.0
          },
          "accepted_fraction": 0.7874613800205973,
          "committed_emitted_tokens": 10000.0,
          "request_tokens_agree": true,
          "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
          "sources": {
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-before.txt": "1ed44508b3ea244ca341d97f5dd473da7972d10374032b9f9af3342b91df3982",
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-after.txt": "5073090e5bc7f524f85a7bff408b5058b951839f7491f1d4b6873525e0a5aa84"
          }
        }
      }
    },
    "weight_dtype": "symmetric RTN W8 in GPTQ-v2 layout; group128, ties-to-even, zero128, FP16 scales",
    "activation_dtype": "FP16",
    "attention_KV_dtype": "FP16",
    "TP": 4,
    "MTP_tokens": 2,
    "MTP_model_class": "vllm.model_executor.models.qwen3_5_mtp.Qwen3_5MTP",
    "MTP_precision": "Five draft projections use the same W8/group128 RTN scheme; remaining draft parameters FP16, embedding/head shared with target. Same official checkpoint revision.",
    "quantized_target_modules_per_rank": 256,
    "quantized_draft_modules_per_rank": 5,
    "packed_component_bytes_per_rank": {
      "target": {
        "qweight": 6087639040,
        "qzeros": 47559680,
        "scales": 95119360,
        "g_idx": 0
      },
      "draft": {
        "qweight": 106168320,
        "qzeros": 829440,
        "scales": 1658880,
        "g_idx": 0
      }
    },
    "packed_target_bytes_per_rank": 6230318080,
    "packed_draft_bytes_per_rank": 108656640,
    "packed_weight_bytes_per_rank": 6338974720,
    "sampled_peak_VRAM_bytes": {
      "0000:16:00.0": 30507261952,
      "0000:19:00.0": 30521655296,
      "0000:1c:00.0": 30521917440,
      "0000:1f:00.0": 30471643136
    },
    "loaded_allocator_and_KV_per_rank": [
      {
        "rank": 0,
        "load_allocated": 8315061248,
        "load_reserved": 8856272896,
        "KV_allocated": 28207806976,
        "KV_reserved": 28718399488,
        "KV_shapes": [
          [
            1425,
            2,
            800,
            1,
            256
          ]
        ]
      },
      {
        "rank": 1,
        "load_allocated": 8315168256,
        "load_reserved": 8856272896,
        "KV_allocated": 28207913984,
        "KV_reserved": 28718399488,
        "KV_shapes": [
          [
            1425,
            2,
            800,
            1,
            256
          ]
        ]
      },
      {
        "rank": 2,
        "load_allocated": 8315156992,
        "load_reserved": 8856272896,
        "KV_allocated": 28207902720,
        "KV_reserved": 28718399488,
        "KV_shapes": [
          [
            1425,
            2,
            800,
            1,
            256
          ]
        ]
      },
      {
        "rank": 3,
        "load_allocated": 8316479488,
        "load_reserved": 8856272896,
        "KV_allocated": 28209225216,
        "KV_reserved": 28718399488,
        "KV_shapes": [
          [
            1425,
            2,
            800,
            1,
            256
          ]
        ]
      }
    ],
    "KV_capacity": {
      "available_GiB": 18.49,
      "reported_grouped_tokens": 284800,
      "reported_concurrency_at131072": 8.24,
      "target_views": 16,
      "draft_views": 1,
      "view_shape": [
        1425,
        2,
        800,
        1,
        256
      ],
      "view_bytes": 1167360000
    },
    "configured_context": 131072,
    "complete_ladders": 1,
    "sample_count": 1,
    "repeatability_established": false,
    "FP16_release_qualification": false,
    "archive": {
      "path": "/home/ai/vnext-v620-qwen38-rtn8-tp4-mtp2-service-r1-20261023/raw-evidence.tar.gz",
      "bytes": 1169597,
      "sha256": "7d1d299f8a809f68f5edf5c4a676fa226f28adb7a6639c80afcfec33169e357c",
      "files": 251,
      "index_sha256": "ade3ca824e03c977660ec0352a4ef2afa3ae2077295f2749bac38fd0203b2f84"
    },
    "local_primary_record": ".work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/measurement-closure.json",
    "review": {
      "path": ".work/fp16-w8-tp-mtp-matrix/reviews/W8-TP4-MTP2-r1-final-cell-review.json",
      "sha256": "fc8198cb6bad9e8c493b850a9742f062a4116bcef3d86daa8f4afdcd44541582"
    },
    "closure": {
      "container_id": "5173931e9d3991c31c191cf6c40ad967ab14b9e8cb2ed4e7ac81df72cb9091c1",
      "image_id": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
      "exit_code": 0,
      "OOMKilled": false,
      "restart_count": 0,
      "bank_idle": true,
      "devices": [
        {
          "pci": "0000:16:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        },
        {
          "pci": "0000:19:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        },
        {
          "pci": "0000:1c:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        },
        {
          "pci": "0000:1f:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        }
      ],
      "started_at": "2026-10-23T15:08:09.043168117Z",
      "finished_at": "2026-10-23T15:54:05.354754912Z",
      "closed_at": "2026-10-23T15:54:06.522428+00:00",
      "kernel_fault_matches": [],
      "startup_requests_completed_and_drained": 3,
      "official_phase_samples": 11,
      "complete_ladders": 1,
      "strict_valid": true,
      "official_fixed_2000_tps": 32.930074165374116,
      "qualifying_ladders": 0,
      "FP16_qualification": false,
      "baseline_kind": "RTN8"
    },
    "final_counters": {
      "requests": 14,
      "prompt_tokens": 6622,
      "committed_generation_tokens": 42037
    },
    "runtime_differences": [
      "Opaque native selector repairs the free-s47 compiler failure; it forwards to the unchanged ordered-FP32 U operator for M<=32 and original safe reconstruction for larger batches. New dispatch flag1 and mapped DSOdf0df1a9; native U45281e2c and safef52f2e8b unchanged.",
      "Native MTP2 target and draft, ordinary decoder/collectives and 800 attention fallback differ from historical non-MTP U profile. No isolated MTP or compiler-speed attribution.",
      "Fresh private rootSSD caches, FP16 activations/attentionKV, memory utilization0.95, context131072, greedy concurrency1 and original requests/scorer retained."
    ],
    "limits": [
      "One sample per tier; no repeatability or final qualification.",
      "Speed numerator uses committed outputs, not draft proposals; acceptance counts precede stop trimming.",
      "Static256 opaque routes plus loaded/mapped libraries are not dynamic per-kernel trace; MTP counters show actual speculative work.",
      "New native opaque selector bypasses compiler symbol defect without changing U/safe numerical operators; MTP uses800 attention fallback and fresh private rootSSD caches. Historical W8/non-MTP cache/KV/backend differences prevent isolated causal attribution.",
      "Logical packed extents and KV views are not summed into physical memory; remaining FP16 tensor bytes not fully itemized.",
      "Earlier initialization NameError retained; inactive failed-tail unused because strict passed."
    ]
  },
  "MTP_2K_draft_batches": 905.0,
  "MTP_2K_proposed": 1810.0,
  "MTP_2K_accepted": 1094.0,
  "MTP_2K_acceptance_percent": 60.44198895027625,
  "MTP_2K_committed": 2000.0,
  "detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
  "detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
  "detail_source_pointer": "FP16_W8_TP_MTP_matrix.cells[4]",
  "retest_strict_tokens": 10158,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:generation_tokens_total": 10158.0,
        "vllm:spec_decode_num_draft_tokens_total": 8090.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6113.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:request_generation_tokens_sum": 10158.0,
        "vllm:spec_decode_num_drafts_total": 4045.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6113.0
      },
      "accepted_per_position": {
        "0": 3356.0,
        "1": 2757.0
      },
      "accepted_fraction": 0.7556242274412855,
      "committed_emitted_tokens": 10158.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "b0c7fc7c40a28cc81666d74d656deded0faa1edf58d638d792732d22f635a0a3",
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "4266e9362ae653507a568738210298328ae118dae456893caf38eac6b5e259c2"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 1758.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1122.0,
        "vllm:spec_decode_num_drafts_total": 879.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1122.0,
        "vllm:request_generation_tokens_sum": 2000.0
      },
      "accepted_per_position": {
        "0": 652.0,
        "1": 470.0
      },
      "accepted_fraction": 0.6382252559726962,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "b96c35f259b8cd18c589eb921758bf9ce6b7e775968528fc12d8e68656e4d816",
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "46b1bbec57fb4711473fc1074c580eeb32642138f4dda7915b8a8b898d7b7b55"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 7984.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6007.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_drafts_total": 3992.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6007.0
      },
      "accepted_per_position": {
        "0": 3303.0,
        "1": 2704.0
      },
      "accepted_fraction": 0.7523797595190381,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "f1cd06ff0d404e30521e2e57fd27914f1d1f3c059744359565eaba6cf8eac279",
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "6393ab308a1f1c437c10bdb67e948cc15e6d647d1a94cc644bb7a5cc99b3840f"
      }
    }
  },
  "retest_MTP_2K_accepted": 1122.0,
  "retest_MTP_2K_proposed": 1758.0,
  "retest_MTP_2K_draft_batches": 879.0,
  "retest_MTP_2K_acceptance_percent": 63.82252559726962,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30542385152,
    "1": 30557487104,
    "2": 30557450240,
    "3": 30507130880
  },
  "retest_memory_observation_count": 46,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-w8-tp4-mtp2/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 18.53,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 18.53,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 284800,
  "retest_memory_log_report": {
    "profile": "v620-w8-tp4-mtp2",
    "image": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
    "container": "5457bd41fa75f10e8266c62e7d825106f9d861c3a122df35c891d462aff0ef09",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 8.25,
    "retest_logged_model_load_seconds": 69.672332,
    "retest_logged_available_KV_GiB": 18.53,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 18.53,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 284800,
    "retest_logged_context_capacity_x": 8.25,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w8-tp4-mtp2/service-final.log",
    "source_log_sha256": "196a6879d46ceafafa4327fa2e08df4d40d490907a2174e6abe1f3d38e50358f",
    "sealed_index_sha256": "02fc37701b599060b58b02a78ac7165d21c59bda1893085577c5b7228fd3431e",
    "source_lines": {
      "model_load": {
        "line": 154,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 19:44:59 [gpu_model_runner.py:4598] Model loading took 8.25 GiB memory and 69.672332 seconds"
      },
      "automatic_KV_budget": {
        "line": 225,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 20:02:31 [gpu_worker.py:453] Available KV cache memory: 18.53 GiB"
      },
      "grouped_KV_tokens": {
        "line": 229,
        "text": "(EngineCore_DP0 pid=85) INFO 10-26 20:02:31 [kv_cache_utils.py:1314] GPU KV cache size: 284,800 tokens"
      },
      "context_capacity": {
        "line": 230,
        "text": "(EngineCore_DP0 pid=85) INFO 10-26 20:02:31 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 8.25x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w8-tp4-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.458877563476562,
  "sampled_peak_VRAM_GiB": 28.425750732421875
}
V620Current Qwen3.88-bit RTNQwen/Qwen3.8-27BTP2MTP22K accepted 1170 / 1658 (70.57%) PASS4872 tokens 25.99821.719 0.802 / 77.74610K: 0.799 / 461.243Strict: 0.802 / 205.310 11.81 / 26.68packed target/draft projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:e3ec7a663ee96460173b72b3be0fb6e5f4c77ab7a32c93b4824d60dcead4bde1
Runtime image config: sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256
mkdir -p launch-tools-v620-serve-r5
tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp2-mtp2
GPU_RENDER_NODES=renderD128,renderD129

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w8-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Preflight cleanup compatibility
After its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged.
Retained lab Docker command · not portable
docker run -d \
  --name serve-v620-w8-tp2-mtp2-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --cpuset-mems 0 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/card5:/dev/dri/card5:rwm \
  --device /dev/dri/card6:/dev/dri/card6:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --user root \
  --workdir /tmp \
  -e VNEXT_RTN_W8_NATIVE_DISPATCH=1 \
  -e VNEXT_RTN_W8_FUSED_CTA=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
  -e ROCM_VERSION=7.14.0 \
  -e TORCH_VERSION=2.12.0+rocm7.14.0 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
  -e PYTORCH_ROCM_ARCH=gfx1030 \
  -e GPU_ARCHS=gfx1030 \
  -e AMDGPU_TARGETS=gfx1030 \
  -e CMAKE_HIP_ARCHITECTURES=gfx1030 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e VNEXT_GDN_HEAD32=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp2/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp2/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp2/cache/torch_extensions,dst=/root/.cache/torch_extensions \
  --entrypoint /usr/bin/env \
  sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567 \
  V756F94_BOUNDARY_MODE=posted \
  DO_NOT_TRACK=1 \
  GPU_DEVICE_ORDINAL=0,1 \
  GPU_MAX_HW_QUEUES=1 \
  HSA_OVERRIDE_GFX_VERSION=10.3.0 \
  OMP_NUM_THREADS=4 \
  TEMP=/tmp/torchinductor_root \
  TMPDIR=/tmp/torchinductor_root \
  TMP=/tmp/torchinductor_root \
  TORCH_BLAS_PREFER_HIPBLASLT=0 \
  TRITON_CACHE_DIR=/root/.triton \
  VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  VLLM_GPU_MEMORY_UTILIZATION=0.95 \
  VLLM_LOGGING_LEVEL=INFO \
  VLLM_PORT=9002 \
  VLLM_RDNA_FORCE_FP16=1 \
  VLLM_WORKER_MULTIPROC_METHOD=spawn \
  NCCL_P2P_DISABLE=0 \
  NCCL_ALGO=Tree \
  NCCL_PROTO=LL \
  NCCL_MIN_NCHANNELS=4 \
  NCCL_MAX_NCHANNELS=4 \
  NCCL_NTHREADS=256 \
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
  HIP_VISIBLE_DEVICES=0,1 \
  CUDA_VISIBLE_DEVICES=0,1 \
  ROCR_VISIBLE_DEVICES=0,1 \
  VLLM_CACHE_ROOT=/root/.cache/vllm \
  PYTORCH_ROCM_ARCH=gfx1030 \
  PYTHONHASHSEED=0 \
  /opt/venv/bin/python \
  -m \
  vllm.vnext_rtn_entry \
  serve \
  /opt/local-models/qwen38-27b-hf \
  --served-model-name \
  Qwen/Qwen3.8-27B \
  --enable-auto-tool-choice \
  --tool-call-parser \
  qwen3_coder \
  --dtype \
  half \
  --host \
  0.0.0.0 \
  --port \
  9002 \
  --tensor-parallel-size \
  2 \
  --speculative-config \
  '{"method":"mtp","num_speculative_tokens":2}' \
  --max-model-len \
  131072 \
  --gpu-memory-utilization \
  0.95 \
  --trust-remote-code \
  --generation-config \
  vllm \
  -O=3 \
  --reasoning-parser \
  qwen3 \
  --async-scheduling \
  --hf-overrides \
  '{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":8,"group_size":128}}' \
  --language-model-only \
  --max-num-seqs \
  2 \
  --max-num-batched-tokens \
  2048 \
  --mamba-ssm-cache-dtype \
  float32 \
  --no-enable-prefix-caching \
  --no-enforce-eager \
  --attention-backend \
  TRITON_ATTN \
  --disable-custom-all-reduce \
  --seed \
  0 \
  --worker-cls \
  vllm.vnext_rtn_worker.Worker \
  --quantization \
  vnext_rtn
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-r4.tar.gz.sha256
mkdir -p launch-tools-v620-r4
tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp2-mtp2
GPU_RENDER_NODES=renderD128,renderD129

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-v620-r4/portable/launch.py --config v620-w8-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-v620-serve-r5.tar.gz · SHA-256
launch-tools-v620-r4.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
Portable observer compatibility
The evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged.
All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 24.072 · eligible10K 20.016 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 7027,
    "content_chars": 6647,
    "content_has_think_close": false,
    "elapsed_seconds": 315.2030647529755,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T00:56:43Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.011896442156285048,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 6647,
    "qwen_answer_text_sha256": "26a2ea768dce161526e7577cced0a08108a8d9b313c768dedbaca547688d8e5b",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 6647,
    "qwen_visible_text_sha256": "26a2ea768dce161526e7577cced0a08108a8d9b313c768dedbaca547688d8e5b",
    "raw_has_think_close": false,
    "reasoning_chars": 27053,
    "text_chars": 33701,
    "text_sha256": "6fa8e4faec13968960681e2d41dd83052329313155b407a52246c75b6b809df4",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 314.38775441702455,
    "vllm_decode_tps": 22.35137947096669,
    "vllm_generation_tokens_delta": 7027.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.7998496741056442,
    "vllm_prefill_tps": 591.361121111773,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 22.293564961073766
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 83.8934364290908,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T00:58:07Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.009258593898266554,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 8929,
    "text_chars": 8929,
    "text_sha256": "e1c5e4a62e7f1e164f56ca453844bfff8198bd48bce270b29a39a7e864e2b3c4",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 83.08291288209148,
    "vllm_decode_tps": 24.07233847034629,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.7974292829167098,
    "vllm_prefill_tps": 593.1560454739459,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 23.839767270595225
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 22460,
    "content_has_think_close": false,
    "elapsed_seconds": 500.42501594801433,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T01:06:28Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.01606009411625564,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 22460,
    "qwen_answer_text_sha256": "31fd810e67d65a343e3a49f3e86e854fc1beadaed3b0984180ad70a959a64dc4",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 22460,
    "qwen_visible_text_sha256": "31fd810e67d65a343e3a49f3e86e854fc1beadaed3b0984180ad70a959a64dc4",
    "raw_has_think_close": false,
    "reasoning_chars": 27053,
    "text_chars": 49514,
    "text_sha256": "56f5602a23640359af1fc4bb9386a4268a92fa54668d3693b5a01b1f3a8098af",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 499.6082297200337,
    "vllm_decode_tps": 20.01568309954325,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8002281670924276,
    "vllm_prefill_tps": 591.081418339237,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 19.98301380089046
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completed; strict failed; fixed speeds diagnostic onlyVerification · Raw retestMTP counter replayFAIL14,095 natural tokens2K 28.828*10K 22.897* · diagnostic2K MTP accepted 1252 / 1494 (83.80%)
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T20:33:03.042335455Z",
      "FinishedAt": "2026-10-26T21:33:17.506571757Z"
    },
    "devices_after": [
      {
        "card": 5,
        "render": 133,
        "pci": "0000:16:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:19:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T21:33:17.649653+00:00"
  },
  "retest_strict": "FAIL",
  "retest_official_2K_eligible": false,
  "retest_2K_TPS": 28.827830303864896,
  "retest_10K_TPS": 22.897475815790525,
  "retest_strict_tokens": 14095,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 14095,
      "elapsed_seconds": 697.7960248619784,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 696.983301272965,
      "vllm_decode_tps": 20.222866135037954,
      "vllm_generation_tokens_delta": 14095.0,
      "vllm_prefill_seconds_delta": 0.801551835029386,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 70.18575884401798,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 69.37740297894925,
      "vllm_decode_tps": 28.827830303864896,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 0.797065292019397,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 437.54100987606216,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 436.72936180606484,
      "vllm_decode_tps": 22.897475815790525,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 0.7997282180003822,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 0.797065292019397,
  "retest_2K_total_seconds": 70.18575884401798,
  "retest_10K_prefill_seconds": 0.7997282180003822,
  "retest_10K_total_seconds": 437.54100987606216,
  "retest_strict_prefill_seconds": 0.801551835029386,
  "retest_strict_total_seconds": 697.7960248619784,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 9314.0,
        "vllm:spec_decode_num_drafts_total": 4780.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:generation_tokens_total": 14095.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 9314.0,
        "vllm:request_generation_tokens_sum": 14095.0,
        "vllm:spec_decode_num_draft_tokens_total": 9560.0
      },
      "accepted_per_position": {
        "0": 4699.0,
        "1": 4615.0
      },
      "accepted_fraction": 0.9742677824267783,
      "committed_emitted_tokens": 14095.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "d499da222db5f4c902370bd706b3037ebd84b7cd1c3ca91296c6f0641cfb1dcf",
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "4c68d0bfce8174357d766b4c218392da6257632f05d570487984b9c97f56b6dc"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_draft_tokens_total": 1494.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1252.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1252.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:spec_decode_num_drafts_total": 747.0,
        "vllm:request_generation_tokens_count": 1.0
      },
      "accepted_per_position": {
        "0": 667.0,
        "1": 585.0
      },
      "accepted_fraction": 0.8380187416331994,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "f7462730d6f10a811663c5107c3584869ce2f10c7a1ace0726aeb09a783c0275",
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "f9eefe90a5ed4ea8030f25ec580474c5d1f9ee6238d18cc2d27bab36615688af"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_draft_tokens_total": 6830.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6586.0,
        "vllm:spec_decode_num_drafts_total": 3415.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6586.0,
        "vllm:generation_tokens_total": 10000.0
      },
      "accepted_per_position": {
        "0": 3335.0,
        "1": 3251.0
      },
      "accepted_fraction": 0.9642752562225476,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "3d9f3d802948b3286135fb227b09e5ae44fa5ee621e625f95dc6f2df71691291",
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "e6abdb87a069a4d72731346e46434237176a71f431b126902abadb04ba82bd7b"
      }
    }
  },
  "retest_MTP_2K_accepted": 1252.0,
  "retest_MTP_2K_proposed": 1494.0,
  "retest_MTP_2K_draft_batches": 747.0,
  "retest_MTP_2K_acceptance_percent": 83.80187416331995,
  "retest_loaded_admission": "Complete record: chart-details/row-19-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 1, \"fixed\": 1, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
  "retest_diagnostic_counter_audit": "retest-evidence/v620-w8-tp2-mtp2/local-diagnostic-fixed-audit.json",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 28642738176,
    "1": 28642762752
  },
  "retest_memory_observation_count": 58,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-w8-tp2-mtp2/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 16.56,
  "retest_logged_model_load_seconds": 123.006295,
  "retest_logged_available_KV_GiB": 9.6,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 9.6,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 73600,
  "retest_logged_context_capacity_x": 2.14,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "v620-w8-tp2-mtp2",
    "image": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
    "container": "21a9828f12b1d34b82e2d0a87d86fb0d50ba8af66ac161c5ef4070a0e8424ed6",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 16.56,
    "retest_logged_model_load_seconds": 123.006295,
    "retest_logged_available_KV_GiB": 9.6,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 9.6,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 73600,
    "retest_logged_context_capacity_x": 2.14,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w8-tp2-mtp2/service-final.log",
    "source_log_sha256": "55ab409a441ae421f1d1b05f492f45e5125d690e7dcbd2140e32c403b07075c4",
    "sealed_index_sha256": "80034e3ab771353bf7a27f0c11c3a059a53363a207eb153ed442827132adb6d0",
    "source_lines": {
      "model_load": {
        "line": 118,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 20:37:10 [gpu_model_runner.py:4598] Model loading took 16.56 GiB memory and 123.006295 seconds"
      },
      "automatic_KV_budget": {
        "line": 172,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 20:57:42 [gpu_worker.py:453] Available KV cache memory: 9.6 GiB"
      },
      "grouped_KV_tokens": {
        "line": 176,
        "text": "(EngineCore_DP0 pid=85) INFO 10-26 20:57:42 [kv_cache_utils.py:1314] GPU KV cache size: 73,600 tokens"
      },
      "context_capacity": {
        "line": 177,
        "text": "(EngineCore_DP0 pid=85) INFO 10-26 20:57:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 2.14x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w8-tp2-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 26.67565155029297
}
Complete retest loaded admission
Retest KV budget: 9.600 GiB (automatic, rounded)73,600 grouped KV tokens · 2.14× reported capacityRetest sampled VRAM: 26.676 GiB max/cardRetest memory reports and limits
Evidence and limits

Completed, independently reviewed W8 TP2 native MTP2 full ladder; strict PASS 4872. One eligible sample, no promotion or FP16 qualification.

{
  "strict_tokens": 4872,
  "source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.FP16_W8_TP_MTP_matrix.cells[5].result",
  "source_revision": "e4e131d2aa3424f1fb968a979a1e34bf5c510a16",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "attention_KV_dtype": "FP16",
  "MTP_model_class": "vllm.model_executor.models.qwen3_5_mtp.Qwen3_5MTP",
  "MTP_precision": "Five draft projections use the same W8/group128 RTN scheme; remaining draft parameters FP16, embedding/head shared with target. Same official checkpoint revision.",
  "memory_accounting_limit": "Packed totals here include separately itemized target and draft projections, excluding retained FP16 parameters/shared embeddings/head. The 16 target plus one draft logical KV views are not a disjoint physical allocation sum. Sampled VRAM, allocator counters and runtime capacity are separate.",
  "load_and_KV_allocator_bytes_per_rank": "[{\"KV_allocated\": 25826995200, \"KV_reserved\": 26908557312, \"KV_shapes\": [[370, 2, 800, 2, 256]], \"load_allocated\": 15485343232, \"load_reserved\": 17783848960, \"rank\": 0}, {\"KV_allocated\": 25826995200, \"KV_reserved\": 26908557312, \"KV_shapes\": [[370, 2, 800, 2, 256]], \"load_allocated\": 15485343232, \"load_reserved\": 17783848960, \"rank\": 1}]",
  "sampled_peak_VRAM_bytes_by_PCI": "{\"0000:16:00.0\": 28642746368, \"0000:19:00.0\": 28642762752}",
  "KV_capacity": "{\"available_GiB\": 9.6, \"draft_views\": 1, \"reported_concurrency_at131072\": 2.14, \"reported_grouped_tokens\": 73600, \"target_views\": 16, \"view_bytes\": 606208000, \"view_shape\": [370, 2, 800, 2, 256]}",
  "attention_KV_logical_view_shape": "[370, 2, 800, 2, 256]",
  "MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 3910.0,
        "vllm:spec_decode_num_drafts_total": 1955.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:request_generation_tokens_sum": 4872.0,
        "vllm:generation_tokens_total": 4872.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 2917.0,
        "vllm:spec_decode_num_accepted_tokens_total": 2917.0
      },
      "accepted_per_position": {
        "0": 1616.0,
        "1": 1301.0
      },
      "accepted_fraction": 0.7460358056265984,
      "committed_emitted_tokens": 4872.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-before.txt": "dc381957e058f7374c01740b893fe43f123756b0c1873e362382cffbab159d63",
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-after.txt": "913477fb591e79e28b0263d22984402724da7441a2c1975e165ae031e8996497"
      }
    },
    "fixed2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1170.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_draft_tokens_total": 1658.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_drafts_total": 829.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1170.0,
        "vllm:generation_tokens_total": 2000.0
      },
      "accepted_per_position": {
        "0": 663.0,
        "1": 507.0
      },
      "accepted_fraction": 0.7056694813027744,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-before.txt": "8b04539b329762ccbad207878cfcc4169eb25bbc24df72877b7b711f86a03b74",
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-after.txt": "7581278c5eb0aa1bf5eb578e68dc608ceb85ceb1717f0e591cb134ef5a25bc47"
      }
    },
    "fixed10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_drafts_total": 3667.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:spec_decode_num_draft_tokens_total": 7334.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6334.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6334.0,
        "vllm:request_generation_tokens_count": 1.0
      },
      "accepted_per_position": {
        "0": 3325.0,
        "1": 3009.0
      },
      "accepted_fraction": 0.8636487592037088,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-before.txt": "cb8714611dbc685931a96f11cf30558a7c0c561a159cfbe9db8d4ae031a792c8",
        "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-after.txt": "a849896389c8c7c5d65d4f8b32a0ef56b02204f365bfbe3803d241f5c8eff407"
      }
    }
  },
  "MTP_accounting_limit": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
  "source_evidence_commit": "1fdf5463420716858c50f6804a9105df16bacc07",
  "comparison_limit": "Opaque native selector repairs the free-s47 compiler failure; it forwards to the unchanged ordered-FP32 U operator for M<=32 and original safe reconstruction for larger batches. New dispatch flag1 and mapped DSOdf0df1a9; native U45281e2c and safef52f2e8b unchanged. | Native MTP2 target and draft, ordinary decoder/collectives and 800 attention fallback differ from historical non-MTP U profile. Same eaf8 image as measured TP4 MTP2; TP2 reshards and KV budget differ. No isolated MTP or compiler-speed attribution. | Fresh private rootSSD caches, FP16 activations/attentionKV, memory utilization0.95, context131072, greedy concurrency1 and original requests/scorer retained. | Existing MTP-scoped post-sharing cache cleanup runs for TP2; the non-MTP TP2 path does not take that branch. Observed allocator reserve and KV budget differ; this is not an isolated MTP gain. | One sample per tier; no repeatability or final qualification. | Committed output only supplies numerator; speculative acceptance precedes stop trimming. | Static256 opaque calls and library mappings do not establish dynamic per-kernel trace; MTP counters show actual speculation. | Same eaf8 image/e4e131d source as TP4 MTP; TP reshards and KV budget differ, native MTP800 attention fallback/fresh private rootSSD caches materially differ from historical nonMTP. | Logical packs/views are not unique whole-model physical memory; remaining FP16 exclusions not fully itemized. | Inactive failed-tail unused because strict passed. | Reported grouped KV tokens and concurrency are runtime capacity indicators, not an independently tested exact request ceiling or demonstrated full-context residency. | Original launch observer ended143 without its SSH child exit receipt. Separate exact-container/owned-waiter verification confirms stopped/drained; no claim that all parents exited0.",
  "recorded_configuration_details": {
    "source": "e4e131d2aa3424f1fb968a979a1e34bf5c510a16",
    "image": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
    "service_manifest_sha256": "c174581c0c18d46646363cc42d8e658ce1aa756746c803e4920aedce0ee737b6",
    "model_repository": "Qwen/Qwen3.8-27B",
    "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "strict_valid": true,
    "strict_tokens": 4872,
    "strict_diagnostic_TPS": 23.824685399012747,
    "official_fixed2K_TPS": 25.99758548499443,
    "diagnostic_fixed10K_TPS": 21.71884469539952,
    "metrics": {
      "strict": {
        "completion_tokens": 4872,
        "finish_reason": "stop",
        "elapsed_seconds": 205.30964287399547,
        "vllm_prefill_seconds_delta": 0.8021039759623818,
        "vllm_decode_seconds_delta": 204.49378106801305,
        "vllm_decode_tps": 23.824685399012747,
        "vllm_generation_tokens_delta": 4872.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "f4c9bd7fda89ab399c5a21af2f247e4dccb8abac2eaa82b175150daacf1b474b",
        "server_e2e_seconds": 205.30365681648254,
        "MTP": {
          "status": "observed",
          "actual_MTP_work_observed": true,
          "counter_deltas": {
            "vllm:spec_decode_num_draft_tokens_total": 3910.0,
            "vllm:spec_decode_num_drafts_total": 1955.0,
            "vllm:request_generation_tokens_count": 1.0,
            "vllm:request_generation_tokens_sum": 4872.0,
            "vllm:generation_tokens_total": 4872.0,
            "vllm:spec_decode_num_accepted_tokens_per_pos_total": 2917.0,
            "vllm:spec_decode_num_accepted_tokens_total": 2917.0
          },
          "accepted_per_position": {
            "0": 1616.0,
            "1": 1301.0
          },
          "accepted_fraction": 0.7460358056265984,
          "committed_emitted_tokens": 4872.0,
          "request_tokens_agree": true,
          "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
          "sources": {
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-before.txt": "dc381957e058f7374c01740b893fe43f123756b0c1873e362382cffbab159d63",
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-after.txt": "913477fb591e79e28b0263d22984402724da7441a2c1975e165ae031e8996497"
          }
        }
      },
      "fixed2K": {
        "completion_tokens": 2000,
        "finish_reason": "length",
        "elapsed_seconds": 77.74615426099626,
        "vllm_prefill_seconds_delta": 0.802028578997124,
        "vllm_decode_seconds_delta": 76.93022112204926,
        "vllm_decode_tps": 25.99758548499443,
        "vllm_generation_tokens_delta": 2000.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "a06c4f4ef2f7bd568ef38d66f317106245a09492ee0e9e8359460231252df5e9",
        "server_e2e_seconds": 77.74068355560303,
        "MTP": {
          "status": "observed",
          "actual_MTP_work_observed": true,
          "counter_deltas": {
            "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1170.0,
            "vllm:request_generation_tokens_sum": 2000.0,
            "vllm:spec_decode_num_draft_tokens_total": 1658.0,
            "vllm:request_generation_tokens_count": 1.0,
            "vllm:spec_decode_num_drafts_total": 829.0,
            "vllm:spec_decode_num_accepted_tokens_total": 1170.0,
            "vllm:generation_tokens_total": 2000.0
          },
          "accepted_per_position": {
            "0": 663.0,
            "1": 507.0
          },
          "accepted_fraction": 0.7056694813027744,
          "committed_emitted_tokens": 2000.0,
          "request_tokens_agree": true,
          "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
          "sources": {
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-before.txt": "8b04539b329762ccbad207878cfcc4169eb25bbc24df72877b7b711f86a03b74",
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-after.txt": "7581278c5eb0aa1bf5eb578e68dc608ceb85ceb1717f0e591cb134ef5a25bc47"
          }
        }
      },
      "fixed10K": {
        "completion_tokens": 10000,
        "finish_reason": "length",
        "elapsed_seconds": 461.2426872580545,
        "vllm_prefill_seconds_delta": 0.7994827929651365,
        "vllm_decode_seconds_delta": 460.42964716802817,
        "vllm_decode_tps": 21.71884469539952,
        "vllm_generation_tokens_delta": 10000.0,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "summary_sha256": "5e067ccaed002e188f86f7f65df7542574524851c2ca1a5e9905cb5af524a7ed",
        "server_e2e_seconds": 461.23744559288025,
        "MTP": {
          "status": "observed",
          "actual_MTP_work_observed": true,
          "counter_deltas": {
            "vllm:spec_decode_num_drafts_total": 3667.0,
            "vllm:generation_tokens_total": 10000.0,
            "vllm:spec_decode_num_draft_tokens_total": 7334.0,
            "vllm:request_generation_tokens_sum": 10000.0,
            "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6334.0,
            "vllm:spec_decode_num_accepted_tokens_total": 6334.0,
            "vllm:request_generation_tokens_count": 1.0
          },
          "accepted_per_position": {
            "0": 3325.0,
            "1": 3009.0
          },
          "accepted_fraction": 0.8636487592037088,
          "committed_emitted_tokens": 10000.0,
          "request_tokens_agree": true,
          "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
          "sources": {
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-before.txt": "cb8714611dbc685931a96f11cf30558a7c0c561a159cfbe9db8d4ae031a792c8",
            "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-after.txt": "a849896389c8c7c5d65d4f8b32a0ef56b02204f365bfbe3803d241f5c8eff407"
          }
        }
      }
    },
    "weight_dtype": "symmetric RTN W8 in GPTQ-v2 layout; group128, ties-to-even, zero128, FP16 scales",
    "activation_dtype": "FP16",
    "attention_KV_dtype": "FP16",
    "TP": 2,
    "MTP_tokens": 2,
    "MTP_model_class": "vllm.model_executor.models.qwen3_5_mtp.Qwen3_5MTP",
    "MTP_precision": "Five draft projections use the same W8/group128 RTN scheme; remaining draft parameters FP16, embedding/head shared with target. Same official checkpoint revision.",
    "quantized_target_modules_per_rank": 256,
    "quantized_draft_modules_per_rank": 5,
    "packed_component_bytes_per_rank": {
      "target": {
        "qweight": 12175278080,
        "qzeros": 95119360,
        "scales": 190238720,
        "g_idx": 0
      },
      "draft": {
        "qweight": 212336640,
        "qzeros": 1658880,
        "scales": 3317760,
        "g_idx": 0
      }
    },
    "packed_target_bytes_per_rank": 12460636160,
    "packed_draft_bytes_per_rank": 217313280,
    "packed_weight_bytes_per_rank": 12677949440,
    "sampled_peak_VRAM_bytes": {
      "0000:16:00.0": 28642746368,
      "0000:19:00.0": 28642762752
    },
    "loaded_allocator_and_KV_per_rank": [
      {
        "rank": 0,
        "load_allocated": 15485343232,
        "load_reserved": 17783848960,
        "KV_allocated": 25826995200,
        "KV_reserved": 26908557312,
        "KV_shapes": [
          [
            370,
            2,
            800,
            2,
            256
          ]
        ]
      },
      {
        "rank": 1,
        "load_allocated": 15485343232,
        "load_reserved": 17783848960,
        "KV_allocated": 25826995200,
        "KV_reserved": 26908557312,
        "KV_shapes": [
          [
            370,
            2,
            800,
            2,
            256
          ]
        ]
      }
    ],
    "KV_capacity": {
      "available_GiB": 9.6,
      "reported_grouped_tokens": 73600,
      "reported_concurrency_at131072": 2.14,
      "target_views": 16,
      "draft_views": 1,
      "view_shape": [
        370,
        2,
        800,
        2,
        256
      ],
      "view_bytes": 606208000
    },
    "configured_context": 131072,
    "complete_ladders": 1,
    "sample_count": 1,
    "repeatability_established": false,
    "FP16_release_qualification": false,
    "archive": {
      "path": "/home/ai/vnext-v620-qwen38-rtn8-tp2-mtp2-service-r1-20261023/raw-evidence.tar.gz",
      "bytes": 781297,
      "sha256": "aee594a5aa2bf1da84a258539e121fb6a5f04b26c370ba01238952389c31a115",
      "files": 243,
      "index_sha256": "f7bef99af0d881dcf489b48bd83031337999e514ae604d07250bddd16dd9c031"
    },
    "local_primary_record": ".work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/measurement-closure.json",
    "review": {
      "path": ".work/fp16-w8-tp-mtp-matrix/reviews/W8-TP2-MTP2-r1-final-cell-review.json",
      "sha256": "028f1e91be7e589908ab9308842bbbd7b72721272a4ce3772c7ad6a93fdaba68"
    },
    "closure": {
      "container_id": "d24d898cadfa99b4a20d3004c4ed56b4100702daa0afe37d2eb1adef9092d4a8",
      "image_id": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
      "exit_code": 0,
      "OOMKilled": false,
      "restart_count": 0,
      "bank_idle": true,
      "devices": [
        {
          "pci": "0000:16:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        },
        {
          "pci": "0000:19:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        }
      ],
      "started_at": "2026-10-23T15:54:41.228163364Z",
      "finished_at": "2026-10-23T16:44:32.609572419Z",
      "closed_at": "2026-10-23T16:44:32.779906+00:00",
      "kernel_fault_matches": [],
      "startup_requests_completed_and_drained": 3,
      "official_phase_samples": 11,
      "complete_ladders": 1,
      "strict_valid": true,
      "official_fixed_2000_tps": 25.99758548499443,
      "qualifying_ladders": 0,
      "FP16_qualification": false,
      "baseline_kind": "RTN8"
    },
    "final_counters": {
      "requests": 14,
      "prompt_tokens": 6622,
      "committed_generation_tokens": 38872
    },
    "runtime_differences": [
      "Opaque native selector repairs the free-s47 compiler failure; it forwards to the unchanged ordered-FP32 U operator for M<=32 and original safe reconstruction for larger batches. New dispatch flag1 and mapped DSOdf0df1a9; native U45281e2c and safef52f2e8b unchanged.",
      "Native MTP2 target and draft, ordinary decoder/collectives and 800 attention fallback differ from historical non-MTP U profile. Same eaf8 image as measured TP4 MTP2; TP2 reshards and KV budget differ. No isolated MTP or compiler-speed attribution.",
      "Fresh private rootSSD caches, FP16 activations/attentionKV, memory utilization0.95, context131072, greedy concurrency1 and original requests/scorer retained.",
      "Existing MTP-scoped post-sharing cache cleanup runs for TP2; the non-MTP TP2 path does not take that branch. Observed allocator reserve and KV budget differ; this is not an isolated MTP gain."
    ],
    "limits": [
      "One sample per tier; no repeatability or final qualification.",
      "Committed output only supplies numerator; speculative acceptance precedes stop trimming.",
      "Static256 opaque calls and library mappings do not establish dynamic per-kernel trace; MTP counters show actual speculation.",
      "Same eaf8 image/e4e131d source as TP4 MTP; TP reshards and KV budget differ, native MTP800 attention fallback/fresh private rootSSD caches materially differ from historical nonMTP.",
      "Logical packs/views are not unique whole-model physical memory; remaining FP16 exclusions not fully itemized.",
      "Inactive failed-tail unused because strict passed.",
      "Reported grouped KV tokens and concurrency are runtime capacity indicators, not an independently tested exact request ceiling or demonstrated full-context residency.",
      "Original launch observer ended143 without its SSH child exit receipt. Separate exact-container/owned-waiter verification confirms stopped/drained; no claim that all parents exited0."
    ],
    "launch_observer_exception": {
      "path": ".work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/launch-observer-terminal.json",
      "sha256": "e537cc8661b1b1021723ef0b22b5d8cc23b4e8ecf855b227bc68646c84ba9608",
      "tool_terminal_exit_code": 143,
      "original_SSH_parent_exit": null,
      "actual_container_exit_code": 0,
      "exact_owned_waiter_absent": true,
      "interpretation": "The original launch-observer tool terminated143 without writing its SSH child parent receipt. Do not report its exit as0. Actual inference and closure are independently retained; exact container is stopped and its exact remote docker-wait observer is absent."
    },
    "parameter_dtypes_per_rank": [
      {
        "rank": 0,
        "parameter_dtype_counts": {
          "torch.float16": 611,
          "torch.int32": 768
        },
        "draft_parameter_dtype_counts": {
          "torch.int32": 15,
          "torch.float16": 14
        }
      },
      {
        "rank": 1,
        "parameter_dtype_counts": {
          "torch.float16": 611,
          "torch.int32": 768
        },
        "draft_parameter_dtype_counts": {
          "torch.int32": 15,
          "torch.float16": 14
        }
      }
    ]
  },
  "MTP_2K_draft_batches": 829.0,
  "MTP_2K_proposed": 1658.0,
  "MTP_2K_accepted": 1170.0,
  "MTP_2K_acceptance_percent": 70.56694813027744,
  "MTP_2K_committed": 2000.0,
  "detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
  "detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
  "detail_source_pointer": "FP16_W8_TP_MTP_matrix.cells[5]",
  "retest_strict_tokens": 14095,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 9314.0,
        "vllm:spec_decode_num_drafts_total": 4780.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:generation_tokens_total": 14095.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 9314.0,
        "vllm:request_generation_tokens_sum": 14095.0,
        "vllm:spec_decode_num_draft_tokens_total": 9560.0
      },
      "accepted_per_position": {
        "0": 4699.0,
        "1": 4615.0
      },
      "accepted_fraction": 0.9742677824267783,
      "committed_emitted_tokens": 14095.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "d499da222db5f4c902370bd706b3037ebd84b7cd1c3ca91296c6f0641cfb1dcf",
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "4c68d0bfce8174357d766b4c218392da6257632f05d570487984b9c97f56b6dc"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_draft_tokens_total": 1494.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1252.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1252.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:spec_decode_num_drafts_total": 747.0,
        "vllm:request_generation_tokens_count": 1.0
      },
      "accepted_per_position": {
        "0": 667.0,
        "1": 585.0
      },
      "accepted_fraction": 0.8380187416331994,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "f7462730d6f10a811663c5107c3584869ce2f10c7a1ace0726aeb09a783c0275",
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "f9eefe90a5ed4ea8030f25ec580474c5d1f9ee6238d18cc2d27bab36615688af"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_draft_tokens_total": 6830.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6586.0,
        "vllm:spec_decode_num_drafts_total": 3415.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6586.0,
        "vllm:generation_tokens_total": 10000.0
      },
      "accepted_per_position": {
        "0": 3335.0,
        "1": 3251.0
      },
      "accepted_fraction": 0.9642752562225476,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "3d9f3d802948b3286135fb227b09e5ae44fa5ee621e625f95dc6f2df71691291",
        "/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "e6abdb87a069a4d72731346e46434237176a71f431b126902abadb04ba82bd7b"
      }
    }
  },
  "retest_MTP_2K_accepted": 1252.0,
  "retest_MTP_2K_proposed": 1494.0,
  "retest_MTP_2K_draft_batches": 747.0,
  "retest_MTP_2K_acceptance_percent": 83.80187416331995,
  "retest_diagnostic_counter_audit": "retest-evidence/v620-w8-tp2-mtp2/local-diagnostic-fixed-audit.json",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 28642738176,
    "1": 28642762752
  },
  "retest_memory_observation_count": 58,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-w8-tp2-mtp2/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 9.6,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 9.6,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 73600,
  "retest_memory_log_report": {
    "profile": "v620-w8-tp2-mtp2",
    "image": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
    "container": "21a9828f12b1d34b82e2d0a87d86fb0d50ba8af66ac161c5ef4070a0e8424ed6",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 16.56,
    "retest_logged_model_load_seconds": 123.006295,
    "retest_logged_available_KV_GiB": 9.6,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 9.6,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 73600,
    "retest_logged_context_capacity_x": 2.14,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w8-tp2-mtp2/service-final.log",
    "source_log_sha256": "55ab409a441ae421f1d1b05f492f45e5125d690e7dcbd2140e32c403b07075c4",
    "sealed_index_sha256": "80034e3ab771353bf7a27f0c11c3a059a53363a207eb153ed442827132adb6d0",
    "source_lines": {
      "model_load": {
        "line": 118,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 20:37:10 [gpu_model_runner.py:4598] Model loading took 16.56 GiB memory and 123.006295 seconds"
      },
      "automatic_KV_budget": {
        "line": 172,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 20:57:42 [gpu_worker.py:453] Available KV cache memory: 9.6 GiB"
      },
      "grouped_KV_tokens": {
        "line": 176,
        "text": "(EngineCore_DP0 pid=85) INFO 10-26 20:57:42 [kv_cache_utils.py:1314] GPU KV cache size: 73,600 tokens"
      },
      "context_capacity": {
        "line": 177,
        "text": "(EngineCore_DP0 pid=85) INFO 10-26 20:57:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 2.14x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w8-tp2-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 26.67565155029297,
  "sampled_peak_VRAM_GiB": 26.67565155029297
}
V620Current Qwen3.84-bit RTNQwen/Qwen3.8-27BTP4MTP off PASS15275 tokens 40.23538.552 1.093 / 50.81610K: 1.093 / 260.499Strict: — / — 2.95 / 21.09packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:22607123e397e8cfd18f2ba40413e2cd7321d435507e65041f1336416114c596
Runtime image config: sha256:b97918608d83591ad38f818bb4e944b5aa8d7d2fa9df2f354d47f1db190d9fcb
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256
mkdir -p launch-tools-v620-serve-r5
tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp4-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w4-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Preflight cleanup compatibility
After its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged.
Retained lab Docker command · not portable
docker run -d \
  --name serve-v620-w4-tp4-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --cpuset-mems 0 \
  --security-opt label=disable \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/card5:/dev/dri/card5:rwm \
  --device /dev/dri/card6:/dev/dri/card6:rwm \
  --device /dev/dri/card7:/dev/dri/card7:rwm \
  --device /dev/dri/card8:/dev/dri/card8:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
  --device /dev/dri/renderD136:/dev/dri/renderD136:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
  -e ROCM_VERSION=7.14.0 \
  -e TORCH_VERSION=2.12.0+rocm7.14.0 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
  -e PYTORCH_ROCM_ARCH=gfx1030 \
  -e GPU_ARCHS=gfx1030 \
  -e AMDGPU_TARGETS=gfx1030 \
  -e CMAKE_HIP_ARCHITECTURES=gfx1030 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e VNEXT_GDN_HEAD32=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0/cache/torch_extensions,dst=/root/.cache/torch_extensions \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
  --entrypoint /usr/bin/env \
  sha256:b97918608d83591ad38f818bb4e944b5aa8d7d2fa9df2f354d47f1db190d9fcb \
  V756F94_BOUNDARY_MODE=posted \
  DO_NOT_TRACK=1 \
  GPU_DEVICE_ORDINAL=0,1,2,3 \
  GPU_MAX_HW_QUEUES=1 \
  HSA_OVERRIDE_GFX_VERSION=10.3.0 \
  OMP_NUM_THREADS=4 \
  TEMP=/tmp/torchinductor_root \
  TMPDIR=/tmp/torchinductor_root \
  TMP=/tmp/torchinductor_root \
  TORCH_BLAS_PREFER_HIPBLASLT=0 \
  TRITON_CACHE_DIR=/root/.triton \
  VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  VLLM_GPU_MEMORY_UTILIZATION=0.95 \
  VLLM_LOGGING_LEVEL=INFO \
  VLLM_PORT=9002 \
  VLLM_RDNA_FORCE_FP16=1 \
  VLLM_WORKER_MULTIPROC_METHOD=spawn \
  NCCL_P2P_DISABLE=0 \
  NCCL_ALGO=Tree \
  NCCL_PROTO=LL \
  NCCL_MIN_NCHANNELS=4 \
  NCCL_MAX_NCHANNELS=4 \
  NCCL_NTHREADS=256 \
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
  HIP_VISIBLE_DEVICES=0,1,2,3 \
  CUDA_VISIBLE_DEVICES=0,1,2,3 \
  ROCR_VISIBLE_DEVICES=0,1,2,3 \
  VLLM_CACHE_ROOT=/root/.cache/vllm \
  PYTORCH_ROCM_ARCH=gfx1030 \
  PYTHONHASHSEED=0 \
  /opt/venv/bin/python \
  -m \
  vllm.vnext_rtn_entry \
  serve \
  /opt/local-models/qwen38-27b-hf \
  --served-model-name \
  Qwen/Qwen3.8-27B \
  --enable-auto-tool-choice \
  --tool-call-parser \
  qwen3_coder \
  --dtype \
  half \
  --host \
  0.0.0.0 \
  --port \
  9002 \
  --tensor-parallel-size \
  4 \
  --max-model-len \
  131072 \
  --gpu-memory-utilization \
  0.95 \
  --trust-remote-code \
  --generation-config \
  vllm \
  -O=3 \
  --reasoning-parser \
  qwen3 \
  --async-scheduling \
  --hf-overrides \
  '{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":4,"group_size":128}}' \
  --language-model-only \
  --max-num-seqs \
  2 \
  --max-num-batched-tokens \
  2048 \
  --mamba-ssm-cache-dtype \
  float32 \
  --no-enable-prefix-caching \
  --no-enforce-eager \
  --attention-backend \
  TRITON_ATTN \
  --disable-custom-all-reduce \
  --seed \
  0 \
  --worker-cls \
  vllm.vnext_rtn_worker.Worker \
  --quantization \
  vnext_rtn
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-r4.tar.gz.sha256
mkdir -p launch-tools-v620-r4
tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp4-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-v620-r4/portable/launch.py --config v620-w4-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-v620-serve-r5.tar.gz · SHA-256
launch-tools-v620-r4.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
Portable observer compatibility
The evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged.
All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 39.296 · eligible10K 37.676 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 5806,
    "content_chars": 5723,
    "content_has_think_close": false,
    "elapsed_seconds": 151.7962831039913,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T13:19:26Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.011801122920587659,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 5723,
    "qwen_answer_text_sha256": "8e59252e3ad8a4168398b9374ee274fc88656e2899d69096795556d8456cfa40",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 5723,
    "qwen_visible_text_sha256": "8e59252e3ad8a4168398b9374ee274fc88656e2899d69096795556d8456cfa40",
    "raw_has_think_close": false,
    "reasoning_chars": 20749,
    "text_chars": 26473,
    "text_sha256": "c8e300e905259796a9b14c161b6d0a6c59a73258ff4f5c6a4e8572a569d17edf",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 150.68790842499584,
    "vllm_decode_tps": 38.529966078133654,
    "vllm_generation_tokens_delta": 5806.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.0972954761236906,
    "vllm_prefill_tps": 431.05982872628005,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 38.24863087077353
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 52.00303460191935,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T13:20:18Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.01141878985799849,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 7704,
    "text_chars": 7704,
    "text_sha256": "e53a38a234f910abe1a3b42cc909a8dec4b0f5e024f3f3cb5552ae1133004552",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 50.895956503925845,
    "vllm_decode_tps": 39.29585250737415,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.0962058920413256,
    "vllm_prefill_tps": 431.48828466812193,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 38.45929406446952
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 17402,
    "content_has_think_close": true,
    "elapsed_seconds": 266.53216635319404,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T13:24:45Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.007715465035289526,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 10070,
    "qwen_answer_text_sha256": "96dd1749d2bef179d10b9d1b6a0afc735325a7ae603c43af72851efd69b35bb2",
    "qwen_gate_close_in_content": true,
    "qwen_gate_close_in_raw": true,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 10070,
    "qwen_visible_text_sha256": "96dd1749d2bef179d10b9d1b6a0afc735325a7ae603c43af72851efd69b35bb2",
    "raw_has_think_close": true,
    "reasoning_chars": 26874,
    "text_chars": 44277,
    "text_sha256": "a080354fab849841fae9032656d66dbbd478d408ad07b3176e2ba72fdf8cf58d",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 265.41904551698826,
    "vllm_decode_tps": 37.67627142401108,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.0997616411186755,
    "vllm_prefill_tps": 430.0931968484237,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 37.518923651220916
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS6,146 natural tokens2K 40.36110K 38.396 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T14:35:39.487703175Z",
      "FinishedAt": "2026-10-26T15:14:01.113195407Z"
    },
    "devices_after": [
      {
        "card": 5,
        "render": 133,
        "pci": "0000:16:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:19:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 7,
        "render": 135,
        "pci": "0000:1c:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 8,
        "render": 136,
        "pci": "0000:1f:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T15:14:01.539984+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 40.36122244861641,
  "retest_10K_TPS": 38.395805792032654,
  "retest_strict_tokens": 6146,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 6146,
      "elapsed_seconds": 156.78889757196885,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 155.68011727603152,
      "vllm_decode_tps": 39.47838752653764,
      "vllm_generation_tokens_delta": 6146.0,
      "vllm_prefill_seconds_delta": 1.095762872020714,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 50.6597384349443,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 49.5525129979942,
      "vllm_decode_tps": 40.36122244861641,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 1.0948759620077908,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 261.55461658199783,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 260.4451135669369,
      "vllm_decode_tps": 38.395805792032654,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 1.094910062965937,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 1.0948759620077908,
  "retest_2K_total_seconds": 50.6597384349443,
  "retest_10K_prefill_seconds": 1.094910062965937,
  "retest_10K_total_seconds": 261.55461658199783,
  "retest_strict_prefill_seconds": 1.095762872020714,
  "retest_strict_total_seconds": 156.78889757196885,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": {
    "all_four_quantized_module_receipts_passed": true,
    "generated_quantized_call_support_complete": true,
    "graphs": [
      {
        "rank": 0,
        "pid": 83,
        "source_sha256": "b8f11a3f101d5df627672bb7bf0cdaf702e53502badd899ed5b9faffb523dcaa",
        "generated_gemm_calls": 256
      },
      {
        "rank": 1,
        "pid": 84,
        "source_sha256": "5bac1879e929112251a66ce5b2b42bddb1c9446e72115e2d982d4d8d7c6e749c",
        "generated_gemm_calls": 256
      },
      {
        "rank": 2,
        "pid": 85,
        "source_sha256": "7f3599a6704c7950176ff6c909c3a8a894837947a21e37b6357c45ecee9ae4de",
        "generated_gemm_calls": 256
      },
      {
        "rank": 3,
        "pid": 86,
        "source_sha256": "b5631bb04aff8712641b794dd9379862ed15578210663671fae4dd54f19b6a5c",
        "generated_gemm_calls": 256
      }
    ],
    "attention_KV_observations": [
      {
        "rank": 3,
        "pid": 86,
        "attention_layers": [
          3,
          7,
          11,
          15,
          19,
          23,
          27,
          31,
          35,
          39,
          43,
          47,
          51,
          55,
          59,
          63
        ],
        "logical_view_bytes": 14758969344,
        "Mamba_SSM_config_dtype": "float32",
        "limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
      },
      {
        "rank": 1,
        "pid": 84,
        "attention_layers": [
          3,
          7,
          11,
          15,
          19,
          23,
          27,
          31,
          35,
          39,
          43,
          47,
          51,
          55,
          59,
          63
        ],
        "logical_view_bytes": 14758969344,
        "Mamba_SSM_config_dtype": "float32",
        "limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
      },
      {
        "rank": 0,
        "pid": 83,
        "attention_layers": [
          3,
          7,
          11,
          15,
          19,
          23,
          27,
          31,
          35,
          39,
          43,
          47,
          51,
          55,
          59,
          63
        ],
        "logical_view_bytes": 14758969344,
        "Mamba_SSM_config_dtype": "float32",
        "limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
      },
      {
        "rank": 2,
        "pid": 85,
        "attention_layers": [
          3,
          7,
          11,
          15,
          19,
          23,
          27,
          31,
          35,
          39,
          43,
          47,
          51,
          55,
          59,
          63
        ],
        "logical_view_bytes": 14758969344,
        "Mamba_SSM_config_dtype": "float32",
        "limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
      }
    ],
    "all_four_KV_receipts_passed": true,
    "limits": "Static generated call support plus bound loaded-module receipts/native mappings and completed startup. Not a GPU completion trace or correctness verdict."
  },
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_logged_model_load_delta_GiB": 13.47,
  "retest_logged_model_load_seconds": 59.703993,
  "retest_logged_available_KV_GiB": 13.76,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 13.76,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 225008,
  "retest_logged_context_capacity_x": 6.72,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "v620-w4-tp4-mtp0",
    "image": "sha256:b97918608d83591ad38f818bb4e944b5aa8d7d2fa9df2f354d47f1db190d9fcb",
    "container": "c1f1d4225224fe57f881a4a2eea02bb60ad836c7c0e96adf895bcd78d6ec7694",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 13.47,
    "retest_logged_model_load_seconds": 59.703993,
    "retest_logged_available_KV_GiB": 13.76,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 13.76,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 225008,
    "retest_logged_context_capacity_x": 6.72,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w4-tp4-mtp0/service-final.log",
    "source_log_sha256": "0e0bfd459de2a7ef66c6f6225696c4722266fd04a763121e3bb1ebc028697ca3",
    "sealed_index_sha256": "30b88085c5128904cba9bfbad035f6f909fd5b6ae8920387bc8fd26a2807844b",
    "source_lines": {
      "model_load": {
        "line": 120,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 14:38:21 [gpu_model_runner.py:4579] Model loading took 13.47 GiB memory and 59.703993 seconds"
      },
      "automatic_KV_budget": {
        "line": 141,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 14:41:09 [gpu_worker.py:453] Available KV cache memory: 13.76 GiB"
      },
      "grouped_KV_tokens": {
        "line": 144,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 14:41:10 [kv_cache_utils.py:1314] GPU KV cache size: 225,008 tokens"
      },
      "context_capacity": {
        "line": 145,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 14:41:10 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.72x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w4-tp4-mtp0/memory-log-report.json"
}
Retest KV budget: 13.760 GiB (automatic, rounded)225,008 grouped KV tokens · 6.72× reported capacityRetest memory reports and limits
Evidence and limits

selected historical RTN4 reference

{
  "strict_tokens": 15275,
  "source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.full_model_results.RTN4",
  "source_revision": "999813806eae1e07fffb3baccd359595fbcc2358",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "comparison_limit": "One quantized sample; no repeated-sample spread. Quantized TP route, Head32/compiler mode and rootSSD/cache warmth differ from preserved FP16 control. No FP16 qualification.",
  "latency_sources": "[{\"path\": \"/home/joe/PhoneSync/Documents/gfx1030/.work/quantized-baselines/rtn4-service-r1/sealed-mirror/retained/evidence/ladder2/c1_2000_proxy/summary.json\", \"sha256\": \"2c0f21a7379f624b75090fba65cb533335f92c6790cc8a85ce3084f140349814\"}, {\"path\": \"/home/joe/PhoneSync/Documents/gfx1030/.work/quantized-baselines/rtn4-service-r1/sealed-mirror/retained/evidence/ladder2/c1_10000_proxy/summary.json\", \"sha256\": \"a743ecb3b71d4aad74a8ee25eae600b9fd3577a2b9bedd6104f33146a99f46f1\"}]",
  "sampled_peak_VRAM_bytes_by_PCI": {
    "0000:16:00.0": 22627618816,
    "0000:19:00.0": 22644396032,
    "0000:1c:00.0": 22644400128,
    "0000:1f:00.0": 22594056192
  },
  "memory_accounting_limit": "Packed target matrices exclude FP16 embeddings/head/norm/recurrent weights. Sampled physical VRAM is not a guaranteed continuous peak.",
  "memory_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#full_model_results.RTN4.memory",
  "memory_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
  "retest_strict_tokens": 6146,
  "retest_MTP_accounting_by_tier": {},
  "retest_logged_available_KV_GiB": 13.76,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 13.76,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 225008,
  "retest_memory_log_report": {
    "profile": "v620-w4-tp4-mtp0",
    "image": "sha256:b97918608d83591ad38f818bb4e944b5aa8d7d2fa9df2f354d47f1db190d9fcb",
    "container": "c1f1d4225224fe57f881a4a2eea02bb60ad836c7c0e96adf895bcd78d6ec7694",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 13.47,
    "retest_logged_model_load_seconds": 59.703993,
    "retest_logged_available_KV_GiB": 13.76,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 13.76,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 225008,
    "retest_logged_context_capacity_x": 6.72,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w4-tp4-mtp0/service-final.log",
    "source_log_sha256": "0e0bfd459de2a7ef66c6f6225696c4722266fd04a763121e3bb1ebc028697ca3",
    "sealed_index_sha256": "30b88085c5128904cba9bfbad035f6f909fd5b6ae8920387bc8fd26a2807844b",
    "source_lines": {
      "model_load": {
        "line": 120,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 14:38:21 [gpu_model_runner.py:4579] Model loading took 13.47 GiB memory and 59.703993 seconds"
      },
      "automatic_KV_budget": {
        "line": 141,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 14:41:09 [gpu_worker.py:453] Available KV cache memory: 13.76 GiB"
      },
      "grouped_KV_tokens": {
        "line": 144,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 14:41:10 [kv_cache_utils.py:1314] GPU KV cache size: 225,008 tokens"
      },
      "context_capacity": {
        "line": 145,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 14:41:10 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.72x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w4-tp4-mtp0/memory-log-report.json",
  "sampled_peak_VRAM_GiB": 21.08924102783203
}
V620Current Qwen3.84-bit RTNQwen/Qwen3.8-27BTP2MTP off PASS6876 tokens 32.80730.823 0.774 / 61.75210K: 0.827 / 325.279Strict: 0.755 / 217.468 5.89 / 12.71packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:b7beecf0afb95cd5501aa395497b11d9eaff91f7e967af31335e825e6715f553
Runtime image config: sha256:a16a6f2d47e7ac6ffc55d7a3d9298cad7e03601d8a30886421e905c92aeae386
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256
mkdir -p launch-tools-v620-serve-r5
tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp2-mtp0
GPU_RENDER_NODES=renderD128,renderD129

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w4-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Preflight cleanup compatibility
After its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged.
Retained lab Docker command · not portable
docker run -d \
  --name serve-v620-w4-tp2-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --cpuset-mems 0 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/card5:/dev/dri/card5:rwm \
  --device /dev/dri/card6:/dev/dri/card6:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
  -e ROCM_VERSION=7.14.0 \
  -e TORCH_VERSION=2.12.0+rocm7.14.0 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
  -e PYTORCH_ROCM_ARCH=gfx1030 \
  -e GPU_ARCHS=gfx1030 \
  -e AMDGPU_TARGETS=gfx1030 \
  -e CMAKE_HIP_ARCHITECTURES=gfx1030 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e VNEXT_GDN_HEAD32=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp0/cache/torch_extensions,dst=/root/.cache/torch_extensions \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp0/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
  --entrypoint /usr/bin/env \
  sha256:a16a6f2d47e7ac6ffc55d7a3d9298cad7e03601d8a30886421e905c92aeae386 \
  V756F94_BOUNDARY_MODE=posted \
  DO_NOT_TRACK=1 \
  GPU_DEVICE_ORDINAL=0,1 \
  GPU_MAX_HW_QUEUES=1 \
  HSA_OVERRIDE_GFX_VERSION=10.3.0 \
  OMP_NUM_THREADS=4 \
  TEMP=/tmp/torchinductor_root \
  TMPDIR=/tmp/torchinductor_root \
  TMP=/tmp/torchinductor_root \
  TORCH_BLAS_PREFER_HIPBLASLT=0 \
  TRITON_CACHE_DIR=/root/.triton \
  VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  VLLM_GPU_MEMORY_UTILIZATION=0.95 \
  VLLM_LOGGING_LEVEL=INFO \
  VLLM_PORT=9002 \
  VLLM_RDNA_FORCE_FP16=1 \
  VLLM_WORKER_MULTIPROC_METHOD=spawn \
  NCCL_P2P_DISABLE=0 \
  NCCL_ALGO=Tree \
  NCCL_PROTO=LL \
  NCCL_MIN_NCHANNELS=4 \
  NCCL_MAX_NCHANNELS=4 \
  NCCL_NTHREADS=256 \
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
  HIP_VISIBLE_DEVICES=0,1 \
  CUDA_VISIBLE_DEVICES=0,1 \
  ROCR_VISIBLE_DEVICES=0,1 \
  VLLM_CACHE_ROOT=/root/.cache/vllm \
  PYTORCH_ROCM_ARCH=gfx1030 \
  PYTHONHASHSEED=0 \
  /opt/venv/bin/python \
  -m \
  vllm.vnext_rtn_entry \
  serve \
  /opt/local-models/qwen38-27b-hf \
  --served-model-name \
  Qwen/Qwen3.8-27B \
  --enable-auto-tool-choice \
  --tool-call-parser \
  qwen3_coder \
  --dtype \
  half \
  --host \
  0.0.0.0 \
  --port \
  9002 \
  --tensor-parallel-size \
  2 \
  --max-model-len \
  131072 \
  --gpu-memory-utilization \
  0.95 \
  --trust-remote-code \
  --generation-config \
  vllm \
  -O=3 \
  --reasoning-parser \
  qwen3 \
  --async-scheduling \
  --hf-overrides \
  '{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":4,"group_size":128}}' \
  --language-model-only \
  --max-num-seqs \
  2 \
  --max-num-batched-tokens \
  2048 \
  --mamba-ssm-cache-dtype \
  float32 \
  --no-enable-prefix-caching \
  --no-enforce-eager \
  --attention-backend \
  TRITON_ATTN \
  --disable-custom-all-reduce \
  --seed \
  0 \
  --worker-cls \
  vllm.vnext_rtn_worker.Worker \
  --quantization \
  vnext_rtn
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-r4.tar.gz.sha256
mkdir -p launch-tools-v620-r4
tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp2-mtp0
GPU_RENDER_NODES=renderD128,renderD129

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-v620-r4/portable/launch.py --config v620-w4-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-v620-serve-r5.tar.gz · SHA-256
launch-tools-v620-r4.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
Portable observer compatibility
The evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged.
All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 32.905 · eligible10K 30.623 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 13863,
    "content_chars": 13741,
    "content_has_think_close": false,
    "elapsed_seconds": 468.5006339759566,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T02:20:17Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.018792743096128106,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 13741,
    "qwen_answer_text_sha256": "2e9c97ea581575ec316e08addf3ccc841d38debee190556442a487c75930119a",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 13741,
    "qwen_visible_text_sha256": "2e9c97ea581575ec316e08addf3ccc841d38debee190556442a487c75930119a",
    "raw_has_think_close": false,
    "reasoning_chars": 45894,
    "text_chars": 59636,
    "text_sha256": "1d3ba196ea092142805106b38c0a5d382fc2c7dc33d86a2e796aa58da9891812",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 467.7273157769814,
    "vllm_decode_tps": 29.63906432740837,
    "vllm_generation_tokens_delta": 13863.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.7524218091275543,
    "vllm_prefill_tps": 628.6367490443312,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 29.59014138860578
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 61.55541231902316,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T02:21:18Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.01002282602712512,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 8669,
    "text_chars": 8669,
    "text_sha256": "0c4f91d92e62192ace2c7cc81141fa7fd2ee22ef862689251a23d577f289357e",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 60.78103750408627,
    "vllm_decode_tps": 32.90499935716861,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.7525132559239864,
    "vllm_prefill_tps": 628.5603559491038,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 32.49105033420298
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 327.3320147569757,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T02:26:46Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.014586291974410415,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 35150,
    "text_chars": 35150,
    "text_sha256": "9328393526b4b230ec4a5b020a16cb345bc0602ac1ca9c27d3c20bbd3698125e",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 326.5560789539013,
    "vllm_decode_tps": 30.622611687506396,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.7571389379445463,
    "vllm_prefill_tps": 624.7202148711086,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 30.550021229742523
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS8,472 natural tokens2K 33.22610K 30.907 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T16:13:32.669837852Z",
      "FinishedAt": "2026-10-26T16:59:16.661768663Z"
    },
    "devices_after": [
      {
        "card": 5,
        "render": 133,
        "pci": "0000:16:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:19:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T16:59:16.794941+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 33.226071157256506,
  "retest_10K_TPS": 30.90653983479376,
  "retest_strict_tokens": 8472,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 8472,
      "elapsed_seconds": 271.0291323700221,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 270.2647703959374,
      "vllm_decode_tps": 31.347037897645837,
      "vllm_generation_tokens_delta": 8472.0,
      "vllm_prefill_seconds_delta": 0.7526620751013979,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 60.957591224927455,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 60.193695202004164,
      "vllm_decode_tps": 33.226071157256506,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 0.7528414420085028,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 324.3207754510222,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 323.556116390042,
      "vllm_decode_tps": 30.90653983479376,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 0.7524013130459934,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 0.7528414420085028,
  "retest_2K_total_seconds": 60.957591224927455,
  "retest_10K_prefill_seconds": 0.7524013130459934,
  "retest_10K_total_seconds": 324.3207754510222,
  "retest_strict_prefill_seconds": 0.7526620751013979,
  "retest_strict_total_seconds": 271.0291323700221,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": {
    "all_two_quantized_module_receipts_passed": true,
    "generated_quantized_call_support_complete": true,
    "graphs": [
      {
        "rank": 0,
        "pid": 83,
        "source_sha256": "40be7ddb392affdf12e3436e1a59fb43d565da92e234b2d5df1f2c4fb7068600",
        "generated_gemm_calls": 256
      },
      {
        "rank": 1,
        "pid": 84,
        "source_sha256": "6ac2e022f4974aab00e5590d8c15af8a94ed08c16b0acd84a38877ae853b9e87",
        "generated_gemm_calls": 256
      }
    ],
    "attention_KV_observations": [
      {
        "rank": 1,
        "pid": 84,
        "attention_layers": [
          3,
          7,
          11,
          15,
          19,
          23,
          27,
          31,
          35,
          39,
          43,
          47,
          51,
          55,
          59,
          63
        ],
        "logical_view_bytes": 1104674816,
        "Mamba_SSM_config_dtype": "float32",
        "limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
      },
      {
        "rank": 0,
        "pid": 83,
        "attention_layers": [
          3,
          7,
          11,
          15,
          19,
          23,
          27,
          31,
          35,
          39,
          43,
          47,
          51,
          55,
          59,
          63
        ],
        "logical_view_bytes": 1104674816,
        "Mamba_SSM_config_dtype": "float32",
        "limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
      }
    ],
    "all_two_KV_receipts_passed": true,
    "limits": "Static generated call support plus bound loaded-module receipts/native mappings and completed startup. Not a GPU completion trace or correctness verdict."
  },
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 13639860224,
    "1": 13642035200
  },
  "retest_memory_observation_count": 33,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-w4-tp2-mtp0/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 25.59,
  "retest_logged_model_load_seconds": 114.199732,
  "retest_logged_available_KV_GiB": 1.05,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 1.05,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 7840,
  "retest_logged_context_capacity_x": 0.25,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "v620-w4-tp2-mtp0",
    "image": "sha256:a16a6f2d47e7ac6ffc55d7a3d9298cad7e03601d8a30886421e905c92aeae386",
    "container": "63d508581fc91c4a06d201f732ab17b0e41917d68c06a98af7a60049e74f0225",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 25.59,
    "retest_logged_model_load_seconds": 114.199732,
    "retest_logged_available_KV_GiB": 1.05,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 1.05,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 7840,
    "retest_logged_context_capacity_x": 0.25,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w4-tp2-mtp0/service-final.log",
    "source_log_sha256": "1de293b6b62bc00ad5bad8ca7ae9b8602161b0137af4ffd056b3e51f09b244b3",
    "sealed_index_sha256": "d5b582c7896849a6be95b24fdc53dcdd4cecd5c8e248a63f69ae296aee7afd9d",
    "source_lines": {
      "model_load": {
        "line": 92,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 16:17:07 [gpu_model_runner.py:4579] Model loading took 25.59 GiB memory and 114.199732 seconds"
      },
      "automatic_KV_budget": {
        "line": 104,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 16:20:42 [gpu_worker.py:453] Available KV cache memory: 1.05 GiB"
      },
      "grouped_KV_tokens": {
        "line": 107,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 16:20:42 [kv_cache_utils.py:1314] GPU KV cache size: 7,840 tokens"
      },
      "context_capacity": {
        "line": 108,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 16:20:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 0.25x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w4-tp2-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 12.705135345458984
}
Retest KV budget: 1.050 GiB (automatic, rounded)7,840 grouped KV tokens · 0.25× reported capacityRetest sampled VRAM: 12.705 GiB max/cardRetest memory reports and limits
Evidence and limits

completed matrix cell

{
  "strict_tokens": 6876,
  "source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.local_RTN4_TP_MTP_matrix.ordered_cells[TP=2,MTP_tokens=0].result",
  "source_revision": "7a6d45978c2ccecdf704899625b813cab2aa7979",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "recorded_configuration_details": {
    "strict_valid": true,
    "strict_tokens": 6876,
    "strict_diagnostic_TPS": 31.730622009286282,
    "official_fixed2K_TPS": 32.806665219045115,
    "diagnostic_fixed10K_TPS": 30.822545818435916,
    "source": "7a6d45978c2ccecdf704899625b813cab2aa7979",
    "image": "sha256:a16a6f2d47e7ac6ffc55d7a3d9298cad7e03601d8a30886421e905c92aeae386",
    "sample_count_per_tier": 1,
    "metrics": {
      "strict": {
        "completion_tokens": 6876,
        "elapsed_seconds": 217.46813429001486,
        "vllm_prefill_seconds_delta": 0.7545584980107378,
        "vllm_decode_seconds_delta": 216.69918723899173,
        "vllm_decode_tps": 31.730622009286282,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_generation_tokens_delta": 6876.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0
      },
      "fixed2K": {
        "completion_tokens": 2000,
        "elapsed_seconds": 61.752023694018135,
        "vllm_prefill_seconds_delta": 0.7744208740186878,
        "vllm_decode_seconds_delta": 60.96322154800873,
        "vllm_decode_tps": 32.806665219045115,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_generation_tokens_delta": 2000.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0
      },
      "fixed10K": {
        "completion_tokens": 10000,
        "elapsed_seconds": 325.2792088240094,
        "vllm_prefill_seconds_delta": 0.8272731059987564,
        "vllm_decode_seconds_delta": 324.437833879987,
        "vllm_decode_tps": 30.822545818435916,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_generation_tokens_delta": 10000.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0
      }
    },
    "closure": {
      "container_id": "69d228271089fe5543cb2c4fcaf89af086fc93ce8b5e3d09571e659e3b6e9d87",
      "image_id": "sha256:a16a6f2d47e7ac6ffc55d7a3d9298cad7e03601d8a30886421e905c92aeae386",
      "exit_code": 0,
      "OOMKilled": false,
      "restart_count": 0,
      "bank_idle": true,
      "devices": [
        {
          "pci": "0000:16:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        },
        {
          "pci": "0000:19:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        }
      ],
      "started_at": "2026-10-22T18:33:31.270581575Z",
      "finished_at": "2026-10-22T19:16:28.568820286Z",
      "closed_at": "2026-10-22T19:16:28.715957+00:00",
      "kernel_fault_matches": [],
      "startup_requests_completed_and_drained": 3,
      "official_phase_samples": 11,
      "complete_ladders": 1,
      "strict_valid": true,
      "official_fixed_2000_tps": 32.806665219045115,
      "qualifying_ladders": 0,
      "FP16_qualification": false,
      "baseline_kind": "RTN4"
    },
    "archive": {
      "path": "/home/ai/vnext-v620-qwen38-rtn4-tp2-service-r0-20261022/raw-evidence.tar.gz",
      "bytes": 734829,
      "sha256": "782012dac7c921179e01dc090de11770841eac6eb853d3964012cc12115c269b",
      "files": 236,
      "index_sha256": "285267ead1cb3460651e70de1e025fadc91e9f3dda778f524f434eb997cb3979"
    },
    "independent_review": {
      "path": ".work/helpers/projection-analyst/.work/native-mtp-review/TP2-sealed/review.json",
      "SHA256": "37f2bbad942bf6980f3827dae33d70447bbae553a64733d4b680b4bcfc82c484"
    },
    "FP16_qualification": false,
    "limits": "One sample per tier; historical TP4 cache/request history differs, so observed TP difference is not isolated causal attribution or repeatability. Context configured131072; runtime-reported capacity limitation retained in deployment observation.",
    "local_evidence": ".work/rtn4-tp-mtp-matrix/tp2-service-r0/sealed-mirror/retained/evidence"
  },
  "detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
  "detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
  "detail_source_pointer": "local_RTN4_TP_MTP_matrix.cells[0]",
  "sampled_VRAM_startup_max_bytes_by_rank": [
    13639901184,
    13642076160
  ],
  "memory_accounting_limit": "Retained own-device observation after admission; not a guaranteed continuous peak.",
  "memory_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#local_RTN4_TP_MTP_matrix",
  "memory_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
  "retest_strict_tokens": 8472,
  "retest_MTP_accounting_by_tier": {},
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 13639860224,
    "1": 13642035200
  },
  "retest_memory_observation_count": 33,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-w4-tp2-mtp0/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 1.05,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 1.05,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 7840,
  "retest_memory_log_report": {
    "profile": "v620-w4-tp2-mtp0",
    "image": "sha256:a16a6f2d47e7ac6ffc55d7a3d9298cad7e03601d8a30886421e905c92aeae386",
    "container": "63d508581fc91c4a06d201f732ab17b0e41917d68c06a98af7a60049e74f0225",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 25.59,
    "retest_logged_model_load_seconds": 114.199732,
    "retest_logged_available_KV_GiB": 1.05,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 1.05,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 7840,
    "retest_logged_context_capacity_x": 0.25,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w4-tp2-mtp0/service-final.log",
    "source_log_sha256": "1de293b6b62bc00ad5bad8ca7ae9b8602161b0137af4ffd056b3e51f09b244b3",
    "sealed_index_sha256": "d5b582c7896849a6be95b24fdc53dcdd4cecd5c8e248a63f69ae296aee7afd9d",
    "source_lines": {
      "model_load": {
        "line": 92,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 16:17:07 [gpu_model_runner.py:4579] Model loading took 25.59 GiB memory and 114.199732 seconds"
      },
      "automatic_KV_budget": {
        "line": 104,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 16:20:42 [gpu_worker.py:453] Available KV cache memory: 1.05 GiB"
      },
      "grouped_KV_tokens": {
        "line": 107,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 16:20:42 [kv_cache_utils.py:1314] GPU KV cache size: 7,840 tokens"
      },
      "context_capacity": {
        "line": 108,
        "text": "(EngineCore_DP0 pid=65) INFO 10-26 16:20:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 0.25x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w4-tp2-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 12.705135345458984,
  "sampled_peak_VRAM_GiB": 12.70517349243164
}
V620Current Qwen3.84-bit RTNQwen/Qwen3.8-27BTP4MTP22K accepted 1091 / 1820 (59.95%) PASS8141 tokens 39.70827.714 1.125 / 51.50510K: 1.124 / 361.961Strict: 1.126 / 260.606 3.00 / 28.68packed target/draft projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:322972df1a8ebf776e497ff3cfecad1ac2a2576d9fb72541c52e22212de6ffc7
Runtime image config: sha256:762cae7dd32e7f3542a57754b547742471c371a40d287a115ac349616ec64681
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256
mkdir -p launch-tools-v620-serve-r5
tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp4-mtp2
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w4-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Preflight cleanup compatibility
After its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged.
Retained lab Docker command · not portable
docker run -d \
  --name serve-v620-w4-tp4-mtp2-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --cpuset-mems 0 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/card5:/dev/dri/card5:rwm \
  --device /dev/dri/card6:/dev/dri/card6:rwm \
  --device /dev/dri/card7:/dev/dri/card7:rwm \
  --device /dev/dri/card8:/dev/dri/card8:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
  --device /dev/dri/renderD136:/dev/dri/renderD136:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
  -e ROCM_VERSION=7.14.0 \
  -e TORCH_VERSION=2.12.0+rocm7.14.0 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
  -e PYTORCH_ROCM_ARCH=gfx1030 \
  -e GPU_ARCHS=gfx1030 \
  -e AMDGPU_TARGETS=gfx1030 \
  -e CMAKE_HIP_ARCHITECTURES=gfx1030 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e VNEXT_GDN_HEAD32=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp2/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp2/cache/torch_extensions,dst=/root/.cache/torch_extensions \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp2/cache/.triton,dst=/root/.triton \
  --entrypoint /usr/bin/env \
  sha256:762cae7dd32e7f3542a57754b547742471c371a40d287a115ac349616ec64681 \
  V756F94_BOUNDARY_MODE=posted \
  DO_NOT_TRACK=1 \
  GPU_DEVICE_ORDINAL=0,1,2,3 \
  GPU_MAX_HW_QUEUES=1 \
  HSA_OVERRIDE_GFX_VERSION=10.3.0 \
  OMP_NUM_THREADS=4 \
  TEMP=/tmp/torchinductor_root \
  TMPDIR=/tmp/torchinductor_root \
  TMP=/tmp/torchinductor_root \
  TORCH_BLAS_PREFER_HIPBLASLT=0 \
  TRITON_CACHE_DIR=/root/.triton \
  VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  VLLM_GPU_MEMORY_UTILIZATION=0.95 \
  VLLM_LOGGING_LEVEL=INFO \
  VLLM_PORT=9002 \
  VLLM_RDNA_FORCE_FP16=1 \
  VLLM_WORKER_MULTIPROC_METHOD=spawn \
  NCCL_P2P_DISABLE=0 \
  NCCL_ALGO=Tree \
  NCCL_PROTO=LL \
  NCCL_MIN_NCHANNELS=4 \
  NCCL_MAX_NCHANNELS=4 \
  NCCL_NTHREADS=256 \
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
  HIP_VISIBLE_DEVICES=0,1,2,3 \
  CUDA_VISIBLE_DEVICES=0,1,2,3 \
  ROCR_VISIBLE_DEVICES=0,1,2,3 \
  VLLM_CACHE_ROOT=/root/.cache/vllm \
  PYTORCH_ROCM_ARCH=gfx1030 \
  PYTHONHASHSEED=0 \
  /opt/venv/bin/python \
  -m \
  vllm.vnext_rtn_entry \
  serve \
  /opt/local-models/qwen38-27b-hf \
  --served-model-name \
  Qwen/Qwen3.8-27B \
  --enable-auto-tool-choice \
  --tool-call-parser \
  qwen3_coder \
  --dtype \
  half \
  --host \
  0.0.0.0 \
  --port \
  9002 \
  --tensor-parallel-size \
  4 \
  --speculative-config \
  '{"method":"mtp","num_speculative_tokens":2}' \
  --max-model-len \
  131072 \
  --gpu-memory-utilization \
  0.95 \
  --trust-remote-code \
  --generation-config \
  vllm \
  -O=3 \
  --reasoning-parser \
  qwen3 \
  --async-scheduling \
  --hf-overrides \
  '{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":4,"group_size":128}}' \
  --language-model-only \
  --max-num-seqs \
  2 \
  --max-num-batched-tokens \
  2048 \
  --mamba-ssm-cache-dtype \
  float32 \
  --no-enable-prefix-caching \
  --no-enforce-eager \
  --attention-backend \
  TRITON_ATTN \
  --disable-custom-all-reduce \
  --seed \
  0 \
  --worker-cls \
  vllm.vnext_rtn_worker.Worker \
  --quantization \
  vnext_rtn
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-r4.tar.gz.sha256
mkdir -p launch-tools-v620-r4
tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp4-mtp2
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-v620-r4/portable/launch.py --config v620-w4-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-v620-serve-r5.tar.gz · SHA-256
launch-tools-v620-r4.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
Portable observer compatibility
The evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged.
All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict FAIL2K 42.203 · diagnostic / ineligible10K 29.563 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 9075,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 269.3403684841469,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T03:35:53Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.012874047039076686,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 15946,
    "text_chars": 15946,
    "text_sha256": "1c2824a1e900e1f1c7b125b67ac45cbbe30faba1360283e1208dd506b70850cb",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 268.0994095131755,
    "vllm_decode_tps": 33.84938451180743,
    "vllm_generation_tokens_delta": 9075.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.2195868210401386,
    "vllm_prefill_tps": 387.8362670372221,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 33.693426837849394
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 48.53436310100369,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T03:36:44Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.006693734088912606,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 8781,
    "text_chars": 8781,
    "text_sha256": "5096b049167f360900c93aca8b79452d6b9cfaab26f2fd552a024bf16485afa0",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 47.39049668004736,
    "vllm_decode_tps": 42.20255410072653,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.125919857993722,
    "vllm_prefill_tps": 420.1009482529594,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 41.207916869906136
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 17340,
    "content_has_think_close": true,
    "elapsed_seconds": 339.4023382640444,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T03:42:23Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.018082647817209363,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 11257,
    "qwen_answer_text_sha256": "1c25c1dd1cc8d000f9c74c3cf23b6a54811153dc389a271e367bea0a1035cc20",
    "qwen_gate_close_in_content": true,
    "qwen_gate_close_in_raw": true,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 11257,
    "qwen_visible_text_sha256": "1c25c1dd1cc8d000f9c74c3cf23b6a54811153dc389a271e367bea0a1035cc20",
    "raw_has_think_close": true,
    "reasoning_chars": 30737,
    "text_chars": 48078,
    "text_sha256": "26508a10a01e51cdd70e874bc2518f39a0e0a445e04b6f15d265a29417caeef4",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 338.26377589209005,
    "vllm_decode_tps": 29.562728003101675,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.1227409939747304,
    "vllm_prefill_tps": 421.2903978196113,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 29.463556589348872
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestMTP counter replayPASS12,049 natural tokens2K 50.03810K 29.474 · diagnostic2K MTP accepted 1247 / 1506 (82.80%)
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T15:18:34.551273762Z",
      "FinishedAt": "2026-10-26T16:11:20.878215369Z"
    },
    "devices_after": [
      {
        "card": 5,
        "render": 133,
        "pci": "0000:16:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:19:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 7,
        "render": 135,
        "pci": "0000:1c:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 8,
        "render": 136,
        "pci": "0000:1f:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T16:11:21.024146+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 50.03782778839164,
  "retest_10K_TPS": 29.473873646089725,
  "retest_strict_tokens": 12049,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 12049,
      "elapsed_seconds": 627.4919198229909,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 626.3237443990074,
      "vllm_decode_tps": 19.237654819492256,
      "vllm_generation_tokens_delta": 12049.0,
      "vllm_prefill_seconds_delta": 1.1552683959016576,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 41.104696169961244,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 39.969760647043586,
      "vllm_decode_tps": 50.03782778839164,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 1.1210625360254198,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 340.4189020500053,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 339.2835336161079,
      "vllm_decode_tps": 29.473873646089725,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 1.1211414389545098,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 1.1210625360254198,
  "retest_2K_total_seconds": 41.104696169961244,
  "retest_10K_prefill_seconds": 1.1211414389545098,
  "retest_10K_total_seconds": 340.4189020500053,
  "retest_strict_prefill_seconds": 1.1552683959016576,
  "retest_strict_total_seconds": 627.4919198229909,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 7368.0,
        "vllm:spec_decode_num_accepted_tokens_total": 7368.0,
        "vllm:spec_decode_num_draft_tokens_total": 9360.0,
        "vllm:generation_tokens_total": 12049.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:request_generation_tokens_sum": 12049.0,
        "vllm:spec_decode_num_drafts_total": 4680.0
      },
      "accepted_per_position": {
        "0": 3987.0,
        "1": 3381.0
      },
      "accepted_fraction": 0.7871794871794872,
      "committed_emitted_tokens": 12049.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "58b60fc09292dc78a04103d15cc41f6c3819f438f8ac6ee3f0c3138210f80a55",
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "4743b077a4308e60d98e2fed918ad8f39a0db6c484f585a10c15203b655627e8"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 1247.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_draft_tokens_total": 1506.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1247.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:spec_decode_num_drafts_total": 753.0,
        "vllm:request_generation_tokens_count": 1.0
      },
      "accepted_per_position": {
        "0": 657.0,
        "1": 590.0
      },
      "accepted_fraction": 0.8280212483399735,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "6305a1606c4c60bcf66a5e25d6687e417b165acded425de5114a1a3794b48b8c",
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "9cfd84f27f62d6ed382652c0b7f2037bdc495760cb18bda854876e88093e84e5"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:spec_decode_num_drafts_total": 3887.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6112.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6112.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_draft_tokens_total": 7774.0
      },
      "accepted_per_position": {
        "0": 3320.0,
        "1": 2792.0
      },
      "accepted_fraction": 0.7862104450733213,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "908c5b8ee8fecb4e947e3a3fd4ea2d996c36f0608e307d3d7320d069de9912c7",
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "a2ef83da43f40f2d0a0a94a6cf25ed6a3b46325cdc00aaf620f9aaf5fc322cc3"
      }
    }
  },
  "retest_MTP_2K_accepted": 1247.0,
  "retest_MTP_2K_proposed": 1506.0,
  "retest_MTP_2K_draft_batches": 753.0,
  "retest_MTP_2K_acceptance_percent": 82.80212483399735,
  "retest_loaded_admission": {
    "all_four_quantized_module_receipts_passed": true,
    "all_four_MTP_draft_receipts_passed": true,
    "draft_sharing_observations": [
      {
        "rank": 0,
        "shared_embedding": true,
        "shared_lm_head": true
      },
      {
        "rank": 3,
        "shared_embedding": true,
        "shared_lm_head": true
      },
      {
        "rank": 1,
        "shared_embedding": true,
        "shared_lm_head": true
      },
      {
        "rank": 2,
        "shared_embedding": true,
        "shared_lm_head": true
      }
    ],
    "MTP_execution_evidence": "See separately retained standard speculative counter deltas; load/source observations alone are not invocation proof",
    "generated_quantized_call_support_complete": true,
    "graphs": [
      {
        "rank": 0,
        "pid": 103,
        "source_sha256": "acb7fa3377c9627dad430edf5a9e679e6d929f8b057350e0c6139f8cbb9bb484",
        "generated_gemm_calls": 256
      },
      {
        "rank": 1,
        "pid": 104,
        "source_sha256": "be1dd3311f27d998762cbe71f0e857aa5baf8fa893a059bfbc1c20c597a876c2",
        "generated_gemm_calls": 256
      },
      {
        "rank": 2,
        "pid": 105,
        "source_sha256": "bfff37de3c45381b038055a2dd7d544c6dbe07f5339fc55d6aa6d67d553c5419",
        "generated_gemm_calls": 256
      },
      {
        "rank": 3,
        "pid": 106,
        "source_sha256": "5520ef1773b673749e04ee7e6584940fe22dc49779fd00d3eb66cda74d80c541",
        "generated_gemm_calls": 256
      }
    ],
    "attention_KV_observations": [
      {
        "rank": 1,
        "pid": 104,
        "attention_layers": [
          3,
          7,
          11,
          15,
          19,
          23,
          27,
          31,
          35,
          39,
          43,
          47,
          51,
          55,
          59,
          63
        ],
        "logical_view_bytes": 21456486400,
        "mtp_draft_attention_KV": [
          {
            "name": "model.layers.0.self_attn.attn",
            "tensors": [
              {
                "shape": [
                  1637,
                  2,
                  800,
                  1,
                  256
                ],
                "dtype": "torch.float16",
                "device": "cuda:1",
                "bytes": 1341030400
              }
            ]
          }
        ],
        "Mamba_SSM_config_dtype": "float32",
        "limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
      },
      {
        "rank": 0,
        "pid": 103,
        "attention_layers": [
          3,
          7,
          11,
          15,
          19,
          23,
          27,
          31,
          35,
          39,
          43,
          47,
          51,
          55,
          59,
          63
        ],
        "logical_view_bytes": 21456486400,
        "mtp_draft_attention_KV": [
          {
            "name": "model.layers.0.self_attn.attn",
            "tensors": [
              {
                "shape": [
                  1637,
                  2,
                  800,
                  1,
                  256
                ],
                "dtype": "torch.float16",
                "device": "cuda:0",
                "bytes": 1341030400
              }
            ]
          }
        ],
        "Mamba_SSM_config_dtype": "float32",
        "limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
      },
      {
        "rank": 2,
        "pid": 105,
        "attention_layers": [
          3,
          7,
          11,
          15,
          19,
          23,
          27,
          31,
          35,
          39,
          43,
          47,
          51,
          55,
          59,
          63
        ],
        "logical_view_bytes": 21456486400,
        "mtp_draft_attention_KV": [
          {
            "name": "model.layers.0.self_attn.attn",
            "tensors": [
              {
                "shape": [
                  1637,
                  2,
                  800,
                  1,
                  256
                ],
                "dtype": "torch.float16",
                "device": "cuda:2",
                "bytes": 1341030400
              }
            ]
          }
        ],
        "Mamba_SSM_config_dtype": "float32",
        "limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
      },
      {
        "rank": 3,
        "pid": 106,
        "attention_layers": [
          3,
          7,
          11,
          15,
          19,
          23,
          27,
          31,
          35,
          39,
          43,
          47,
          51,
          55,
          59,
          63
        ],
        "logical_view_bytes": 21456486400,
        "mtp_draft_attention_KV": [
          {
            "name": "model.layers.0.self_attn.attn",
            "tensors": [
              {
                "shape": [
                  1637,
                  2,
                  800,
                  1,
                  256
                ],
                "dtype": "torch.float16",
                "device": "cuda:3",
                "bytes": 1341030400
              }
            ]
          }
        ],
        "Mamba_SSM_config_dtype": "float32",
        "limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
      }
    ],
    "all_four_KV_receipts_passed": true,
    "limits": "Static generated call support plus bound loaded-module receipts/native mappings and completed startup. Not a GPU completion trace or correctness verdict."
  },
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
  "retest_logged_model_load_delta_GiB": 5.51,
  "retest_logged_model_load_seconds": 60.300757,
  "retest_logged_available_KV_GiB": 21.24,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 21.24,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 327200,
  "retest_logged_context_capacity_x": 9.46,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "v620-w4-tp4-mtp2",
    "image": "sha256:762cae7dd32e7f3542a57754b547742471c371a40d287a115ac349616ec64681",
    "container": "cb24d05f08e6f9ae6d34cb9601d716a131da7448c1b3cb35342c550e9caedbdd",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 5.51,
    "retest_logged_model_load_seconds": 60.300757,
    "retest_logged_available_KV_GiB": 21.24,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 21.24,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 327200,
    "retest_logged_context_capacity_x": 9.46,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w4-tp4-mtp2/service-final.log",
    "source_log_sha256": "0807c48469b50f3cc209a1eaa5f6f67d3306b2fc2d612248ef1f5e64527c26a5",
    "sealed_index_sha256": "fe7ba6674f7a15c60fd9db595447d9b2e0bb7728107d4990e709736e59b41617",
    "source_lines": {
      "model_load": {
        "line": 154,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 15:21:33 [gpu_model_runner.py:4598] Model loading took 5.51 GiB memory and 60.300757 seconds"
      },
      "automatic_KV_budget": {
        "line": 223,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 15:38:24 [gpu_worker.py:453] Available KV cache memory: 21.24 GiB"
      },
      "grouped_KV_tokens": {
        "line": 228,
        "text": "(EngineCore_DP0 pid=85) INFO 10-26 15:38:24 [kv_cache_utils.py:1314] GPU KV cache size: 327,200 tokens"
      },
      "context_capacity": {
        "line": 229,
        "text": "(EngineCore_DP0 pid=85) INFO 10-26 15:38:24 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 9.46x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w4-tp4-mtp2/memory-log-report.json"
}
Retest KV budget: 21.240 GiB (automatic, rounded)327,200 grouped KV tokens · 9.46× reported capacityRetest memory reports and limits
Evidence and limits

completed matrix cell

{
  "strict_tokens": 8141,
  "source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.local_RTN4_TP_MTP_matrix.ordered_cells[TP=4,MTP_tokens=2].result",
  "source_revision": "1e540c9b0c93a8a336dc7862d01d05f033971cca",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "recorded_configuration_details": {
    "strict_valid": true,
    "strict_tokens": 8141,
    "strict_diagnostic_TPS": 31.37568371725553,
    "official_fixed2K_TPS": 39.708142381881636,
    "diagnostic_fixed10K_TPS": 27.71436282221888,
    "source": "1e540c9b0c93a8a336dc7862d01d05f033971cca",
    "image": "sha256:762cae7dd32e7f3542a57754b547742471c371a40d287a115ac349616ec64681",
    "manifest": "8b48dab1371cf69d2a1ac70be2c6194297fcb1db13d28b180b767bba59e90dc6",
    "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "TP": 4,
    "MTP_tokens": 2,
    "sample_count_per_tier": 1,
    "metrics": {
      "strict": {
        "completion_tokens": 8141,
        "elapsed_seconds": 260.6062123499869,
        "vllm_prefill_seconds_delta": 1.1256862390146125,
        "vllm_decode_seconds_delta": 259.4684493049863,
        "vllm_decode_tps": 31.37568371725553,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_generation_tokens_delta": 8141.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "server_e2e_seconds": 260.6011629104614
      },
      "fixed2K": {
        "completion_tokens": 2000,
        "elapsed_seconds": 51.50504913899931,
        "vllm_prefill_seconds_delta": 1.1254832400009036,
        "vllm_decode_seconds_delta": 50.367503489978844,
        "vllm_decode_tps": 39.708142381881636,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_generation_tokens_delta": 2000.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "server_e2e_seconds": 51.500343799591064
      },
      "fixed10K": {
        "completion_tokens": 10000,
        "elapsed_seconds": 361.96059344001696,
        "vllm_prefill_seconds_delta": 1.1244875820120797,
        "vllm_decode_seconds_delta": 360.82373836799525,
        "vllm_decode_tps": 27.71436282221888,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_generation_tokens_delta": 10000.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "server_e2e_seconds": 361.9555811882019
      }
    },
    "MTP_fixed2K_accounting": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 1091.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_draft_tokens_total": 1820.0,
        "vllm:spec_decode_num_drafts_total": 910.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1091.0
      },
      "accepted_per_position": {
        "0": 658.0,
        "1": 433.0
      },
      "accepted_fraction": 0.5994505494505494,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/vnext-v620-qwen38-rtn4-tp4-mtp2-service-r4-20261022/ladder1/c1_2000_proxy/metrics-before.txt": "edb17f0e277b98783f4f4c340c6ffe8b0dcb544ebbe326b7b8c6f53725d91d2f",
        "/home/ai/vnext-v620-qwen38-rtn4-tp4-mtp2-service-r4-20261022/ladder1/c1_2000_proxy/metrics-after.txt": "356fe17092373cc916f1ac374c0753eb42dfb7dff341ca2e59be20a99dfca3c6"
      }
    },
    "loaded_packed_bytes_per_rank": {
      "target": 3162718720,
      "draft": 55157760,
      "total": 3217876480
    },
    "maximum_sampled_VRAM_bytes_during_loaded_serving": {
      "0000:16:00.0": 30775201792,
      "0000:19:00.0": 30792024064,
      "0000:1c:00.0": 30792175616,
      "0000:1f:00.0": 30741901312
    },
    "VRAM_limit": "Periodic own-device observations; not a continuous peak. Includes runtime/KV and cannot be equated to packed weight storage.",
    "actual_KV": {
      "path": ".work/rtn4-tp-mtp-matrix/tp4-mtp2-service-r4/KV-observation",
      "review_path": ".work/helpers/projection-analyst/.work/native-mtp-review/TP4-r4-KV-observation-review.json",
      "review_SHA256": "85781eaf749586e0c67d931df073cc977768f8fa57e350068de5b6b3cc38804f",
      "target_views_per_rank": 16,
      "draft_views_per_rank": 1,
      "dtype": "torch.float16",
      "shape": [
        1636,
        2,
        800,
        1,
        256
      ],
      "logical_bytes_per_view": 1340211200,
      "allocated_bytes_by_rank": [
        27974586368,
        27974586368,
        27974586368,
        27974717440
      ],
      "reserved_bytes_by_rank": [
        28730982400,
        28730982400,
        28730982400,
        28730982400
      ],
      "runtime_KV_GiB": 21.27,
      "runtime_KV_tokens": 327200,
      "runtime_max_concurrency_at_131072": 9.46,
      "SSM_config_dtype": "float32",
      "limits": "Logical views may alias and are not physical allocation; hybrid cache estimates are not full resident-context proof. Actual completed native MTP counters and strict-valid inference are retained separately in this result."
    },
    "precision": {
      "target_parameter_dtype_counts": {
        "torch.float16": 611,
        "torch.int32": 768
      },
      "draft_parameter_dtype_counts": {
        "torch.int32": 15,
        "torch.float16": 14
      },
      "draft_class": "vllm.model_executor.models.qwen3_5_mtp.Qwen3_5MTP",
      "shared_embedding": true,
      "shared_lm_head": true,
      "exclusions": "LM head/embedding, norms, convolution, recurrent parameters retain FP16; Mamba SSM cache retains original float32",
      "attention_KV": "FP16",
      "SSM_configuration": "float32",
      "BF16_observed": false
    },
    "historical_TP4_off_comparison": {
      "official_fixed2K_TPS": 40.235022545409606,
      "fixed2K_delta_percent": -1.3095063210995517,
      "diagnostic_fixed10K_TPS": 38.55185454344116,
      "fixed10K_delta_percent": -28.111466619615744,
      "limits": "One sample per cell. Historical code-cache seed/request history differs from fresh-cache matrix. MTP adds head and requires800-block existing Triton fallback for both target/draft; historical TP4off uses784 native attention. Cache budget also differs; no isolated causal or repeatability claim."
    },
    "closure": {
      "container_id": "e80cfd8a3ef2b42799e6c77c83ca38b5f96ea60f0dbfec5435aee66e74a0e75e",
      "image_id": "sha256:762cae7dd32e7f3542a57754b547742471c371a40d287a115ac349616ec64681",
      "exit_code": 0,
      "OOMKilled": false,
      "restart_count": 0,
      "bank_idle": true,
      "devices": [
        {
          "pci": "0000:16:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        },
        {
          "pci": "0000:19:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        },
        {
          "pci": "0000:1c:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        },
        {
          "pci": "0000:1f:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        }
      ],
      "started_at": "2026-10-22T21:30:17.277864186Z",
      "finished_at": "2026-10-22T22:14:55.056663491Z",
      "closed_at": "2026-10-22T22:14:55.216256+00:00",
      "kernel_fault_matches": [],
      "startup_requests_completed_and_drained": 3,
      "official_phase_samples": 11,
      "complete_ladders": 1,
      "strict_valid": true,
      "official_fixed_2000_tps": 39.708142381881636,
      "qualifying_ladders": 0,
      "FP16_qualification": false,
      "baseline_kind": "RTN4"
    },
    "archive": {
      "path": "/home/ai/vnext-v620-qwen38-rtn4-tp4-mtp2-service-r4-20261022/raw-evidence.tar.gz",
      "bytes": 1148107,
      "sha256": "88dc5e837386b24d30eadde8175b601fa14b4b5231172aa0a3f47920ea43e977",
      "files": 248,
      "index_sha256": "e0768f050d92ebd38b6dc6be432f8642e0046985728fca0dd6a119e41a8b62a4"
    },
    "evidence": ".work/rtn4-tp-mtp-matrix/tp4-mtp2-service-r4/sealed-mirror/retained/evidence",
    "all_248_files_verified": true,
    "independent_review": {
      "path": ".work/helpers/projection-analyst/.work/native-mtp-review/TP4-MTP-sealed/review.json",
      "SHA256": "0cf11e9c9122a2559eb8a9e01a929123af46651dafec7711b10476e7eb879c05",
      "decision": "One strict-valid completed TP4 RTN4 native-MTP2 ladder; official2K observation, no repeatability or FP16 qualification.",
      "unchanged_CPU_auditors_all_passed": true,
      "binding_observation_correction": {
        "only_binding_field_changed": "maps_sha256",
        "explanation": "Complete process mapping text changes across execution; selected mapped library identities and worker starts are unchanged. Initial overly broad whole-binding-equality assertion failed on this field only; no evidence discrepancy."
      }
    },
    "FP16_qualification": false,
    "accounting": {
      "startup": 3,
      "warmups": 8,
      "strict": 1,
      "fixed": 2,
      "requests": 14,
      "prompt": 6622,
      "generated": 42141,
      "drained": true
    },
    "per_tier_MTP": {
      "strict": {
        "draft_batches": 3226.0,
        "proposed": 6452.0,
        "accepted": 4914.0,
        "accepted_per_position": {
          "0": 2688.0,
          "1": 2226.0
        },
        "committed": 8141.0,
        "request_tokens_agree": true,
        "accepted_fraction": 0.7616243025418475,
        "committed_per_reported_spec_batch": 2.523558586484811,
        "audit_sha256": "8338b2b0a35c39d728de6e8a2631c8b8c7ca5990d002a0a8b82e03e48db53dda"
      },
      "official2K": {
        "draft_batches": 910.0,
        "proposed": 1820.0,
        "accepted": 1091.0,
        "accepted_per_position": {
          "0": 658.0,
          "1": 433.0
        },
        "committed": 2000.0,
        "request_tokens_agree": true,
        "accepted_fraction": 0.5994505494505494,
        "committed_per_reported_spec_batch": 2.197802197802198,
        "audit_sha256": "59da4cc5801af8b8f73ea453308b2ad6333e52031ad5f675411f4c014d683c76"
      },
      "diagnostic10K": {
        "draft_batches": 4056.0,
        "proposed": 8112.0,
        "accepted": 5943.0,
        "accepted_per_position": {
          "0": 3293.0,
          "1": 2650.0
        },
        "committed": 10000.0,
        "request_tokens_agree": true,
        "accepted_fraction": 0.7326183431952663,
        "committed_per_reported_spec_batch": 2.465483234714004,
        "audit_sha256": "679f2a8b18c045c5ef6ae506b33a716eb60f084145da9fafd293fb8761fee7bb"
      }
    },
    "MTP_ratio_limits": "Committed tokens per reported speculative batch only; not all scheduler/GPU/decode steps. Acceptance precedes stop trimming. Proposed/rejected drafts never enter speed numerator."
  },
  "MTP_accounting_by_tier": {
    "strict": {
      "draft_batches": 3226.0,
      "proposed": 6452.0,
      "accepted": 4914.0,
      "accepted_per_position": {
        "0": 2688.0,
        "1": 2226.0
      },
      "committed": 8141.0,
      "request_tokens_agree": true,
      "accepted_fraction": 0.7616243025418475,
      "committed_per_reported_spec_batch": 2.523558586484811,
      "audit_sha256": "8338b2b0a35c39d728de6e8a2631c8b8c7ca5990d002a0a8b82e03e48db53dda"
    },
    "official2K": {
      "draft_batches": 910.0,
      "proposed": 1820.0,
      "accepted": 1091.0,
      "accepted_per_position": {
        "0": 658.0,
        "1": 433.0
      },
      "committed": 2000.0,
      "request_tokens_agree": true,
      "accepted_fraction": 0.5994505494505494,
      "committed_per_reported_spec_batch": 2.197802197802198,
      "audit_sha256": "59da4cc5801af8b8f73ea453308b2ad6333e52031ad5f675411f4c014d683c76"
    },
    "diagnostic10K": {
      "draft_batches": 4056.0,
      "proposed": 8112.0,
      "accepted": 5943.0,
      "accepted_per_position": {
        "0": 3293.0,
        "1": 2650.0
      },
      "committed": 10000.0,
      "request_tokens_agree": true,
      "accepted_fraction": 0.7326183431952663,
      "committed_per_reported_spec_batch": 2.465483234714004,
      "audit_sha256": "679f2a8b18c045c5ef6ae506b33a716eb60f084145da9fafd293fb8761fee7bb"
    }
  },
  "MTP_2K_draft_batches": 910.0,
  "MTP_2K_proposed": 1820.0,
  "MTP_2K_accepted": 1091.0,
  "MTP_2K_acceptance_percent": 59.94505494505494,
  "MTP_2K_committed": 2000.0,
  "detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
  "detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
  "detail_source_pointer": "local_RTN4_TP_MTP_matrix.cells[1]",
  "sampled_peak_VRAM_bytes_by_PCI": {
    "0000:16:00.0": 30775201792,
    "0000:19:00.0": 30792024064,
    "0000:1c:00.0": 30792175616,
    "0000:1f:00.0": 30741901312
  },
  "MTP_fixed2K_accounting": {
    "status": "observed",
    "actual_MTP_work_observed": true,
    "counter_deltas": {
      "vllm:spec_decode_num_accepted_tokens_total": 1091.0,
      "vllm:request_generation_tokens_count": 1.0,
      "vllm:generation_tokens_total": 2000.0,
      "vllm:request_generation_tokens_sum": 2000.0,
      "vllm:spec_decode_num_draft_tokens_total": 1820.0,
      "vllm:spec_decode_num_drafts_total": 910.0,
      "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1091.0
    },
    "accepted_per_position": {
      "0": 658.0,
      "1": 433.0
    },
    "accepted_fraction": 0.5994505494505494,
    "committed_emitted_tokens": 2000.0,
    "request_tokens_agree": true,
    "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
    "sources": {
      "/home/ai/vnext-v620-qwen38-rtn4-tp4-mtp2-service-r4-20261022/ladder1/c1_2000_proxy/metrics-before.txt": "edb17f0e277b98783f4f4c340c6ffe8b0dcb544ebbe326b7b8c6f53725d91d2f",
      "/home/ai/vnext-v620-qwen38-rtn4-tp4-mtp2-service-r4-20261022/ladder1/c1_2000_proxy/metrics-after.txt": "356fe17092373cc916f1ac374c0753eb42dfb7dff341ca2e59be20a99dfca3c6"
    }
  },
  "memory_accounting_limit": "Periodic own-device observations; not a continuous peak. Includes runtime/KV and cannot be equated to packed weight storage.",
  "memory_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#local_RTN4_TP_MTP_matrix",
  "memory_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
  "retest_strict_tokens": 12049,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 7368.0,
        "vllm:spec_decode_num_accepted_tokens_total": 7368.0,
        "vllm:spec_decode_num_draft_tokens_total": 9360.0,
        "vllm:generation_tokens_total": 12049.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:request_generation_tokens_sum": 12049.0,
        "vllm:spec_decode_num_drafts_total": 4680.0
      },
      "accepted_per_position": {
        "0": 3987.0,
        "1": 3381.0
      },
      "accepted_fraction": 0.7871794871794872,
      "committed_emitted_tokens": 12049.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "58b60fc09292dc78a04103d15cc41f6c3819f438f8ac6ee3f0c3138210f80a55",
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "4743b077a4308e60d98e2fed918ad8f39a0db6c484f585a10c15203b655627e8"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 1247.0,
        "vllm:request_generation_tokens_sum": 2000.0,
        "vllm:spec_decode_num_draft_tokens_total": 1506.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1247.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:spec_decode_num_drafts_total": 753.0,
        "vllm:request_generation_tokens_count": 1.0
      },
      "accepted_per_position": {
        "0": 657.0,
        "1": 590.0
      },
      "accepted_fraction": 0.8280212483399735,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "6305a1606c4c60bcf66a5e25d6687e417b165acded425de5114a1a3794b48b8c",
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "9cfd84f27f62d6ed382652c0b7f2037bdc495760cb18bda854876e88093e84e5"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:spec_decode_num_drafts_total": 3887.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6112.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6112.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_draft_tokens_total": 7774.0
      },
      "accepted_per_position": {
        "0": 3320.0,
        "1": 2792.0
      },
      "accepted_fraction": 0.7862104450733213,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "908c5b8ee8fecb4e947e3a3fd4ea2d996c36f0608e307d3d7320d069de9912c7",
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "a2ef83da43f40f2d0a0a94a6cf25ed6a3b46325cdc00aaf620f9aaf5fc322cc3"
      }
    }
  },
  "retest_MTP_2K_accepted": 1247.0,
  "retest_MTP_2K_proposed": 1506.0,
  "retest_MTP_2K_draft_batches": 753.0,
  "retest_MTP_2K_acceptance_percent": 82.80212483399735,
  "retest_logged_available_KV_GiB": 21.24,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 21.24,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 327200,
  "retest_memory_log_report": {
    "profile": "v620-w4-tp4-mtp2",
    "image": "sha256:762cae7dd32e7f3542a57754b547742471c371a40d287a115ac349616ec64681",
    "container": "cb24d05f08e6f9ae6d34cb9601d716a131da7448c1b3cb35342c550e9caedbdd",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 5.51,
    "retest_logged_model_load_seconds": 60.300757,
    "retest_logged_available_KV_GiB": 21.24,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 21.24,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 327200,
    "retest_logged_context_capacity_x": 9.46,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w4-tp4-mtp2/service-final.log",
    "source_log_sha256": "0807c48469b50f3cc209a1eaa5f6f67d3306b2fc2d612248ef1f5e64527c26a5",
    "sealed_index_sha256": "fe7ba6674f7a15c60fd9db595447d9b2e0bb7728107d4990e709736e59b41617",
    "source_lines": {
      "model_load": {
        "line": 154,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 15:21:33 [gpu_model_runner.py:4598] Model loading took 5.51 GiB memory and 60.300757 seconds"
      },
      "automatic_KV_budget": {
        "line": 223,
        "text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 15:38:24 [gpu_worker.py:453] Available KV cache memory: 21.24 GiB"
      },
      "grouped_KV_tokens": {
        "line": 228,
        "text": "(EngineCore_DP0 pid=85) INFO 10-26 15:38:24 [kv_cache_utils.py:1314] GPU KV cache size: 327,200 tokens"
      },
      "context_capacity": {
        "line": 229,
        "text": "(EngineCore_DP0 pid=85) INFO 10-26 15:38:24 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 9.46x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w4-tp4-mtp2/memory-log-report.json",
  "sampled_peak_VRAM_GiB": 28.677448272705078
}
V620Current Qwen3.84-bit RTNQwen/Qwen3.8-27BTP2MTP22K accepted 1104 / 1790 (61.68%) PASS7252 tokens 36.42725.228 0.774 / 55.69210K: 0.772 / 397.171Strict: 0.773 / 239.844 5.99 / 26.93packed target/draft projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:81115520e0ed7fc632670d872835ceefd1bea3a1ce7aada4ea7b6035eb04edba
Runtime image config: sha256:7f987d7773de6755f27160795f693fb1374eb42f54f2f322e917432c0886fd84
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256
mkdir -p launch-tools-v620-serve-r5
tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp2-mtp2
GPU_RENDER_NODES=renderD128,renderD129

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w4-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Preflight cleanup compatibility
After its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged.
Retained lab Docker command · not portable
docker run -d \
  --name serve-v620-w4-tp2-mtp2-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --cpuset-mems 0 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/card5:/dev/dri/card5:rwm \
  --device /dev/dri/card6:/dev/dri/card6:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
  -e ROCM_VERSION=7.14.0 \
  -e TORCH_VERSION=2.12.0+rocm7.14.0 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
  -e PYTORCH_ROCM_ARCH=gfx1030 \
  -e GPU_ARCHS=gfx1030 \
  -e AMDGPU_TARGETS=gfx1030 \
  -e CMAKE_HIP_ARCHITECTURES=gfx1030 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e VNEXT_GDN_HEAD32=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp2/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp2/cache/torch_extensions,dst=/root/.cache/torch_extensions \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp2/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
  --entrypoint /usr/bin/env \
  sha256:7f987d7773de6755f27160795f693fb1374eb42f54f2f322e917432c0886fd84 \
  V756F94_BOUNDARY_MODE=posted \
  DO_NOT_TRACK=1 \
  GPU_DEVICE_ORDINAL=0,1 \
  GPU_MAX_HW_QUEUES=1 \
  HSA_OVERRIDE_GFX_VERSION=10.3.0 \
  OMP_NUM_THREADS=4 \
  TEMP=/tmp/torchinductor_root \
  TMPDIR=/tmp/torchinductor_root \
  TMP=/tmp/torchinductor_root \
  TORCH_BLAS_PREFER_HIPBLASLT=0 \
  TRITON_CACHE_DIR=/root/.triton \
  VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  VLLM_GPU_MEMORY_UTILIZATION=0.95 \
  VLLM_LOGGING_LEVEL=INFO \
  VLLM_PORT=9002 \
  VLLM_RDNA_FORCE_FP16=1 \
  VLLM_WORKER_MULTIPROC_METHOD=spawn \
  NCCL_P2P_DISABLE=0 \
  NCCL_ALGO=Tree \
  NCCL_PROTO=LL \
  NCCL_MIN_NCHANNELS=4 \
  NCCL_MAX_NCHANNELS=4 \
  NCCL_NTHREADS=256 \
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
  HIP_VISIBLE_DEVICES=0,1 \
  CUDA_VISIBLE_DEVICES=0,1 \
  ROCR_VISIBLE_DEVICES=0,1 \
  VLLM_CACHE_ROOT=/root/.cache/vllm \
  PYTORCH_ROCM_ARCH=gfx1030 \
  PYTHONHASHSEED=0 \
  /opt/venv/bin/python \
  -m \
  vllm.vnext_rtn_entry \
  serve \
  /opt/local-models/qwen38-27b-hf \
  --served-model-name \
  Qwen/Qwen3.8-27B \
  --enable-auto-tool-choice \
  --tool-call-parser \
  qwen3_coder \
  --dtype \
  half \
  --host \
  0.0.0.0 \
  --port \
  9002 \
  --tensor-parallel-size \
  2 \
  --speculative-config \
  '{"method":"mtp","num_speculative_tokens":2}' \
  --max-model-len \
  131072 \
  --gpu-memory-utilization \
  0.95 \
  --trust-remote-code \
  --generation-config \
  vllm \
  -O=3 \
  --reasoning-parser \
  qwen3 \
  --async-scheduling \
  --hf-overrides \
  '{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":4,"group_size":128}}' \
  --language-model-only \
  --max-num-seqs \
  2 \
  --max-num-batched-tokens \
  2048 \
  --mamba-ssm-cache-dtype \
  float32 \
  --no-enable-prefix-caching \
  --no-enforce-eager \
  --attention-backend \
  TRITON_ATTN \
  --disable-custom-all-reduce \
  --seed \
  0 \
  --worker-cls \
  vllm.vnext_rtn_worker.Worker \
  --quantization \
  vnext_rtn
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-r4.tar.gz.sha256
mkdir -p launch-tools-v620-r4
tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp2-mtp2
GPU_RENDER_NODES=renderD128,renderD129

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-v620-r4/portable/launch.py --config v620-w4-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-v620-serve-r5.tar.gz · SHA-256
launch-tools-v620-r4.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
Portable observer compatibility
The evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged.
All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 37.875 · eligible10K 27.890 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 11944,
    "content_chars": 6434,
    "content_has_think_close": false,
    "elapsed_seconds": 479.41922588297166,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T04:58:26Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.022947221994400024,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 6434,
    "qwen_answer_text_sha256": "1e3200097878d38d8f28fe4b337f3fc70a55ddb199b09679ee5f03ae93f795f7",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 6434,
    "qwen_visible_text_sha256": "1e3200097878d38d8f28fe4b337f3fc70a55ddb199b09679ee5f03ae93f795f7",
    "raw_has_think_close": false,
    "reasoning_chars": 48599,
    "text_chars": 55034,
    "text_sha256": "a26575ceb56cd1dc8e86e4d77caf57f02012b0a5b23e81532fe1e46cb3ca1f9d",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 478.62756001902744,
    "vllm_decode_tps": 24.954685015474613,
    "vllm_generation_tokens_delta": 11944.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.7719826169777662,
    "vllm_prefill_tps": 612.7080967855819,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 24.91347729745737
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 53.60688337893225,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T04:59:20Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.01787939900532365,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 10224,
    "text_chars": 10224,
    "text_sha256": "2bd04ea6e797c0d8b618e371dc199f3ed0c4e852ee8a94d627773b5e456d0365",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 52.80462608905509,
    "vllm_decode_tps": 37.87546940730149,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.770544630009681,
    "vllm_prefill_tps": 613.8515299159989,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 37.30864161347625
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 26224,
    "content_has_think_close": true,
    "elapsed_seconds": 359.33965503191575,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T05:05:20Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.011819053906947374,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 19549,
    "qwen_answer_text_sha256": "bd3e61386f31470e92406a3955ba35684361276a8c308bc35322506677832345",
    "qwen_gate_close_in_content": true,
    "qwen_gate_close_in_raw": true,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 19549,
    "qwen_visible_text_sha256": "bd3e61386f31470e92406a3955ba35684361276a8c308bc35322506677832345",
    "raw_has_think_close": true,
    "reasoning_chars": 23179,
    "text_chars": 49404,
    "text_sha256": "86255e1edfe19b984dae99b7aed2fa59484e03d7a4787afe609f6e0bdc7137a1",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 358.54745610407554,
    "vllm_decode_tps": 27.89031083544294,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.7714070088695735,
    "vllm_prefill_tps": 613.1652870164329,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 27.828823955184745
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestMTP counter replayPASS8,588 natural tokens2K 35.84710K 27.756 · diagnostic2K MTP accepted 1089 / 1820 (59.84%)
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T17:01:27.927094303Z",
      "FinishedAt": "2026-10-26T17:46:22.725664239Z"
    },
    "devices_after": [
      {
        "card": 5,
        "render": 133,
        "pci": "0000:16:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:19:00.0",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T17:46:22.864001+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 35.84675810076804,
  "retest_10K_TPS": 27.756224855608366,
  "retest_strict_tokens": 8588,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 8588,
      "elapsed_seconds": 300.9188194599701,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 300.1355729380157,
      "vllm_decode_tps": 28.613735839215575,
      "vllm_generation_tokens_delta": 8588.0,
      "vllm_prefill_seconds_delta": 0.7710430989973247,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 56.57390893495176,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 55.79305091907736,
      "vllm_decode_tps": 35.84675810076804,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 0.7696576459566131,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 361.0622771610506,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 360.2795427700039,
      "vllm_decode_tps": 27.756224855608366,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 0.7706899599870667,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 0.7696576459566131,
  "retest_2K_total_seconds": 56.57390893495176,
  "retest_10K_prefill_seconds": 0.7706899599870667,
  "retest_10K_total_seconds": 361.0622771610506,
  "retest_strict_prefill_seconds": 0.7710430989973247,
  "retest_strict_total_seconds": 300.9188194599701,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:generation_tokens_total": 8588.0,
        "vllm:spec_decode_num_draft_tokens_total": 6822.0,
        "vllm:request_generation_tokens_sum": 8588.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 5177.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_drafts_total": 3411.0,
        "vllm:spec_decode_num_accepted_tokens_total": 5177.0
      },
      "accepted_per_position": {
        "0": 2845.0,
        "1": 2332.0
      },
      "accepted_fraction": 0.7588683670477866,
      "committed_emitted_tokens": 8588.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "7bb29d1e750c70d3e30f302f68bce4a3031519b03fbaa7f796ac3ef15f050166",
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "e4a731472fb2f1212b357ecc4d1bee15736d1a42fd7dc0c79956048e43d3c4a2"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 1089.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 1820.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1089.0,
        "vllm:spec_decode_num_drafts_total": 910.0,
        "vllm:request_generation_tokens_sum": 2000.0
      },
      "accepted_per_position": {
        "0": 653.0,
        "1": 436.0
      },
      "accepted_fraction": 0.5983516483516483,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "71672c9bc8139659a7c942a3d660e713cc869f85dbaa51d989cd9c7a54d0c06e",
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "5f853b7052353c1649940ce9512aab00fab221e37ee9e1b7accfe0196947d95b"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 7686.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6157.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_drafts_total": 3843.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6157.0,
        "vllm:generation_tokens_total": 10000.0
      },
      "accepted_per_position": {
        "0": 3321.0,
        "1": 2836.0
      },
      "accepted_fraction": 0.8010668748373666,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "22a11ba0d334e8412b04fd0cc21e3c4ad023fd1dcce9a8129d9122ecc9d6d707",
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "08899f1240a16630dd09114338fbfe2ddb3bae95b4a9cf14f0af8266e0d86e9b"
      }
    }
  },
  "retest_MTP_2K_accepted": 1089.0,
  "retest_MTP_2K_proposed": 1820.0,
  "retest_MTP_2K_draft_batches": 910.0,
  "retest_MTP_2K_acceptance_percent": 59.83516483516483,
  "retest_loaded_admission": {
    "all_two_quantized_module_receipts_passed": true,
    "all_two_MTP_draft_receipts_passed": true,
    "draft_sharing_observations": [
      {
        "rank": 1,
        "shared_embedding": true,
        "shared_lm_head": true
      },
      {
        "rank": 0,
        "shared_embedding": true,
        "shared_lm_head": true
      }
    ],
    "MTP_execution_evidence": "See separately retained standard speculative counter deltas; load/source observations alone are not invocation proof",
    "generated_quantized_call_support_complete": true,
    "graphs": [
      {
        "rank": 0,
        "pid": 102,
        "source_sha256": "02e19c8d98c52bfc070e61d7ee0084e9719dfedcc2f7574218332c129643cc0e",
        "generated_gemm_calls": 256
      },
      {
        "rank": 1,
        "pid": 103,
        "source_sha256": "35d844a586d8d59fdeb96ad2d03968cc12dcaa3ce039f08b2bb812afecb08632",
        "generated_gemm_calls": 256
      }
    ],
    "attention_KV_observations": [
      {
        "rank": 1,
        "pid": 103,
        "attention_layers": [
          3,
          7,
          11,
          15,
          19,
          23,
          27,
          31,
          35,
          39,
          43,
          47,
          51,
          55,
          59,
          63
        ],
        "logical_view_bytes": 15335424000,
        "mtp_draft_attention_KV": [
          {
            "name": "model.layers.0.self_attn.attn",
            "tensors": [
              {
                "shape": [
                  585,
                  2,
                  800,
                  2,
                  256
                ],
                "dtype": "torch.float16",
                "device": "cuda:1",
                "bytes": 958464000
              }
            ]
          }
        ],
        "Mamba_SSM_config_dtype": "float32",
        "limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
      },
      {
        "rank": 0,
        "pid": 102,
        "attention_layers": [
          3,
          7,
          11,
          15,
          19,
          23,
          27,
          31,
          35,
          39,
          43,
          47,
          51,
          55,
          59,
          63
        ],
        "logical_view_bytes": 15335424000,
        "mtp_draft_attention_KV": [
          {
            "name": "model.layers.0.self_attn.attn",
            "tensors": [
              {
                "shape": [
                  585,
                  2,
                  800,
                  2,
                  256
                ],
                "dtype": "torch.float16",
                "device": "cuda:0",
                "bytes": 958464000
              }
            ]
          }
        ],
        "Mamba_SSM_config_dtype": "float32",
        "limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
      }
    ],
    "all_two_KV_receipts_passed": true,
    "limits": "Static generated call support plus bound loaded-module receipts/native mappings and completed startup. Not a GPU completion trace or correctness verdict."
  },
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30544822272,
    "1": 30544805888
  },
  "retest_memory_observation_count": 45,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-w4-tp2-mtp2/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 10.97,
  "retest_logged_model_load_seconds": 106.4305,
  "retest_logged_available_KV_GiB": 15.19,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 15.19,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 116800,
  "retest_logged_context_capacity_x": 3.38,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "v620-w4-tp2-mtp2",
    "image": "sha256:7f987d7773de6755f27160795f693fb1374eb42f54f2f322e917432c0886fd84",
    "container": "70e39d1b172d119241eb03dbd6619fcb0c16edb72e4d6c821d31d0d3628f146b",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 10.97,
    "retest_logged_model_load_seconds": 106.4305,
    "retest_logged_available_KV_GiB": 15.19,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 15.19,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 116800,
    "retest_logged_context_capacity_x": 3.38,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w4-tp2-mtp2/service-final.log",
    "source_log_sha256": "38a702105b6b0ffe6eca92cfc94fe05a20949d83e51f76d0326d098ca29c1e05",
    "sealed_index_sha256": "89aaf41caccca05a57040a8728c276f5acab0ccd3fc718e6700e103b8e71b36e",
    "source_lines": {
      "model_load": {
        "line": 118,
        "text": "(Worker pid=102) (Worker_TP0 pid=102) INFO 10-26 17:05:15 [gpu_model_runner.py:4598] Model loading took 10.97 GiB memory and 106.430500 seconds"
      },
      "automatic_KV_budget": {
        "line": 168,
        "text": "(Worker pid=102) (Worker_TP0 pid=102) INFO 10-26 17:21:47 [gpu_worker.py:453] Available KV cache memory: 15.19 GiB"
      },
      "grouped_KV_tokens": {
        "line": 172,
        "text": "(EngineCore_DP0 pid=84) INFO 10-26 17:21:47 [kv_cache_utils.py:1314] GPU KV cache size: 116,800 tokens"
      },
      "context_capacity": {
        "line": 173,
        "text": "(EngineCore_DP0 pid=84) INFO 10-26 17:21:47 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 3.38x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w4-tp2-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.44708251953125
}
Retest KV budget: 15.190 GiB (automatic, rounded)116,800 grouped KV tokens · 3.38× reported capacityRetest sampled VRAM: 28.447 GiB max/cardRetest memory reports and limits
Evidence and limits

completed matrix cell

{
  "strict_tokens": 7252,
  "source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.local_RTN4_TP_MTP_matrix.ordered_cells[TP=2,MTP_tokens=2].result",
  "source_revision": "9463306390eb4f2e48cc2b9a2c4f03b52ed2b279",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "recorded_configuration_details": {
    "strict_valid": true,
    "strict_tokens": 7252,
    "strict_diagnostic_TPS": 30.335624585208077,
    "official_fixed2K_TPS": 36.42672258849918,
    "diagnostic_fixed10K_TPS": 25.227890996166266,
    "source": "9463306390eb4f2e48cc2b9a2c4f03b52ed2b279",
    "image": "sha256:7f987d7773de6755f27160795f693fb1374eb42f54f2f322e917432c0886fd84",
    "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "TP": 2,
    "MTP_tokens": 2,
    "sample_count_per_tier": 1,
    "metrics": {
      "strict": {
        "completion_tokens": 7252,
        "elapsed_seconds": 239.84428551400197,
        "vllm_prefill_seconds_delta": 0.7732716030150186,
        "vllm_decode_seconds_delta": 239.0588655799802,
        "vllm_decode_tps": 30.335624585208077,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_generation_tokens_delta": 7252.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0
      },
      "fixed2K": {
        "completion_tokens": 2000,
        "elapsed_seconds": 55.69153628099593,
        "vllm_prefill_seconds_delta": 0.774255597993033,
        "vllm_decode_seconds_delta": 54.904747335996944,
        "vllm_decode_tps": 36.42672258849918,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_generation_tokens_delta": 2000.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "server_e2e_seconds": 55.68696188926697
      },
      "fixed10K": {
        "completion_tokens": 10000,
        "elapsed_seconds": 397.1708856849873,
        "vllm_prefill_seconds_delta": 0.772096765023889,
        "vllm_decode_seconds_delta": 396.386681768985,
        "vllm_decode_tps": 25.227890996166266,
        "vllm_prompt_tokens_delta": 473.0,
        "vllm_generation_tokens_delta": 10000.0,
        "vllm_prefill_request_count_delta": 1.0,
        "vllm_decode_request_count_delta": 1.0,
        "server_e2e_seconds": 397.16564750671387
      }
    },
    "MTP_fixed2K_accounting": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1104.0,
        "vllm:spec_decode_num_draft_tokens_total": 1790.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_drafts_total": 895.0,
        "vllm:spec_decode_num_accepted_tokens_total": 1104.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_generation_tokens_sum": 2000.0
      },
      "accepted_per_position": {
        "0": 663.0,
        "1": 441.0
      },
      "accepted_fraction": 0.6167597765363129,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/vnext-v620-qwen38-rtn4-tp2-mtp2-service-r2-20261022/ladder1/c1_2000_proxy/metrics-before.txt": "03a95fac30c1499bd5a55e705fc31f7ea7e30dbc77cc4488c26bcf0eb33bd326",
        "/home/ai/vnext-v620-qwen38-rtn4-tp2-mtp2-service-r2-20261022/ladder1/c1_2000_proxy/metrics-after.txt": "ca4f7307de7e6e6661190706fdfea725617d84290c6ea1b6098a146b78d92499"
      }
    },
    "loaded_packed_bytes_per_rank": {
      "target": 6325437440,
      "draft": 110315520,
      "total": 6435752960
    },
    "maximum_sampled_VRAM_bytes_during_loaded_serving": {
      "0000:16:00.0": 28912955392,
      "0000:19:00.0": 28915073024
    },
    "VRAM_limit": "Periodic own-device observations; not a continuous peak measurement. Includes runtime/KV and cannot be equated to packed weights.",
    "actual_KV": {
      "evidence": ".work/rtn4-tp-mtp-matrix/tp2-mtp2-service-r2/kv-audit-observation",
      "dtype": "FP16",
      "shape_per_target_and_draft_view": [
        585,
        2,
        800,
        2,
        256
      ],
      "target_views_per_rank": 16,
      "draft_views_per_rank": 1,
      "available_KV_GiB": 15.19,
      "runtime_reported_tokens": 116800,
      "runtime_reported_maximum_concurrency_at_131072": 3.38,
      "configured_context": 131072,
      "HTTP_ready_utc": "2026-10-22T21:05:05.112523Z",
      "independent_review": {
        "path": ".work/helpers/projection-analyst/.work/native-mtp-review/TP2-r2-KV-observation-addendum.json",
        "SHA256": "e96f3a09681c1fe2b5d2e7afc43e5ccf4d3e963216fdcad9da1550e999945921"
      },
      "limits": "Distinct hybrid runtime estimates; not full resident context proof. Logical views may alias. Cache budget differs from TP2off1.05GiB/7840/0.25x; The original KV observation preceded inference; the completed strict-valid ladder is retained separately in this result."
    },
    "TP2_off_comparison": {
      "official_fixed2K_TPS": 32.806665219045115,
      "fixed2K_delta_percent": 11.034517971526503,
      "diagnostic_fixed10K_TPS": 30.822545818435916,
      "fixed10K_delta_percent": -18.151176918433887,
      "limits": "One sample per cell; changed MTP/cache budget and different completion length. No repeatability or isolated causal claim."
    },
    "closure": {
      "container_id": "b413eb7b3441ca6e3d817ee3b55419667627355373c0160876dff95dbff3d4c9",
      "image_id": "sha256:7f987d7773de6755f27160795f693fb1374eb42f54f2f322e917432c0886fd84",
      "exit_code": 0,
      "OOMKilled": false,
      "restart_count": 0,
      "bank_idle": true,
      "devices": [
        {
          "pci": "0000:16:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        },
        {
          "pci": "0000:19:00.0",
          "vram_used_bytes": 21401600,
          "busy_percent": 0,
          "dpm": "auto"
        }
      ],
      "started_at": "2026-10-22T20:41:17.418699404Z",
      "finished_at": "2026-10-22T21:29:41.181240206Z",
      "closed_at": "2026-10-22T21:29:41.328370+00:00",
      "kernel_fault_matches": [],
      "startup_requests_completed_and_drained": 3,
      "official_phase_samples": 11,
      "complete_ladders": 1,
      "strict_valid": true,
      "official_fixed_2000_tps": 36.42672258849918,
      "qualifying_ladders": 0,
      "FP16_qualification": false,
      "baseline_kind": "RTN4"
    },
    "archive": {
      "path": "/home/ai/vnext-v620-qwen38-rtn4-tp2-mtp2-service-r2-20261022/raw-evidence.tar.gz",
      "bytes": 787728,
      "sha256": "fde54375455d96f4963ab3e33dd01a55d58fc2125e7a03b347140d5988ef05ee",
      "files": 240,
      "index_sha256": "8a576a5a3498120a15bf429ea77a59a394774f2662f6f446fda006b4f7e44d7e"
    },
    "evidence": ".work/rtn4-tp-mtp-matrix/tp2-mtp2-service-r2/sealed-mirror/retained/evidence",
    "all_240_files_verified": true,
    "independent_review": {
      "path": ".work/helpers/projection-analyst/.work/native-mtp-review/TP2-MTP-sealed/review.json",
      "SHA256": "6ac5f5145a81338f728c4df684b8686a73efed2785624497b76a7ca5c7e1da2c",
      "decision": "One strict-valid completed TP2 RTN4 native-MTP2 ladder; admitted official2K result, no repeatability or FP16 release qualification"
    },
    "FP16_qualification": false,
    "per_tier_MTP": {
      "strict": {
        "draft_batches": 2896.0,
        "proposed": 5792.0,
        "accepted": 4355.0,
        "accepted_per_position": {
          "0": 2406.0,
          "1": 1949.0
        },
        "committed": 7252.0,
        "request_tokens_agree": true,
        "accepted_fraction": 0.7518991712707183,
        "committed_per_reported_spec_batch": 2.50414364640884,
        "audit_sha256": "8904ecfb2e41db11ecb76b952e5913e932ea8fa9f479366b94f473353043cf44"
      },
      "official2K": {
        "draft_batches": 895.0,
        "proposed": 1790.0,
        "accepted": 1104.0,
        "accepted_per_position": {
          "0": 663.0,
          "1": 441.0
        },
        "committed": 2000.0,
        "request_tokens_agree": true,
        "accepted_fraction": 0.6167597765363129,
        "committed_per_reported_spec_batch": 2.2346368715083798,
        "audit_sha256": "91ce1c75f9f2618df4f28a82be243d8cc33243d19587853e7f4b535e1e7e0f95"
      },
      "diagnostic10K": {
        "draft_batches": 4157.0,
        "proposed": 8314.0,
        "accepted": 5843.0,
        "accepted_per_position": {
          "0": 3318.0,
          "1": 2525.0
        },
        "committed": 10000.0,
        "request_tokens_agree": true,
        "accepted_fraction": 0.7027904738994467,
        "committed_per_reported_spec_batch": 2.4055809477988936,
        "audit_sha256": "71b2602c5fa4ca02be8a008f0297a9089cd1c7c10fb01ba5ef3b5ace1a1a2ace"
      }
    },
    "MTP_ratio_limits": "2000/895=committed tokens per recorded speculative verification batch only, not all scheduler/GPU steps. Accepted1104 precedes stop trimming; proposals1790 never enter speed numerator.",
    "accounting": {
      "startup": 3,
      "warmups": 8,
      "strict": 1,
      "fixed": 2,
      "requests": 14,
      "prompt": 6622,
      "generated": 41252,
      "drained": true
    }
  },
  "MTP_accounting_by_tier": {
    "strict": {
      "draft_batches": 2896.0,
      "proposed": 5792.0,
      "accepted": 4355.0,
      "accepted_per_position": {
        "0": 2406.0,
        "1": 1949.0
      },
      "committed": 7252.0,
      "request_tokens_agree": true,
      "accepted_fraction": 0.7518991712707183,
      "committed_per_reported_spec_batch": 2.50414364640884,
      "audit_sha256": "8904ecfb2e41db11ecb76b952e5913e932ea8fa9f479366b94f473353043cf44"
    },
    "official2K": {
      "draft_batches": 895.0,
      "proposed": 1790.0,
      "accepted": 1104.0,
      "accepted_per_position": {
        "0": 663.0,
        "1": 441.0
      },
      "committed": 2000.0,
      "request_tokens_agree": true,
      "accepted_fraction": 0.6167597765363129,
      "committed_per_reported_spec_batch": 2.2346368715083798,
      "audit_sha256": "91ce1c75f9f2618df4f28a82be243d8cc33243d19587853e7f4b535e1e7e0f95"
    },
    "diagnostic10K": {
      "draft_batches": 4157.0,
      "proposed": 8314.0,
      "accepted": 5843.0,
      "accepted_per_position": {
        "0": 3318.0,
        "1": 2525.0
      },
      "committed": 10000.0,
      "request_tokens_agree": true,
      "accepted_fraction": 0.7027904738994467,
      "committed_per_reported_spec_batch": 2.4055809477988936,
      "audit_sha256": "71b2602c5fa4ca02be8a008f0297a9089cd1c7c10fb01ba5ef3b5ace1a1a2ace"
    }
  },
  "MTP_2K_draft_batches": 895.0,
  "MTP_2K_proposed": 1790.0,
  "MTP_2K_accepted": 1104.0,
  "MTP_2K_acceptance_percent": 61.675977653631286,
  "MTP_2K_committed": 2000.0,
  "detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
  "detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
  "detail_source_pointer": "local_RTN4_TP_MTP_matrix.cells[2]",
  "sampled_peak_VRAM_bytes_by_PCI": {
    "0000:16:00.0": 28912955392,
    "0000:19:00.0": 28915073024
  },
  "MTP_fixed2K_accounting": {
    "status": "observed",
    "actual_MTP_work_observed": true,
    "counter_deltas": {
      "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1104.0,
      "vllm:spec_decode_num_draft_tokens_total": 1790.0,
      "vllm:request_generation_tokens_count": 1.0,
      "vllm:spec_decode_num_drafts_total": 895.0,
      "vllm:spec_decode_num_accepted_tokens_total": 1104.0,
      "vllm:generation_tokens_total": 2000.0,
      "vllm:request_generation_tokens_sum": 2000.0
    },
    "accepted_per_position": {
      "0": 663.0,
      "1": 441.0
    },
    "accepted_fraction": 0.6167597765363129,
    "committed_emitted_tokens": 2000.0,
    "request_tokens_agree": true,
    "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
    "sources": {
      "/home/ai/vnext-v620-qwen38-rtn4-tp2-mtp2-service-r2-20261022/ladder1/c1_2000_proxy/metrics-before.txt": "03a95fac30c1499bd5a55e705fc31f7ea7e30dbc77cc4488c26bcf0eb33bd326",
      "/home/ai/vnext-v620-qwen38-rtn4-tp2-mtp2-service-r2-20261022/ladder1/c1_2000_proxy/metrics-after.txt": "ca4f7307de7e6e6661190706fdfea725617d84290c6ea1b6098a146b78d92499"
    }
  },
  "memory_accounting_limit": "Periodic own-device observations; not a continuous peak measurement. Includes runtime/KV and cannot be equated to packed weights.",
  "memory_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#local_RTN4_TP_MTP_matrix",
  "memory_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
  "retest_strict_tokens": 8588,
  "retest_MTP_accounting_by_tier": {
    "strict": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:generation_tokens_total": 8588.0,
        "vllm:spec_decode_num_draft_tokens_total": 6822.0,
        "vllm:request_generation_tokens_sum": 8588.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 5177.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_drafts_total": 3411.0,
        "vllm:spec_decode_num_accepted_tokens_total": 5177.0
      },
      "accepted_per_position": {
        "0": 2845.0,
        "1": 2332.0
      },
      "accepted_fraction": 0.7588683670477866,
      "committed_emitted_tokens": 8588.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "7bb29d1e750c70d3e30f302f68bce4a3031519b03fbaa7f796ac3ef15f050166",
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "e4a731472fb2f1212b357ecc4d1bee15736d1a42fd7dc0c79956048e43d3c4a2"
      }
    },
    "2K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_accepted_tokens_total": 1089.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_draft_tokens_total": 1820.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 1089.0,
        "vllm:spec_decode_num_drafts_total": 910.0,
        "vllm:request_generation_tokens_sum": 2000.0
      },
      "accepted_per_position": {
        "0": 653.0,
        "1": 436.0
      },
      "accepted_fraction": 0.5983516483516483,
      "committed_emitted_tokens": 2000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "71672c9bc8139659a7c942a3d660e713cc869f85dbaa51d989cd9c7a54d0c06e",
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "5f853b7052353c1649940ce9512aab00fab221e37ee9e1b7accfe0196947d95b"
      }
    },
    "10K": {
      "status": "observed",
      "actual_MTP_work_observed": true,
      "counter_deltas": {
        "vllm:spec_decode_num_draft_tokens_total": 7686.0,
        "vllm:request_generation_tokens_count": 1.0,
        "vllm:spec_decode_num_accepted_tokens_per_pos_total": 6157.0,
        "vllm:request_generation_tokens_sum": 10000.0,
        "vllm:spec_decode_num_drafts_total": 3843.0,
        "vllm:spec_decode_num_accepted_tokens_total": 6157.0,
        "vllm:generation_tokens_total": 10000.0
      },
      "accepted_per_position": {
        "0": 3321.0,
        "1": 2836.0
      },
      "accepted_fraction": 0.8010668748373666,
      "committed_emitted_tokens": 10000.0,
      "request_tokens_agree": true,
      "limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
      "sources": {
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "22a11ba0d334e8412b04fd0cc21e3c4ad023fd1dcce9a8129d9122ecc9d6d707",
        "/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "08899f1240a16630dd09114338fbfe2ddb3bae95b4a9cf14f0af8266e0d86e9b"
      }
    }
  },
  "retest_MTP_2K_accepted": 1089.0,
  "retest_MTP_2K_proposed": 1820.0,
  "retest_MTP_2K_draft_batches": 910.0,
  "retest_MTP_2K_acceptance_percent": 59.83516483516483,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30544822272,
    "1": 30544805888
  },
  "retest_memory_observation_count": 45,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-w4-tp2-mtp2/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 15.19,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 15.19,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 116800,
  "retest_memory_log_report": {
    "profile": "v620-w4-tp2-mtp2",
    "image": "sha256:7f987d7773de6755f27160795f693fb1374eb42f54f2f322e917432c0886fd84",
    "container": "70e39d1b172d119241eb03dbd6619fcb0c16edb72e4d6c821d31d0d3628f146b",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 10.97,
    "retest_logged_model_load_seconds": 106.4305,
    "retest_logged_available_KV_GiB": 15.19,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 15.19,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 116800,
    "retest_logged_context_capacity_x": 3.38,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w4-tp2-mtp2/service-final.log",
    "source_log_sha256": "38a702105b6b0ffe6eca92cfc94fe05a20949d83e51f76d0326d098ca29c1e05",
    "sealed_index_sha256": "89aaf41caccca05a57040a8728c276f5acab0ccd3fc718e6700e103b8e71b36e",
    "source_lines": {
      "model_load": {
        "line": 118,
        "text": "(Worker pid=102) (Worker_TP0 pid=102) INFO 10-26 17:05:15 [gpu_model_runner.py:4598] Model loading took 10.97 GiB memory and 106.430500 seconds"
      },
      "automatic_KV_budget": {
        "line": 168,
        "text": "(Worker pid=102) (Worker_TP0 pid=102) INFO 10-26 17:21:47 [gpu_worker.py:453] Available KV cache memory: 15.19 GiB"
      },
      "grouped_KV_tokens": {
        "line": 172,
        "text": "(EngineCore_DP0 pid=84) INFO 10-26 17:21:47 [kv_cache_utils.py:1314] GPU KV cache size: 116,800 tokens"
      },
      "context_capacity": {
        "line": 173,
        "text": "(EngineCore_DP0 pid=84) INFO 10-26 17:21:47 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 3.38x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w4-tp2-mtp2/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.44708251953125,
  "sampled_peak_VRAM_GiB": 26.92926025390625
}
V620Additional candidate4-bit RTN X (unpromoted)Qwen/Qwen3.8-27BTP4MTP off PASS6851 tokens 41.17639.369 1.100 / 49.68410K: 1.098 / 255.115Strict: — / — 2.95 / 21.04packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:fa94293d3aa1bc4db60d2a27d6dafbefdd5c0709e736315b64faf66bc2e4da4d
Runtime image config: sha256:23efb9f20a02e5b7bd69ed0004a7a1976d35ba728e4fd52f644b4645eadd5ed1
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256
mkdir -p launch-tools-v620-serve-r5
tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp4-mtp0-x
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w4-tp4-mtp0-x --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Preflight cleanup compatibility
After its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged.
Retained lab Docker command · not portable
docker run -d \
  --name serve-v620-w4-tp4-mtp0-x-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-7,16-23 \
  --cpuset-mems 0 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/card5:/dev/dri/card5:rwm \
  --device /dev/dri/card6:/dev/dri/card6:rwm \
  --device /dev/dri/card7:/dev/dri/card7:rwm \
  --device /dev/dri/card8:/dev/dri/card8:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
  --device /dev/dri/renderD136:/dev/dri/renderD136:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e VNEXT_RTN_W4_ORDERED=1 \
  -e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
  -e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
  -e ROCM_VERSION=7.14.0 \
  -e TORCH_VERSION=2.12.0+rocm7.14.0 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
  -e PYTORCH_ROCM_ARCH=gfx1030 \
  -e GPU_ARCHS=gfx1030 \
  -e AMDGPU_TARGETS=gfx1030 \
  -e CMAKE_HIP_ARCHITECTURES=gfx1030 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e VNEXT_GDN_HEAD32=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0-x/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0-x/cache/torch_extensions,dst=/root/.cache/torch_extensions \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0-x/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0-x/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
  --entrypoint /usr/bin/env \
  sha256:23efb9f20a02e5b7bd69ed0004a7a1976d35ba728e4fd52f644b4645eadd5ed1 \
  V756F94_BOUNDARY_MODE=posted \
  DO_NOT_TRACK=1 \
  GPU_DEVICE_ORDINAL=0,1,2,3 \
  GPU_MAX_HW_QUEUES=1 \
  HSA_OVERRIDE_GFX_VERSION=10.3.0 \
  OMP_NUM_THREADS=4 \
  TEMP=/tmp/torchinductor_root \
  TMPDIR=/tmp/torchinductor_root \
  TMP=/tmp/torchinductor_root \
  TORCH_BLAS_PREFER_HIPBLASLT=0 \
  TRITON_CACHE_DIR=/root/.triton \
  VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  VLLM_GPU_MEMORY_UTILIZATION=0.95 \
  VLLM_LOGGING_LEVEL=INFO \
  VLLM_PORT=9002 \
  VLLM_RDNA_FORCE_FP16=1 \
  VLLM_WORKER_MULTIPROC_METHOD=spawn \
  NCCL_P2P_DISABLE=0 \
  NCCL_ALGO=Tree \
  NCCL_PROTO=LL \
  NCCL_MIN_NCHANNELS=4 \
  NCCL_MAX_NCHANNELS=4 \
  NCCL_NTHREADS=256 \
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
  HIP_VISIBLE_DEVICES=0,1,2,3 \
  CUDA_VISIBLE_DEVICES=0,1,2,3 \
  ROCR_VISIBLE_DEVICES=0,1,2,3 \
  VLLM_CACHE_ROOT=/root/.cache/vllm \
  PYTORCH_ROCM_ARCH=gfx1030 \
  PYTHONHASHSEED=0 \
  /opt/venv/bin/python \
  -m \
  vllm.vnext_rtn_entry \
  serve \
  /opt/local-models/qwen38-27b-hf \
  --served-model-name \
  Qwen/Qwen3.8-27B \
  --enable-auto-tool-choice \
  --tool-call-parser \
  qwen3_coder \
  --dtype \
  half \
  --host \
  0.0.0.0 \
  --port \
  9002 \
  --tensor-parallel-size \
  4 \
  --max-model-len \
  131072 \
  --gpu-memory-utilization \
  0.95 \
  --trust-remote-code \
  --generation-config \
  vllm \
  -O=3 \
  --reasoning-parser \
  qwen3 \
  --async-scheduling \
  --hf-overrides \
  '{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":4,"group_size":128}}' \
  --language-model-only \
  --max-num-seqs \
  2 \
  --max-num-batched-tokens \
  2048 \
  --mamba-ssm-cache-dtype \
  float32 \
  --no-enable-prefix-caching \
  --no-enforce-eager \
  --attention-backend \
  TRITON_ATTN \
  --disable-custom-all-reduce \
  --seed \
  0 \
  --worker-cls \
  vllm.vnext_rtn_worker.Worker \
  --quantization \
  vnext_rtn
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-v620-r4.tar.gz.sha256
mkdir -p launch-tools-v620-r4
tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp4-mtp0-x
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-v620-r4/portable/launch.py --config v620-w4-tp4-mtp0-x --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-v620-serve-r5.tar.gz · SHA-256
launch-tools-v620-r4.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
Portable observer compatibility
The evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged.
All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 41.539 · eligible10K 39.761 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 8394,
    "content_chars": 5030,
    "content_has_think_close": false,
    "elapsed_seconds": 210.26249684090726,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T06:14:00Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.013499699998646975,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 5030,
    "qwen_answer_text_sha256": "a9a4d27533a8a0819fb3b1b995314fa366cdb37d74a8f623b76570a16020cb95",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 5030,
    "qwen_visible_text_sha256": "a9a4d27533a8a0819fb3b1b995314fa366cdb37d74a8f623b76570a16020cb95",
    "raw_has_think_close": false,
    "reasoning_chars": 30971,
    "text_chars": 36002,
    "text_sha256": "e50e5c9b3705f244b624038ee60d85bce937e8071f04fa10daef3d8edc680da8",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 209.1481526561547,
    "vllm_decode_tps": 40.134229699843274,
    "vllm_generation_tokens_delta": 8394.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.0931877410039306,
    "vllm_prefill_tps": 432.6795684386469,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 39.92152726290141
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 49.2615391809959,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T06:14:50Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.01474340702407062,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 8176,
    "text_chars": 8176,
    "text_sha256": "a547216ac88e1666e05daa4fe0dde87d974517b3902aaefa2b345516b1df762d",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 48.14751691184938,
    "vllm_decode_tps": 41.53900612697617,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.0958944531157613,
    "vllm_prefill_tps": 431.6109080169203,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 40.59962464127713
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 13264,
    "content_has_think_close": false,
    "elapsed_seconds": 252.61663698311895,
    "endpoint": "http://127.0.0.1:19092/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T06:19:03Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.015581564977765083,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:9002/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 13264,
    "qwen_answer_text_sha256": "cae96e1476d2bdd78c4c7701b9ef70f8882093ae3a3bbdcd5e209da6637c21a9",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 13264,
    "qwen_visible_text_sha256": "cae96e1476d2bdd78c4c7701b9ef70f8882093ae3a3bbdcd5e209da6637c21a9",
    "raw_has_think_close": false,
    "reasoning_chars": 30971,
    "text_chars": 44236,
    "text_sha256": "004705f78884acd7d254e9109d3e79e2e49bedc484368a349825d41bebec297a",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 251.50327909295447,
    "vllm_decode_tps": 39.76091300306285,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 1.0949219870381057,
    "vllm_prefill_tps": 431.99424762628183,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 39.585674638952014
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestEarlier setup attempt retainedAdmission observer repair and replayPASS7,550 natural tokens2K 41.19510K 39.455 · diagnostic
Retest latency / counters / KV
{
  "retest_prior_attempts": [
    {
      "attempt": "v620-w4-tp4-mtp0-x-r0",
      "complete": false,
      "error": "CalledProcessError(2, ['/usr/bin/python3', '/home/ai/qwen38-release-retests-20261026/package-r1/profiles/v620-w4-tp4-mtp0-x/original-inputs/audit_rtn_admission.py', '/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp0-x-r0', '--bits', '4', '--pins', '/home/ai/qwen38-release-retests-20261026/package-r1/profiles/v620-w4-tp4-mtp0-x/original-inputs/rtn-admission-pins.json'])",
      "evidence": "retest-evidence/v620-w4-tp4-mtp0-x/attempts/v620-w4-tp4-mtp0-x-r0/evidence.tar.gz",
      "result": "retest-evidence/v620-w4-tp4-mtp0-x/attempts/v620-w4-tp4-mtp0-x-r0/result.json",
      "verification": "retest-evidence/v620-w4-tp4-mtp0-x/attempts/v620-w4-tp4-mtp0-x-r0/local-verification.json",
      "archive_sha256": "776e2317325cdc10fce98d45e6b7793e5b590b16c811a40e18338576cc466c30"
    }
  ],
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-27T01:35:53.161956943Z",
      "FinishedAt": "2026-10-27T02:15:21.513031262Z"
    },
    "devices_after": [
      {
        "card": 5,
        "render": 133,
        "pci": "0000:16:00.0",
        "actual_card": "card5",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:19:00.0",
        "actual_card": "card6",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 7,
        "render": 135,
        "pci": "0000:1c:00.0",
        "actual_card": "card7",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      },
      {
        "card": 8,
        "render": 136,
        "pci": "0000:1f:00.0",
        "actual_card": "card8",
        "busy_percent": 0,
        "vram_bytes": 21401600,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-27T02:15:21.654509+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 41.19541881788233,
  "retest_10K_TPS": 39.45477749262219,
  "retest_strict_tokens": 7550,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 7550,
      "elapsed_seconds": 189.7973721170565,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 188.68804516794626,
      "vllm_decode_tps": 40.01313381184242,
      "vllm_generation_tokens_delta": 7550.0,
      "vllm_prefill_seconds_delta": 1.0972324630711228,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 49.65721135598142,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 48.54908767505549,
      "vllm_decode_tps": 41.19541881788233,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 1.0951066319830716,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 254.5613265290158,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 253.4547305930173,
      "vllm_decode_tps": 39.45477749262219,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 1.0946464809821919,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 1.0951066319830716,
  "retest_2K_total_seconds": 49.65721135598142,
  "retest_10K_prefill_seconds": 1.0946464809821919,
  "retest_10K_total_seconds": 254.5613265290158,
  "retest_strict_prefill_seconds": 1.0972324630711228,
  "retest_strict_total_seconds": 189.7973721170565,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": "Complete record: chart-details/row-24-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 22575697920,
    "1": 22592425984,
    "2": 22592434176,
    "3": 22542094336
  },
  "retest_memory_observation_count": 37,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-w4-tp4-mtp0-x/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 13.57,
  "retest_logged_model_load_seconds": 65.883079,
  "retest_logged_available_KV_GiB": 13.65,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 13.65,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 223440,
  "retest_logged_context_capacity_x": 6.67,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "v620-w4-tp4-mtp0-x",
    "image": "sha256:23efb9f20a02e5b7bd69ed0004a7a1976d35ba728e4fd52f644b4645eadd5ed1",
    "container": "ec3518784deb32352092d77a2decf4d6324f174f83299da367262e1603a263bb",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 13.57,
    "retest_logged_model_load_seconds": 65.883079,
    "retest_logged_available_KV_GiB": 13.65,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 13.65,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 223440,
    "retest_logged_context_capacity_x": 6.67,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w4-tp4-mtp0-x/service-final.log",
    "source_log_sha256": "3ef08b7b4c885cc867a908039180b10d89226cb9991c59b40a34cce08d3cfc4c",
    "sealed_index_sha256": "f0b1471b89d3b00a16d5794a170ae274389ce3e0a9a1ca8f0b2ff7e9866e5f7e",
    "source_lines": {
      "model_load": {
        "line": 120,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-27 01:38:37 [gpu_model_runner.py:4579] Model loading took 13.57 GiB memory and 65.883079 seconds"
      },
      "automatic_KV_budget": {
        "line": 140,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-27 01:42:20 [gpu_worker.py:453] Available KV cache memory: 13.65 GiB"
      },
      "grouped_KV_tokens": {
        "line": 145,
        "text": "(EngineCore_DP0 pid=65) INFO 10-27 01:42:20 [kv_cache_utils.py:1314] GPU KV cache size: 223,440 tokens"
      },
      "context_capacity": {
        "line": 146,
        "text": "(EngineCore_DP0 pid=65) INFO 10-27 01:42:20 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.67x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w4-tp4-mtp0-x/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 21.040843963623047
}
Complete retest loaded admission
Retest KV budget: 13.650 GiB (automatic, rounded)223,440 grouped KV tokens · 6.67× reported capacityRetest sampled VRAM: 21.041 GiB max/cardRetest memory reports and limits
Evidence and limits

newer X candidate; unpromoted, historical reference retained

{
  "strict_tokens": 6851,
  "source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.RTN4_ordered_half_partials.serving_result",
  "source_revision": "4f5967174dde96d94c30f9f347f93ffe3da73b56",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "sampled_VRAM_startup_max_bytes_by_rank": "[22575652864, 22592380928, 22592438272, 22542057472]",
  "comparison_limit": "One sample; fresh versus seeded caches; KV1140 vs1149; no dynamic X invocation trace; original launch/startup/ladder parent exits unknown.",
  "retest_strict_tokens": 7550,
  "retest_MTP_accounting_by_tier": {},
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 22575697920,
    "1": 22592425984,
    "2": 22592434176,
    "3": 22542094336
  },
  "retest_memory_observation_count": 37,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/v620-w4-tp4-mtp0-x/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 13.65,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 13.65,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 223440,
  "retest_memory_log_report": {
    "profile": "v620-w4-tp4-mtp0-x",
    "image": "sha256:23efb9f20a02e5b7bd69ed0004a7a1976d35ba728e4fd52f644b4645eadd5ed1",
    "container": "ec3518784deb32352092d77a2decf4d6324f174f83299da367262e1603a263bb",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 13.57,
    "retest_logged_model_load_seconds": 65.883079,
    "retest_logged_available_KV_GiB": 13.65,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 13.65,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 223440,
    "retest_logged_context_capacity_x": 6.67,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/v620-w4-tp4-mtp0-x/service-final.log",
    "source_log_sha256": "3ef08b7b4c885cc867a908039180b10d89226cb9991c59b40a34cce08d3cfc4c",
    "sealed_index_sha256": "f0b1471b89d3b00a16d5794a170ae274389ce3e0a9a1ca8f0b2ff7e9866e5f7e",
    "source_lines": {
      "model_load": {
        "line": 120,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-27 01:38:37 [gpu_model_runner.py:4579] Model loading took 13.57 GiB memory and 65.883079 seconds"
      },
      "automatic_KV_budget": {
        "line": 140,
        "text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-27 01:42:20 [gpu_worker.py:453] Available KV cache memory: 13.65 GiB"
      },
      "grouped_KV_tokens": {
        "line": 145,
        "text": "(EngineCore_DP0 pid=65) INFO 10-27 01:42:20 [kv_cache_utils.py:1314] GPU KV cache size: 223,440 tokens"
      },
      "context_capacity": {
        "line": 146,
        "text": "(EngineCore_DP0 pid=65) INFO 10-27 01:42:20 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.67x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/v620-w4-tp4-mtp0-x/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 21.040843963623047,
  "sampled_peak_VRAM_GiB": 21.040847778320312
}
MI50Earlier Qwen3.8 TP8FP16Qwen/Qwen3.8-27BTP8MTP off PASS6151 tokens 67.30563.426 14.437 / 44.16110K: 14.813 / 172.486Strict: 13.197 / 107.360 6.30 / 31.33FP16 parameter extents / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:7d9a5e4969001af7785d888350428ad81fec613027782d326b42516fd6bc3dcd
Runtime image config: sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp8-mtp0-previous
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-r2/portable/launch.py --config mi50-fp16-tp8-mtp0-previous --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-fp16-tp8-mtp0-previous-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-15 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD128:/dev/dri/renderD128:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \
  --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
  --user root \
  --workdir /tmp \
  -e TRANSFORMERS_OFFLINE=1 \
  -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e HF_HUB_OFFLINE=1 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0-previous/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0-previous/inputs/profiles/mi50-fp16-tp8-mtp0-previous/overlays/vllm_command.sh,dst=/opt/vnext/profile/vllm_command.sh,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0-previous/inputs/profiles/mi50-fp16-tp8-mtp0-previous/overlays/mi50_tp8_worker.py,dst=/opt/mi50-tp8-observer/mi50_tp8_worker.py,readonly \
  --mount type=bind,src=/usr/share/ollama/vnext-offload/models/Qwen3.8-27B-1d4bf0f2ff60,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0-previous/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0-previous,dst=/runwork \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0-previous/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0-previous/cache/vllm,dst=/root/.cache/vllm \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp8-mtp0-previous
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-r2/portable/launch.py --config mi50-fp16-tp8-mtp0-previous --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict FAIL2K 67.367 · diagnostic / ineligible10K 63.511 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 10805,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 186.761883944273,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T21:57:45Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.012507418170571327,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 48647,
    "text_chars": 48647,
    "text_sha256": "9d9059ffcf98b5f4c6a9e0da75090231aaaf3a77127a7e5ca3da19c24aa0afc0",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 171.0692141884938,
    "vllm_decode_tps": 63.16156914180033,
    "vllm_generation_tokens_delta": 10805.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.682509177364409,
    "vllm_prefill_tps": 30.160989842283136,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 57.85441746359795
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 45.63218120113015,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T21:58:33Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008618236519396305,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 8718,
    "text_chars": 8718,
    "text_sha256": "e85c1b58cf5ead754a97bf391e47b1010ea5c827678a1fd1e4e73600326c0709",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 29.68798005953431,
    "vllm_decode_tps": 67.36733169415137,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.935353471897542,
    "vllm_prefill_tps": 29.68242912428326,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 43.82871796517294
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 172.98016954772174,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T22:01:26Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008071170188486576,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 45045,
    "text_chars": 45045,
    "text_sha256": "e2394ceddab1ea404538250f6ffcba7785f236db39ec71498d31ae36b66c8e1b",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 157.45232519879937,
    "vllm_decode_tps": 63.511288178018305,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.5185188902542,
    "vllm_prefill_tps": 30.47971287369758,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 57.81009479957297
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS6,117 natural tokens2K 67.60310K 63.714 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T17:09:31.497941141Z",
      "FinishedAt": "2026-10-26T17:30:28.269542448Z"
    },
    "devices_after": [
      {
        "card": 0,
        "render": 128,
        "pci": "0000:06:00.0",
        "actual_card": "card1",
        "busy_percent": 0,
        "vram_bytes": 20000714752,
        "dpm": "auto"
      },
      {
        "card": 1,
        "render": 129,
        "pci": "0000:09:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 21793456128,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:45:00.0",
        "actual_card": "card3",
        "busy_percent": 0,
        "vram_bytes": 21791809536,
        "dpm": "auto"
      },
      {
        "card": 3,
        "render": 131,
        "pci": "0000:48:00.0",
        "actual_card": "card4",
        "busy_percent": 0,
        "vram_bytes": 21790257152,
        "dpm": "auto"
      },
      {
        "card": 4,
        "render": 132,
        "pci": "0000:89:00.0",
        "actual_card": "card5",
        "busy_percent": 0,
        "vram_bytes": 21793210368,
        "dpm": "auto"
      },
      {
        "card": 5,
        "render": 133,
        "pci": "0000:8c:00.0",
        "actual_card": "card6",
        "busy_percent": 0,
        "vram_bytes": 21792894976,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:c5:00.0",
        "actual_card": "card7",
        "busy_percent": 0,
        "vram_bytes": 21791252480,
        "dpm": "auto"
      },
      {
        "card": 7,
        "render": 135,
        "pci": "0000:c8:00.0",
        "actual_card": "card8",
        "busy_percent": 0,
        "vram_bytes": 21738868736,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T17:30:28.397358+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 67.60343526281542,
  "retest_10K_TPS": 63.7137636364964,
  "retest_strict_tokens": 6117,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 6117,
      "elapsed_seconds": 108.61691073700786,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 93.25116309709847,
      "vllm_decode_tps": 65.59703704318012,
      "vllm_generation_tokens_delta": 6117.0,
      "vllm_prefill_seconds_delta": 15.356500206049532,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 45.094846380874515,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 29.584295416716486,
      "vllm_decode_tps": 67.60343526281542,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 15.502325372304767,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 171.70430587418377,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 156.95195871731266,
      "vllm_decode_tps": 63.7137636364964,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 14.742029319982976,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 15.502325372304767,
  "retest_2K_total_seconds": 45.094846380874515,
  "retest_10K_prefill_seconds": 14.742029319982976,
  "retest_10K_total_seconds": 171.70430587418377,
  "retest_strict_prefill_seconds": 15.356500206049532,
  "retest_strict_total_seconds": 108.61691073700786,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": "Complete record: chart-details/row-25-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33637744640,
    "1": 33584029696,
    "2": 33584132096,
    "3": 33584074752,
    "4": 33584177152,
    "5": 33584128000,
    "6": 33584078848,
    "7": 33533796352
  },
  "retest_memory_observation_count": 21,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-fp16-tp8-mtp0-previous/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 6.82,
  "retest_logged_model_load_seconds": 7.578667,
  "retest_logged_available_KV_GiB": 22.67,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 22.67,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 371072,
  "retest_logged_context_capacity_x": 11.26,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-fp16-tp8-mtp0-previous",
    "image": "sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04",
    "container": "8b859435de245533d593ce5501e9d10ad6d8dcc76d593b11ac845b427b5c56c2",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 6.82,
    "retest_logged_model_load_seconds": 7.578667,
    "retest_logged_available_KV_GiB": 22.67,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 22.67,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 371072,
    "retest_logged_context_capacity_x": 11.26,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-fp16-tp8-mtp0-previous/service-final.log",
    "source_log_sha256": "071c179f82976ea238ccede6a3e54f18071f17adf763c73baff256e6cfaa164f",
    "sealed_index_sha256": "36dc26d55a69b2bb0238dd8b67f921f2e7195d0d7794c8a7336849ac6cc94e73",
    "source_lines": {
      "model_load": {
        "line": 207,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:11:11 [gpu_model_runner.py:4584] Model loading took 6.82 GiB memory and 7.578667 seconds"
      },
      "automatic_KV_budget": {
        "line": 256,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:12:42 [gpu_worker.py:453] Available KV cache memory: 22.67 GiB"
      },
      "grouped_KV_tokens": {
        "line": 264,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 17:12:42 [kv_cache_utils.py:1314] GPU KV cache size: 371,072 tokens"
      },
      "context_capacity": {
        "line": 265,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 17:12:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.26x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-fp16-tp8-mtp0-previous/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 31.327590942382812
}
Complete retest loaded admission
Retest KV budget: 22.670 GiB (automatic, rounded)371,072 grouped KV tokens · 11.26× reported capacityRetest sampled VRAM: 31.328 GiB max/cardRetest memory reports and limits
Evidence and limits

Completed, independently reviewed strict-valid TP8 sample. Stopped because official fixed2K did not exceed 71.0 TPS; no promotion or TP4 goal credit.

{
  "strict_tokens": 6151,
  "source": "/home/joe/PhoneSync/Documents/gfx906/tuning/qwen38-fp16-matrix/tp8-as-is/result-20261023.json#$",
  "source_revision": "57f22fd70c859e4df9036fcc075a798fb032c31d",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "attention_KV_dtype": "FP16",
  "memory_accounting_limit": "Logical tensor/cache views may alias. Allocator counts and sampled physical VRAM are distinct; block geometry is not proof of a full-context request.",
  "sampled_VRAM_before_ladder_bytes_by_rank": "{\"0\": 33637703680, \"1\": 33584119808, \"2\": 33584070656, \"3\": 33584037888, \"4\": 33584087040, \"5\": 33584066560, \"6\": 33584033792, \"7\": 33533755392}",
  "sampled_VRAM_after_ladder_bytes_by_rank": "{\"0\": 33637703680, \"1\": 33584119808, \"2\": 33584070656, \"3\": 33584037888, \"4\": 33584087040, \"5\": 33584066560, \"6\": 33584033792, \"7\": 33533755392}",
  "KV_capacity": "{\"GDN_SSM\": {\"bytes\": 1403584512, \"dtype\": \"torch.float16\", \"shape\": [7139, 6, 128, 128]}, \"GDN_conv\": {\"bytes\": 54827520, \"dtype\": \"torch.float16\", \"shape\": [7139, 3, 1280]}, \"SSM_precision\": \"actual torch.float16 (original auto policy)\", \"attention\": {\"bytes\": 1520549888, \"dtype\": \"torch.float16\", \"shape\": [7139, 2, 208, 1, 256]}, \"block_size\": 208, \"blocks\": 7139}",
  "post_KV_allocator_allocated_bytes_by_rank": "{\"0\": 31288186880, \"1\": 31288186880, \"2\": 31288186880, \"3\": 31288186880, \"4\": 31288186880, \"5\": 31288186880, \"6\": 31288186880, \"7\": 31288186880}",
  "post_KV_allocator_reserved_bytes_by_rank": "{\"0\": 31669092352, \"1\": 31669092352, \"2\": 31669092352, \"3\": 31669092352, \"4\": 31669092352, \"5\": 31669092352, \"6\": 31669092352, \"7\": 31669092352}",
  "source_evidence_commit": "c7b96bd",
  "comparison_limit": "Exact original 57f image/native arithmetic; mounted TP8 profile and observation worker only. | Eight .30 cards and CPU0-15 differ from .50 TP4 control hardware/affinity/storage. No transferred causal gain. | TP4-only packed/interleaved/posted guards fall back; no forced TP4 shapes or new dispatch. | Actual target, attention KV, convolution state and SSM are FP16; not the separate MTP FP32-SSM composition. | Non-eager O3, maxseq2/maxbatched4, context131072, greedy original requests/scorer preserved. | Single sample: no repeatability, causal performance or historical Qwen3.6 transfer claim. | Strict is sole correctness authority; forced-length 2K/10K verdict fields retain their raw meanings. Strict TPS and 10K are diagnostic. | Logical tensor bytes are not unique physical storage; cache views can alias shared allocations. Do not sum all per-layer view bytes into physical VRAM. | Metadata and generated source/maps do not establish executed kernel traces. Wrapper source hash is directly observed; underlying qwen source is separately retained in effective-profile evidence. | 7139x208 describes allocated block geometry, not an executed 131072-token request or a naive usable-token guarantee. | Outer counters do not prove scorer internal scrape alignment or complete settlement. | Read-only retention preserves regular files; original modes/times, symlinks and socket inventory separately retained, sockets are not byte-hashed. | Assigned PCI06 kernel log appends 11 vm_bo_update_detail diagnostic lines during ladder and three more before closure; no identical-kernel-log or host-exclusivity claim. | Outer transport statuses supplied by owner; preserve remote successful command receipts separately. Login SSH1 is not a runtime failure.",
  "recorded_configuration_details": {
    "cell": "FP16_TP8_MTP_off",
    "status": "completed",
    "TP": 8,
    "native_MTP_tokens": 0,
    "host": "10.0.0.30",
    "strict_pass": true,
    "strict_tokens": 6151,
    "strict_diagnostic_TPS": 65.3299318056544,
    "official_2K_TPS": 67.30474071347291,
    "official_2K_tokens": 2000,
    "official_2K_prefill_seconds": 14.436688052956015,
    "official_2K_total_seconds": 44.16095239017159,
    "diagnostic_10K_TPS": 63.42627998549319,
    "diagnostic_10K_tokens": 10000,
    "diagnostic_10K_prefill_seconds": 14.813438802957535,
    "diagnostic_10K_total_seconds": 172.48586302390322,
    "SSM_dtype": "torch.float16",
    "source": "57f22fd70c859e4df9036fcc075a798fb032c31d",
    "image": "sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04",
    "primary": "tuning/qwen38-fp16-matrix/tp8-as-is/result-20261023.json",
    "strict_prefill_seconds": 13.196696631144732,
    "strict_total_seconds": 107.35985162388533,
    "per_rank_memory_JSON": [
      {
        "rank": 0,
        "pid": 196,
        "parameter_count": 611,
        "parameter_logical_bytes": 6767123584,
        "parameter_dtype": "torch.float16",
        "quantized_packed_bytes": 0,
        "observed_fixed3_buffer_bytes": 0,
        "after_load_allocated_bytes": 6958575104,
        "after_load_reserved_bytes": 7308574720,
        "after_cache_allocated_bytes": 31288186880,
        "after_cache_reserved_bytes": 31669092352,
        "attention_layers": 16,
        "GDN_layers": 48
      },
      {
        "rank": 1,
        "pid": 197,
        "parameter_count": 611,
        "parameter_logical_bytes": 6767123584,
        "parameter_dtype": "torch.float16",
        "quantized_packed_bytes": 0,
        "observed_fixed3_buffer_bytes": 0,
        "after_load_allocated_bytes": 6958575104,
        "after_load_reserved_bytes": 7308574720,
        "after_cache_allocated_bytes": 31288186880,
        "after_cache_reserved_bytes": 31669092352,
        "attention_layers": 16,
        "GDN_layers": 48
      },
      {
        "rank": 2,
        "pid": 198,
        "parameter_count": 611,
        "parameter_logical_bytes": 6767123584,
        "parameter_dtype": "torch.float16",
        "quantized_packed_bytes": 0,
        "observed_fixed3_buffer_bytes": 0,
        "after_load_allocated_bytes": 6958575104,
        "after_load_reserved_bytes": 7308574720,
        "after_cache_allocated_bytes": 31288186880,
        "after_cache_reserved_bytes": 31669092352,
        "attention_layers": 16,
        "GDN_layers": 48
      },
      {
        "rank": 3,
        "pid": 199,
        "parameter_count": 611,
        "parameter_logical_bytes": 6767123584,
        "parameter_dtype": "torch.float16",
        "quantized_packed_bytes": 0,
        "observed_fixed3_buffer_bytes": 0,
        "after_load_allocated_bytes": 6958575104,
        "after_load_reserved_bytes": 7308574720,
        "after_cache_allocated_bytes": 31288186880,
        "after_cache_reserved_bytes": 31669092352,
        "attention_layers": 16,
        "GDN_layers": 48
      },
      {
        "rank": 4,
        "pid": 200,
        "parameter_count": 611,
        "parameter_logical_bytes": 6767123584,
        "parameter_dtype": "torch.float16",
        "quantized_packed_bytes": 0,
        "observed_fixed3_buffer_bytes": 0,
        "after_load_allocated_bytes": 6958575104,
        "after_load_reserved_bytes": 7308574720,
        "after_cache_allocated_bytes": 31288186880,
        "after_cache_reserved_bytes": 31669092352,
        "attention_layers": 16,
        "GDN_layers": 48
      },
      {
        "rank": 5,
        "pid": 201,
        "parameter_count": 611,
        "parameter_logical_bytes": 6767123584,
        "parameter_dtype": "torch.float16",
        "quantized_packed_bytes": 0,
        "observed_fixed3_buffer_bytes": 0,
        "after_load_allocated_bytes": 6958575104,
        "after_load_reserved_bytes": 7308574720,
        "after_cache_allocated_bytes": 31288186880,
        "after_cache_reserved_bytes": 31669092352,
        "attention_layers": 16,
        "GDN_layers": 48
      },
      {
        "rank": 6,
        "pid": 202,
        "parameter_count": 611,
        "parameter_logical_bytes": 6767123584,
        "parameter_dtype": "torch.float16",
        "quantized_packed_bytes": 0,
        "observed_fixed3_buffer_bytes": 0,
        "after_load_allocated_bytes": 6958575104,
        "after_load_reserved_bytes": 7308574720,
        "after_cache_allocated_bytes": 31288186880,
        "after_cache_reserved_bytes": 31669092352,
        "attention_layers": 16,
        "GDN_layers": 48
      },
      {
        "rank": 7,
        "pid": 203,
        "parameter_count": 611,
        "parameter_logical_bytes": 6767123584,
        "parameter_dtype": "torch.float16",
        "quantized_packed_bytes": 0,
        "observed_fixed3_buffer_bytes": 0,
        "after_load_allocated_bytes": 6958575104,
        "after_load_reserved_bytes": 7308574720,
        "after_cache_allocated_bytes": 31288186880,
        "after_cache_reserved_bytes": 31669092352,
        "attention_layers": 16,
        "GDN_layers": 48
      }
    ],
    "KV_JSON": {
      "attention": {
        "shape": [
          7139,
          2,
          208,
          1,
          256
        ],
        "dtype": "torch.float16",
        "bytes": 1520549888
      },
      "GDN_conv": {
        "shape": [
          7139,
          3,
          1280
        ],
        "dtype": "torch.float16",
        "bytes": 54827520
      },
      "GDN_SSM": {
        "shape": [
          7139,
          6,
          128,
          128
        ],
        "dtype": "torch.float16",
        "bytes": 1403584512
      },
      "blocks": 7139,
      "block_size": 208,
      "SSM_precision": "actual torch.float16 (original auto policy)"
    },
    "VRAM_JSON": {
      "before": "renderD128 33637703680\nrenderD129 33584119808\nrenderD130 33584070656\nrenderD131 33584037888\nrenderD132 33584087040\nrenderD133 33584066560\nrenderD134 33584033792\nrenderD135 33533755392",
      "after": "renderD128 33637703680\nrenderD129 33584119808\nrenderD130 33584070656\nrenderD131 33584037888\nrenderD132 33584087040\nrenderD133 33584066560\nrenderD134 33584033792\nrenderD135 33533755392"
    },
    "attention_KV_dtype": "torch.float16",
    "context": 131072,
    "speculative_accounting_JSON": null
  },
  "detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/fp16-matrix.csv",
  "detail_source_sha256": "8643d90cf1f211a7f77be9f6212cd7706e6dbae3ae84b9bc80b5f42fa199ffe0",
  "detail_source_pointer": "FP16_TP8_MTP_off",
  "retest_strict_tokens": 6117,
  "retest_MTP_accounting_by_tier": {},
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33637744640,
    "1": 33584029696,
    "2": 33584132096,
    "3": 33584074752,
    "4": 33584177152,
    "5": 33584128000,
    "6": 33584078848,
    "7": 33533796352
  },
  "retest_memory_observation_count": 21,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-fp16-tp8-mtp0-previous/runtime-observations.jsonl",
  "retest_logged_available_KV_GiB": 22.67,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 22.67,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 371072,
  "retest_memory_log_report": {
    "profile": "mi50-fp16-tp8-mtp0-previous",
    "image": "sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04",
    "container": "8b859435de245533d593ce5501e9d10ad6d8dcc76d593b11ac845b427b5c56c2",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 6.82,
    "retest_logged_model_load_seconds": 7.578667,
    "retest_logged_available_KV_GiB": 22.67,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 22.67,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 371072,
    "retest_logged_context_capacity_x": 11.26,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-fp16-tp8-mtp0-previous/service-final.log",
    "source_log_sha256": "071c179f82976ea238ccede6a3e54f18071f17adf763c73baff256e6cfaa164f",
    "sealed_index_sha256": "36dc26d55a69b2bb0238dd8b67f921f2e7195d0d7794c8a7336849ac6cc94e73",
    "source_lines": {
      "model_load": {
        "line": 207,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:11:11 [gpu_model_runner.py:4584] Model loading took 6.82 GiB memory and 7.578667 seconds"
      },
      "automatic_KV_budget": {
        "line": 256,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:12:42 [gpu_worker.py:453] Available KV cache memory: 22.67 GiB"
      },
      "grouped_KV_tokens": {
        "line": 264,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 17:12:42 [kv_cache_utils.py:1314] GPU KV cache size: 371,072 tokens"
      },
      "context_capacity": {
        "line": 265,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 17:12:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.26x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-fp16-tp8-mtp0-previous/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 31.327590942382812,
  "sampled_peak_VRAM_GiB": 31.327552795410156
}
MI50Current Qwen3.8 TP8FP16Qwen/Qwen3.8-27BTP8MTP off PASS14795 tokens 70.37266.406 15.524 / 43.95310K: 14.785 / 165.383Strict: 14.693 / 245.164 6.30 / 31.46retained FP16 parameter extents; projection buffers separate / VRAM GiBSeparate packed-buffer extents: 2.188 GiB/rank; includes alternate views, not additive physical storage. Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:6d33e0476cbb771490316875aaa8bbc4d2a24c84c5c82cb979f3729518971b46
Runtime image config: sha256:0aab9fc8441d0d25b576bdb70ef0c9cbb6982eb7ac37a7f108fee2dba77a40ce
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp8-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-r2/portable/launch.py --config mi50-fp16-tp8-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-fp16-tp8-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-15 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD128:/dev/dri/renderD128:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \
  --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
  --user root \
  --workdir /tmp \
  -e TRANSFORMERS_OFFLINE=1 \
  -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e HF_HUB_OFFLINE=1 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  --mount type=bind,src=/usr/share/ollama/vnext-offload/models/Qwen3.8-27B-1d4bf0f2ff60,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0/inputs/profiles/mi50-fp16-tp8-mtp0/overlays/vllm_command.sh,dst=/opt/vnext/profile/vllm_command.sh,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0/inputs/profiles/mi50-fp16-tp8-mtp0/overlays/mi50_tp8_worker.py,dst=/opt/mi50-tp8-observer/mi50_tp8_worker.py,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0,dst=/runwork \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0/inputs/profiles/mi50-fp16-tp8-mtp0/overlays/serving-pins.json,dst=/opt/mi50-tp8-observer/serving-pins.json,readonly \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:0aab9fc8441d0d25b576bdb70ef0c9cbb6982eb7ac37a7f108fee2dba77a40ce
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp8-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-r2/portable/launch.py --config mi50-fp16-tp8-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict FAIL2K 70.046 · diagnostic / ineligible10K 66.129 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 15240,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 254.38865686114877,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T12:30:08Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.013873212039470673,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 32507,
    "text_chars": 32507,
    "text_sha256": "97b5e51a52f86183973ec7e490147143eccaa1ab9f5cec6ee32a7a52a9f138b9",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 239.16702868323773,
    "vllm_decode_tps": 63.721157903351546,
    "vllm_generation_tokens_delta": 15240.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.211745806969702,
    "vllm_prefill_tps": 31.094392846301794,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 59.908331558660436
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 43.61238981690258,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T12:30:53Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.009367557242512703,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 7986,
    "text_chars": 7986,
    "text_sha256": "a90fec54997946ef62a25d7700199fcb23aa456107b173b2007db62135aeae9d",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 28.552836760878563,
    "vllm_decode_tps": 70.04557959509941,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.050994507968426,
    "vllm_prefill_tps": 31.426494757511225,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 45.85852800996639
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 166.6537444461137,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T12:33:40Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008907177485525608,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 22835,
    "text_chars": 22835,
    "text_sha256": "2bff7094c90509554fe661af4c5fa3e872ed92c8f90ff7ad0cca77dd4bd7ab89",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 151.21882070507854,
    "vllm_decode_tps": 66.12933465142517,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.426380563527346,
    "vllm_prefill_tps": 30.66176139322764,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 60.004652360112004
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS10,210 natural tokens2K 70.34010K 66.386 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T15:57:46.384320926Z",
      "FinishedAt": "2026-10-26T16:20:30.268261294Z"
    },
    "devices_after": [
      {
        "card": 0,
        "render": 128,
        "pci": "0000:06:00.0",
        "actual_card": "card1",
        "busy_percent": 0,
        "vram_bytes": 21298290688,
        "dpm": "auto"
      },
      {
        "card": 1,
        "render": 129,
        "pci": "0000:09:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 21178646528,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:45:00.0",
        "actual_card": "card3",
        "busy_percent": 0,
        "vram_bytes": 21178335232,
        "dpm": "auto"
      },
      {
        "card": 3,
        "render": 131,
        "pci": "0000:48:00.0",
        "actual_card": "card4",
        "busy_percent": 0,
        "vram_bytes": 22552678400,
        "dpm": "auto"
      },
      {
        "card": 4,
        "render": 132,
        "pci": "0000:89:00.0",
        "actual_card": "card5",
        "busy_percent": 0,
        "vram_bytes": 22555885568,
        "dpm": "auto"
      },
      {
        "card": 5,
        "render": 133,
        "pci": "0000:8c:00.0",
        "actual_card": "card6",
        "busy_percent": 0,
        "vram_bytes": 22555213824,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:c5:00.0",
        "actual_card": "card7",
        "busy_percent": 0,
        "vram_bytes": 22556356608,
        "dpm": "auto"
      },
      {
        "card": 7,
        "render": 135,
        "pci": "0000:c8:00.0",
        "actual_card": "card8",
        "busy_percent": 0,
        "vram_bytes": 22505455616,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T16:20:30.387589+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 70.33971228281604,
  "retest_10K_TPS": 66.38633126361084,
  "retest_strict_tokens": 10210,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 10210,
      "elapsed_seconds": 168.26249494263902,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 153.90207020984963,
      "vllm_decode_tps": 66.34088798206801,
      "vllm_generation_tokens_delta": 10210.0,
      "vllm_prefill_seconds_delta": 14.347171621862799,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 44.575539912097156,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 28.433440159074962,
      "vllm_decode_tps": 70.33971228281604,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 16.134185431990772,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 164.31250084890053,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 150.6334182000719,
      "vllm_decode_tps": 66.38633126361084,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 13.67035356303677,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 16.134185431990772,
  "retest_2K_total_seconds": 44.575539912097156,
  "retest_10K_prefill_seconds": 13.67035356303677,
  "retest_10K_total_seconds": 164.31250084890053,
  "retest_strict_prefill_seconds": 14.347171621862799,
  "retest_strict_total_seconds": 168.26249494263902,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": "Complete record: chart-details/row-26-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_logged_model_load_delta_GiB": 9.07,
  "retest_logged_model_load_seconds": 22.691548,
  "retest_logged_available_KV_GiB": 20.57,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 20.57,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 336752,
  "retest_logged_context_capacity_x": 10.22,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-fp16-tp8-mtp0",
    "image": "sha256:0aab9fc8441d0d25b576bdb70ef0c9cbb6982eb7ac37a7f108fee2dba77a40ce",
    "container": "ae82bff4b20a076f77a910c5721aa5ac0fc946b40a75fd1ab511e08b08f418aa",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 9.07,
    "retest_logged_model_load_seconds": 22.691548,
    "retest_logged_available_KV_GiB": 20.57,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 20.57,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 336752,
    "retest_logged_context_capacity_x": 10.22,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-fp16-tp8-mtp0/service-final.log",
    "source_log_sha256": "73b42cdf85e17a4b8fc446101f58a84f60f280529af15dc1711fd4dada425b46",
    "sealed_index_sha256": "4578ad1dc43c02e27e9e24e23f5962720d6906c26ef4715afe24cb4f342f5279",
    "source_lines": {
      "model_load": {
        "line": 216,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 15:59:42 [gpu_model_runner.py:4584] Model loading took 9.07 GiB memory and 22.691548 seconds"
      },
      "automatic_KV_budget": {
        "line": 267,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:02:06 [gpu_worker.py:453] Available KV cache memory: 20.57 GiB"
      },
      "grouped_KV_tokens": {
        "line": 274,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:02:06 [kv_cache_utils.py:1314] GPU KV cache size: 336,752 tokens"
      },
      "context_capacity": {
        "line": 275,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:02:06 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 10.22x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-fp16-tp8-mtp0/memory-log-report.json"
}
Complete retest loaded admission
Retest KV budget: 20.570 GiB (automatic, rounded)336,752 grouped KV tokens · 10.22× reported capacityRetest memory reports and limitsAccepted service restored on .30 · VerificationFinal HTTP / container check
Evidence and limits

Accepted TP8 deployment baseline under strict PASS and >=70 TPS. Exact tested image/profile restored as a new service instance on .30; completed inference and HTTP200 verified. Original measured cohort remains stopped under historical >71 rule. One eligible benchmark sample, no TP4 qualification.

{
  "source": "/home/joe/PhoneSync/Documents/gfx906/tuning/fixed3-tp8-gateup/serving/result-20261023.json#$",
  "source_revision": "f3aff262c7ce2ed5b65a156ce1f7cd5dc5bb6d7c",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "strict_tokens": 14795,
  "attention_KV_dtype": "FP16",
  "memory_accounting_limit": "Target parameter extents and lossless packed gate/up buffers are separate representations, not a disjoint unique whole-model footprint. Logical KV views may alias. Allocator counters and sampled physical VRAM are separate. No full-context request was tested.",
  "sampled_VRAM_before_ladder_bytes_by_rank": "{\"0\": 33783861248, \"1\": 33730301952, \"2\": 33730269184, \"3\": 33730314240, \"4\": 33730240512, \"5\": 33730314240, \"6\": 33730297856, \"7\": 33679798272}",
  "sampled_VRAM_after_ladder_bytes_by_rank": "{\"0\": 33783861248, \"1\": 33730301952, \"2\": 33730269184, \"3\": 33730314240, \"4\": 33730240512, \"5\": 33730314240, \"6\": 33730297856, \"7\": 33679798272}",
  "KV_capacity": "{\"actual_attention_shape\": [6477, 2, 208, 1, 256], \"actual_blocks\": 6477, \"available_KV_GiB\": 20.57, \"limit\": \"Engine grouped-cache interpretation differs from raw view extent; no full-context inference performed.\", \"reported_GPU_KV_tokens\": 336752, \"reported_max_concurrency_at_131072\": 10.22}",
  "attention_KV_logical_view_shape": "[6477, 2, 208, 1, 256]",
  "post_KV_allocator_allocated_bytes_by_rank": "{\"0\": 31438801408, \"1\": 31438786560, \"2\": 31438564864, \"3\": 31438415360, \"4\": 31438394880, \"5\": 31438513664, \"6\": 31438672896, \"7\": 31438678016}",
  "post_KV_allocator_reserved_bytes_by_rank": "{\"0\": 31803310080, \"1\": 31803310080, \"2\": 31803310080, \"3\": 31803310080, \"4\": 31803310080, \"5\": 31803310080, \"6\": 31803310080, \"7\": 31803310080}",
  "source_evidence_commit": "ca367c2",
  "comparison_limit": "New host-only TP8 dispatch and separate gate/up packing; full fixed3 device ELF unchanged. | Dense gate-then-up weight layout preserved. M1 fused K5120; M2..4 original opaque GEMM plus exact retained compiled TI7 activation; M>4 original Python path. | Only VLLM_GFX906_TP8_GATE_FIXED3=1 added to original TP8 profile. | Mounted intended profile deliberately differs from stock image profile in installed.sha256. Full manifest passes at image build; runtime observes this replacement separately. | New private rootFS cache; other projection/collective eligibility and all requests/scorer unchanged. | Outer scrapes in either cohort do not expose internal scorer scrapes or the full settlement trajectory. | One official sample per configuration; no repeatability or causal claim. | Fixed-tier strict flags are not model correctness verdicts; uncapped strict is sole correctness authority. | No TP4 qualification, formal promotion or model causal attribution. | Gate/up-only packed route on the same .30 host; actual eight FP16 ranks admitted. Other projection families retain fallback. Generated callsites are not dynamic per-token kernel traces. | Fresh private rootSSD caches; actual KV blocks 6477 versus original dense 7139. One sample per configuration; observed gain is not isolated causal attribution. | Scoped kernel logs changed; retained bounded fault/reset/error signatures have zero matches. Active debug logging is not assumed performance-neutral. | Original >71 rule closed the measured cohort. Joe subsequently accepted strict-valid >=70 TPS; a service restoration does not create a new benchmark sample. | Fixed-tier raw strict flags are not model verdicts; uncapped strict PASS determines eligibility.",
  "recorded_configuration_details": {
    "cell": "FP16_TP8_gateup_MTP_off",
    "status": "completed",
    "TP": 8,
    "native_MTP_tokens": 0,
    "host": "10.0.0.30",
    "strict_pass": true,
    "strict_tokens": 14795,
    "strict_diagnostic_TPS": 64.19733379026958,
    "official_2K_TPS": 70.37197138638763,
    "official_2K_tokens": 2000,
    "official_2K_prefill_seconds": 15.52441488718614,
    "official_2K_total_seconds": 43.95317275123671,
    "diagnostic_10K_TPS": 66.40603683633941,
    "diagnostic_10K_tokens": 10000,
    "diagnostic_10K_prefill_seconds": 14.784574894234538,
    "diagnostic_10K_total_seconds": 165.38285010913387,
    "SSM_dtype": "torch.float16",
    "source": "f3aff262c7ce2ed5b65a156ce1f7cd5dc5bb6d7c",
    "image": "sha256:0aab9fc8441d0d25b576bdb70ef0c9cbb6982eb7ac37a7f108fee2dba77a40ce",
    "primary": "tuning/fixed3-tp8-gateup/serving/result-20261023.json",
    "strict_prefill_seconds": 14.693329483736306,
    "strict_total_seconds": 245.16398947313428,
    "per_rank_memory_JSON": [
      {
        "rank": 0,
        "worker_pid": 196,
        "device": "cuda:0",
        "target_parameter_logical_bytes": 6767123584,
        "gateup_packed_owners": 64,
        "gateup_packed_bytes": 2349609008,
        "after_load_allocated_bytes": 9359171584,
        "after_load_reserved_bytes": 9724493824,
        "after_cache_allocated_bytes": 31438801408,
        "after_cache_reserved_bytes": 31803310080
      },
      {
        "rank": 1,
        "worker_pid": 197,
        "device": "cuda:1",
        "target_parameter_logical_bytes": 6767123584,
        "gateup_packed_owners": 64,
        "gateup_packed_bytes": 2349593600,
        "after_load_allocated_bytes": 9359156736,
        "after_load_reserved_bytes": 9724493824,
        "after_cache_allocated_bytes": 31438786560,
        "after_cache_reserved_bytes": 31803310080
      },
      {
        "rank": 2,
        "worker_pid": 198,
        "device": "cuda:2",
        "target_parameter_logical_bytes": 6767123584,
        "gateup_packed_owners": 64,
        "gateup_packed_bytes": 2349373240,
        "after_load_allocated_bytes": 9358935040,
        "after_load_reserved_bytes": 9724493824,
        "after_cache_allocated_bytes": 31438564864,
        "after_cache_reserved_bytes": 31803310080
      },
      {
        "rank": 3,
        "worker_pid": 199,
        "device": "cuda:3",
        "target_parameter_logical_bytes": 6767123584,
        "gateup_packed_owners": 64,
        "gateup_packed_bytes": 2349224056,
        "after_load_allocated_bytes": 9358785536,
        "after_load_reserved_bytes": 9724493824,
        "after_cache_allocated_bytes": 31438415360,
        "after_cache_reserved_bytes": 31803310080
      },
      {
        "rank": 4,
        "worker_pid": 200,
        "device": "cuda:4",
        "target_parameter_logical_bytes": 6767123584,
        "gateup_packed_owners": 64,
        "gateup_packed_bytes": 2349203776,
        "after_load_allocated_bytes": 9358765056,
        "after_load_reserved_bytes": 9724493824,
        "after_cache_allocated_bytes": 31438394880,
        "after_cache_reserved_bytes": 31803310080
      },
      {
        "rank": 5,
        "worker_pid": 201,
        "device": "cuda:5",
        "target_parameter_logical_bytes": 6767123584,
        "gateup_packed_owners": 64,
        "gateup_packed_bytes": 2349321712,
        "after_load_allocated_bytes": 9358883840,
        "after_load_reserved_bytes": 9724493824,
        "after_cache_allocated_bytes": 31438513664,
        "after_cache_reserved_bytes": 31803310080
      },
      {
        "rank": 6,
        "worker_pid": 202,
        "device": "cuda:6",
        "target_parameter_logical_bytes": 6767123584,
        "gateup_packed_owners": 64,
        "gateup_packed_bytes": 2349479896,
        "after_load_allocated_bytes": 9359043072,
        "after_load_reserved_bytes": 9724493824,
        "after_cache_allocated_bytes": 31438672896,
        "after_cache_reserved_bytes": 31803310080
      },
      {
        "rank": 7,
        "worker_pid": 203,
        "device": "cuda:7",
        "target_parameter_logical_bytes": 6767123584,
        "gateup_packed_owners": 64,
        "gateup_packed_bytes": 2349484816,
        "after_load_allocated_bytes": 9359048192,
        "after_load_reserved_bytes": 9724493824,
        "after_cache_allocated_bytes": 31438678016,
        "after_cache_reserved_bytes": 31803310080
      }
    ],
    "KV_JSON": {
      "blocks": 6477,
      "block_size": 208,
      "attention_example": {
        "shape": [
          6477,
          2,
          208,
          1,
          256
        ],
        "stride": [
          106496,
          53248,
          256,
          256,
          1
        ],
        "dtype": "torch.float16",
        "device": "cuda:0",
        "bytes": 1379549184,
        "storage_offset": 0
      },
      "conv_example": {
        "shape": [
          6477,
          3,
          1280
        ],
        "stride": [
          106496,
          1280,
          1
        ],
        "dtype": "torch.float16",
        "device": "cuda:0",
        "bytes": 49743360,
        "storage_offset": 0
      },
      "SSM_example": {
        "shape": [
          6477,
          6,
          128,
          128
        ],
        "stride": [
          106496,
          16384,
          128,
          1
        ],
        "dtype": "torch.float16",
        "device": "cuda:0",
        "bytes": 1273430016,
        "storage_offset": 3840
      },
      "attention_layers_per_rank": 16,
      "GDN_layers_per_rank": 48,
      "configured_SSM_dtype": "auto",
      "configured_conv_dtype": "auto",
      "logged_available_KV_GiB": 20.57,
      "logged_GPU_KV_tokens": 336752,
      "logged_max_concurrency_at_131072": 10.22,
      "limits": "Actual views may share/interleave physical grouped allocation. Logical bytes and raw block extents are not additive physical storage or a full-context execution proof; engine grouped token/concurrency reports use different accounting."
    },
    "VRAM_JSON": {
      "before": [
        {
          "render": 128,
          "PCI": "0000:06:00.0",
          "bytes": 33783861248,
          "busy": 0
        },
        {
          "render": 129,
          "PCI": "0000:09:00.0",
          "bytes": 33730301952,
          "busy": 0
        },
        {
          "render": 130,
          "PCI": "0000:45:00.0",
          "bytes": 33730269184,
          "busy": 0
        },
        {
          "render": 131,
          "PCI": "0000:48:00.0",
          "bytes": 33730314240,
          "busy": 0
        },
        {
          "render": 132,
          "PCI": "0000:89:00.0",
          "bytes": 33730240512,
          "busy": 0
        },
        {
          "render": 133,
          "PCI": "0000:8c:00.0",
          "bytes": 33730314240,
          "busy": 0
        },
        {
          "render": 134,
          "PCI": "0000:c5:00.0",
          "bytes": 33730297856,
          "busy": 0
        },
        {
          "render": 135,
          "PCI": "0000:c8:00.0",
          "bytes": 33679798272,
          "busy": 0
        }
      ],
      "after": [
        {
          "render": 128,
          "PCI": "0000:06:00.0",
          "bytes": 33783861248,
          "busy": 0
        },
        {
          "render": 129,
          "PCI": "0000:09:00.0",
          "bytes": 33730301952,
          "busy": 0
        },
        {
          "render": 130,
          "PCI": "0000:45:00.0",
          "bytes": 33730269184,
          "busy": 0
        },
        {
          "render": 131,
          "PCI": "0000:48:00.0",
          "bytes": 33730314240,
          "busy": 0
        },
        {
          "render": 132,
          "PCI": "0000:89:00.0",
          "bytes": 33730240512,
          "busy": 0
        },
        {
          "render": 133,
          "PCI": "0000:8c:00.0",
          "bytes": 33730314240,
          "busy": 0
        },
        {
          "render": 134,
          "PCI": "0000:c5:00.0",
          "bytes": 33730297856,
          "busy": 0
        },
        {
          "render": 135,
          "PCI": "0000:c8:00.0",
          "bytes": 33679798272,
          "busy": 0
        }
      ],
      "limits": "Two own-card samples, not peaks/continuous measurement or model-only allocation."
    },
    "attention_KV_dtype": "torch.float16",
    "context": 131072,
    "speculative_accounting_JSON": null
  },
  "detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/fp16-matrix.csv",
  "detail_source_sha256": "8643d90cf1f211a7f77be9f6212cd7706e6dbae3ae84b9bc80b5f42fa199ffe0",
  "detail_source_pointer": "FP16_TP8_gateup_MTP_off",
  "retest_strict_tokens": 10210,
  "retest_MTP_accounting_by_tier": {},
  "retest_logged_available_KV_GiB": 20.57,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 20.57,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 336752,
  "retest_memory_log_report": {
    "profile": "mi50-fp16-tp8-mtp0",
    "image": "sha256:0aab9fc8441d0d25b576bdb70ef0c9cbb6982eb7ac37a7f108fee2dba77a40ce",
    "container": "ae82bff4b20a076f77a910c5721aa5ac0fc946b40a75fd1ab511e08b08f418aa",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 9.07,
    "retest_logged_model_load_seconds": 22.691548,
    "retest_logged_available_KV_GiB": 20.57,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 20.57,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 336752,
    "retest_logged_context_capacity_x": 10.22,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-fp16-tp8-mtp0/service-final.log",
    "source_log_sha256": "73b42cdf85e17a4b8fc446101f58a84f60f280529af15dc1711fd4dada425b46",
    "sealed_index_sha256": "4578ad1dc43c02e27e9e24e23f5962720d6906c26ef4715afe24cb4f342f5279",
    "source_lines": {
      "model_load": {
        "line": 216,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 15:59:42 [gpu_model_runner.py:4584] Model loading took 9.07 GiB memory and 22.691548 seconds"
      },
      "automatic_KV_budget": {
        "line": 267,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:02:06 [gpu_worker.py:453] Available KV cache memory: 20.57 GiB"
      },
      "grouped_KV_tokens": {
        "line": 274,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:02:06 [kv_cache_utils.py:1314] GPU KV cache size: 336,752 tokens"
      },
      "context_capacity": {
        "line": 275,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:02:06 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 10.22x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-fp16-tp8-mtp0/memory-log-report.json",
  "sampled_peak_VRAM_GiB": 31.463672637939453
}
MI50Historical vNext Qwen3.6Host 10.0.0.20 · Validated Profile Results / table 1GGUF F16Qwen/Qwen3.6-27BTP8MTP off PASS3527 tokens 70.75966.316 13.168 / 41.44210K: 13.567 / 164.371Strict: 13.609 / 64.065Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 22.79 GiB · engine report373,360 grouped KV tokens · 11.32× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-27
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-27 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row27-9d44a1a055-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-27/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-27/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-27/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-27/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-27/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-27/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense"
python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-27
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-27 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 71.013 · valid historical-model sample10K 66.570 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3421,
    "content_chars": 5352,
    "content_has_think_close": false,
    "elapsed_seconds": 63.232292518951,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T23:52:09Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.012749253772199154,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 5352,
    "qwen_answer_text_sha256": "f123bac2c95b41007733c66683e97c2cd19f181be25caa76dd4d86ef924aab84",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 5352,
    "qwen_visible_text_sha256": "f123bac2c95b41007733c66683e97c2cd19f181be25caa76dd4d86ef924aab84",
    "raw_has_think_close": false,
    "reasoning_chars": 11201,
    "text_chars": 16554,
    "text_sha256": "95314c43ec2e43a890c71f9cbe34a9813336a76dbae28f504f567f35986f987a",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 48.71209065709263,
    "vllm_decode_tps": 70.22897095675962,
    "vllm_generation_tokens_delta": 3421.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.510479955002666,
    "vllm_prefill_tps": 29.702670162292424,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 54.10210295593365
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 43.64001283608377,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T23:52:53Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.006201061420142651,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9558,
    "text_chars": 9558,
    "text_sha256": "f2524b3bdc797c6ebdb939c9ed1502650b1763a8a7da2a6a6a6b4b4f3cc0c1a9",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 28.163926037028432,
    "vllm_decode_tps": 71.01282674050863,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.466903982684016,
    "vllm_prefill_tps": 27.865951743317627,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 45.82950072705521
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 43611,
    "content_has_think_close": false,
    "elapsed_seconds": 163.7763210022822,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T23:55:37Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.00586194172501564,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 43611,
    "qwen_answer_text_sha256": "f92ff16a3422cc7f045c52069c185be251115758eea20cd7ee6491dbc461eea6",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 43611,
    "qwen_visible_text_sha256": "f92ff16a3422cc7f045c52069c185be251115758eea20cd7ee6491dbc461eea6",
    "raw_has_think_close": false,
    "reasoning_chars": 11201,
    "text_chars": 54813,
    "text_sha256": "77ec6bd95bfb18605654bceeb0d512f67f3524a5bca93893f178f8e563bf88f2",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 150.21777959540486,
    "vllm_decode_tps": 66.57001605891064,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 13.549391930922866,
    "vllm_prefill_tps": 31.809545564650595,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 61.058887748862375
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 27, 28, 32, 33, 34. One new sample.PASS4,508 natural tokens2K 71.02510K 66.531 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-08T22:18:52.716405342Z",
    "FinishedAt": "2026-11-08T22:40:00.260467584Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "9a55da176fe75043aa7dc1ca12883e4f952738b9ebf2616b6f0edce4cb417ed7",
  "retest_strict": "PASS",
  "retest_strict_tokens": 4508,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 4508,
      "decode_tps": 69.61732325093088,
      "prefill_seconds": 14.042354583740234,
      "client_total_seconds": 78.80580217391253,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 64.75399784836918,
        "vllm:request_prefill_time_seconds_sum": 14.042354583740234,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 4508.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
      "summary_sha256": "754f0c5eeaf987089a936b03a08af821069b38f7e14fa59e17115de3fc4d2fae"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 71.02517985930231,
      "prefill_seconds": 13.268245195038617,
      "client_total_seconds": 41.43482113070786,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 28.15902760066092,
        "vllm:request_prefill_time_seconds_sum": 13.268245195038617,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
      "summary_sha256": "aff3ec02e75684eb8bd8ecdd4936b31fc9fbb3754970d3b0a39d80e61d360495"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 66.53116656725801,
      "prefill_seconds": 12.96059922594577,
      "client_total_seconds": 163.27578648924828,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 150.3054961450398,
        "vllm:request_prefill_time_seconds_sum": 12.96059922594577,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
      "summary_sha256": "29705fa055020f3834638960e4af19073f76c5672f7f2d3c621b46c7bf61f6ef"
    }
  },
  "retest_strict_TPS": 69.61732325093088,
  "retest_strict_prefill_seconds": 14.042354583740234,
  "retest_strict_total_seconds": 78.80580217391253,
  "retest_2K_TPS": 71.02517985930231,
  "retest_2K_prefill_seconds": 13.268245195038617,
  "retest_2K_total_seconds": 41.43482113070786,
  "retest_10K_TPS": 66.53116656725801,
  "retest_10K_prefill_seconds": 12.96059922594577,
  "retest_10K_total_seconds": 163.27578648924828,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33758752768,
    "1": 33634103296,
    "2": 33684492288,
    "3": 33634070528,
    "4": 33684193280,
    "5": 33633898496,
    "6": 33684754432,
    "7": 33634291712
  },
  "retest_sampled_peak_VRAM_GiB": 31.440288543701172,
  "retest_memory_observation_count": 601,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 6.75,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_logged_context_capacity_x": 11.32,
  "retest_memory_log_source": "historical-recovered/fresh-row-27/result.json"
}
Fresh sampled VRAM: 31.440 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 22.79 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid; ai-info 10K gate cleared

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 3527,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r2.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-27.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 27,
    "original_host": "10.0.0.20",
    "logged_model_load_delta_GiB": 6.75,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 109.895363,
    "logged_available_KV_GiB": 22.79,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 373360,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 11.32,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "1983ec5523a46271ddc4be71898407fe3211b4ad867af0641f94f757d35f9721",
    "source_archive": "historical-recovered/host10.0.0.20-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/gguf-vllm-test/vnext-contract-runs/20260629T003817Z-gguf-dense27b-tp8",
    "source_lines": {
      "load": {
        "number": 4589,
        "text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 00:43:29 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 109.895363 seconds"
      },
      "KV": {
        "number": 5172,
        "text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 00:46:17 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB"
      },
      "tokens": {
        "number": 5175,
        "text": "(EngineCore_DP0 pid=200) INFO 06-29 00:46:17 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens"
      },
      "capacity": {
        "number": 5176,
        "text": "(EngineCore_DP0 pid=200) INFO 06-29 00:46:17 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x"
      },
      "engine": {
        "number": 64,
        "text": "(EngineCore_DP0 pid=200) INFO 06-29 00:41:15 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 22.79,
  "logged_GPU_KV_tokens": 373360,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-27/result.json",
  "retest_strict_tokens": 4508,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33758752768,
    "1": 33634103296,
    "2": 33684492288,
    "3": 33634070528,
    "4": 33684193280,
    "5": 33633898496,
    "6": 33684754432,
    "7": 33634291712
  },
  "retest_sampled_peak_VRAM_GiB": 31.440288543701172,
  "retest_memory_observation_count": 601,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_memory_log_source": "historical-recovered/fresh-row-27/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.30 · Validated Profile Results / table 1GGUF F16Qwen/Qwen3.6-27BTP8MTP off PASS3932 tokens 70.95966.437 14.844 / 43.03810K: 13.178 / 163.706Strict: 12.720 / 69.021Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 22.79 GiB · engine report373,360 grouped KV tokens · 11.32× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-28
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-28 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row28-77dcebbf74-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-28/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-28/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-28/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-28/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-28/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-28/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense"
python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-28
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-28 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 70.958 · valid historical-model sample10K 66.457 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3694,
    "content_chars": 7845,
    "content_has_think_close": false,
    "elapsed_seconds": 66.78168513812125,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T00:29:30Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.012624590657651424,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 7845,
    "qwen_answer_text_sha256": "19bb233b93af9aeff8cb300a49e6cecc8bb250160e82ad7eb86c8f9f5cab80b1",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 7845,
    "qwen_visible_text_sha256": "19bb233b93af9aeff8cb300a49e6cecc8bb250160e82ad7eb86c8f9f5cab80b1",
    "raw_has_think_close": false,
    "reasoning_chars": 11535,
    "text_chars": 19381,
    "text_sha256": "fa4ebf6dc67dc272987c639514d4a2b8e1dae60a4d10405870575fa479b48aef",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 52.78669147565961,
    "vllm_decode_tps": 69.97975998748349,
    "vllm_generation_tokens_delta": 3694.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 13.98595976922661,
    "vllm_prefill_tps": 30.816619460634502,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 55.31456704573841
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 43.153645584359765,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T00:30:13Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.0065198661759495735,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 10166,
    "text_chars": 10166,
    "text_sha256": "4969f1b7c1cd06df2bd4cf27b864e30d6b33cd319dd4dbf7cbd9d77dd8c1fc43",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 28.185694557614625,
    "vllm_decode_tps": 70.95798174892523,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.959511124528944,
    "vllm_prefill_tps": 28.811101941245532,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 46.34602645772441
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 43778,
    "content_has_think_close": false,
    "elapsed_seconds": 165.04912922065705,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T00:32:58Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008748846128582954,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 43778,
    "qwen_answer_text_sha256": "8d4913aa03499a61f358951edc53ab88f28f697c35447d590c08cf4c001f1f33",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 43778,
    "qwen_visible_text_sha256": "8d4913aa03499a61f358951edc53ab88f28f697c35447d590c08cf4c001f1f33",
    "raw_has_think_close": false,
    "reasoning_chars": 11535,
    "text_chars": 55314,
    "text_sha256": "42b70c87469625bee9338143ee5b8cd53052c0cbcb52664510fdfc9aa2ea4312",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 150.4740112675354,
    "vllm_decode_tps": 66.45665863336687,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.564758247695863,
    "vllm_prefill_tps": 29.591977612686016,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 60.588020350176016
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 27, 28, 32, 33, 34. One new sample.PASS4,508 natural tokens2K 71.02510K 66.531 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-08T22:18:52.716405342Z",
    "FinishedAt": "2026-11-08T22:40:00.260467584Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "9a55da176fe75043aa7dc1ca12883e4f952738b9ebf2616b6f0edce4cb417ed7",
  "retest_strict": "PASS",
  "retest_strict_tokens": 4508,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 4508,
      "decode_tps": 69.61732325093088,
      "prefill_seconds": 14.042354583740234,
      "client_total_seconds": 78.80580217391253,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 64.75399784836918,
        "vllm:request_prefill_time_seconds_sum": 14.042354583740234,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 4508.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
      "summary_sha256": "754f0c5eeaf987089a936b03a08af821069b38f7e14fa59e17115de3fc4d2fae"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 71.02517985930231,
      "prefill_seconds": 13.268245195038617,
      "client_total_seconds": 41.43482113070786,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 28.15902760066092,
        "vllm:request_prefill_time_seconds_sum": 13.268245195038617,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
      "summary_sha256": "aff3ec02e75684eb8bd8ecdd4936b31fc9fbb3754970d3b0a39d80e61d360495"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 66.53116656725801,
      "prefill_seconds": 12.96059922594577,
      "client_total_seconds": 163.27578648924828,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 150.3054961450398,
        "vllm:request_prefill_time_seconds_sum": 12.96059922594577,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
      "summary_sha256": "29705fa055020f3834638960e4af19073f76c5672f7f2d3c621b46c7bf61f6ef"
    }
  },
  "retest_strict_TPS": 69.61732325093088,
  "retest_strict_prefill_seconds": 14.042354583740234,
  "retest_strict_total_seconds": 78.80580217391253,
  "retest_2K_TPS": 71.02517985930231,
  "retest_2K_prefill_seconds": 13.268245195038617,
  "retest_2K_total_seconds": 41.43482113070786,
  "retest_10K_TPS": 66.53116656725801,
  "retest_10K_prefill_seconds": 12.96059922594577,
  "retest_10K_total_seconds": 163.27578648924828,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33758752768,
    "1": 33634103296,
    "2": 33684492288,
    "3": 33634070528,
    "4": 33684193280,
    "5": 33633898496,
    "6": 33684754432,
    "7": 33634291712
  },
  "retest_sampled_peak_VRAM_GiB": 31.440288543701172,
  "retest_memory_observation_count": 601,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 6.75,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_logged_context_capacity_x": 11.32,
  "retest_memory_log_source": "historical-recovered/fresh-row-27/result.json"
}
Fresh sampled VRAM: 31.440 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 22.79 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid; ai-info 10K gate cleared

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 3932,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r2.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-28.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 28,
    "original_host": "10.0.0.30",
    "logged_model_load_delta_GiB": 6.75,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 121.718155,
    "logged_available_KV_GiB": 22.79,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 373360,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 11.32,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "c954e80c7e73a96ae72d149bfefde298e35ede878ea1d50e3d8a8b4051bd22aa",
    "source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/gguf-vllm-test/vnext-contract-runs/20260629T003817Z-gguf-dense27b-tp8",
    "source_lines": {
      "load": {
        "number": 4589,
        "text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 00:43:42 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 121.718155 seconds"
      },
      "KV": {
        "number": 5172,
        "text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 00:46:32 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB"
      },
      "tokens": {
        "number": 5175,
        "text": "(EngineCore_DP0 pid=200) INFO 06-29 00:46:32 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens"
      },
      "capacity": {
        "number": 5176,
        "text": "(EngineCore_DP0 pid=200) INFO 06-29 00:46:32 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x"
      },
      "engine": {
        "number": 64,
        "text": "(EngineCore_DP0 pid=200) INFO 06-29 00:41:16 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 22.79,
  "logged_GPU_KV_tokens": 373360,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-27/result.json",
  "retest_strict_tokens": 4508,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33758752768,
    "1": 33634103296,
    "2": 33684492288,
    "3": 33634070528,
    "4": 33684193280,
    "5": 33633898496,
    "6": 33684754432,
    "7": 33634291712
  },
  "retest_sampled_peak_VRAM_GiB": 31.440288543701172,
  "retest_memory_observation_count": 601,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_memory_log_source": "historical-recovered/fresh-row-27/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.20 · Validated Profile Results / table 1HF FP16Qwen/Qwen3.6-27BTP8MTP off PASS4120 tokens 71.31666.824 13.802 / 41.85410K: 14.736 / 164.390Strict: 14.254 / 72.978Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.84 GiB (rank 0)Available KV: 22.64 GiB · engine report370,864 grouped KV tokens · 11.25× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-29
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-29 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row29-db10e5f4ad-reproduction \
  --network \
  host \
  --ipc \
  host \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-29/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-29/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-29/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-29/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-29/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-29/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/opt/local-models/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-29
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-29 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 71.157 · valid historical-model sample10K 66.643 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3234,
    "content_chars": 5935,
    "content_has_think_close": false,
    "elapsed_seconds": 60.3153067426756,
    "endpoint": "http://127.0.0.1:19078/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T00:59:25Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.013067858293652534,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8078/metrics",
    "model": "Qwen3.6-27B-HF-CausalLM",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 5935,
    "qwen_answer_text_sha256": "52e4d8f1dd1b31e6269fcaa4b1424c12f64ce1d16e326317fb8f98e0833b2856",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 5935,
    "qwen_visible_text_sha256": "52e4d8f1dd1b31e6269fcaa4b1424c12f64ce1d16e326317fb8f98e0833b2856",
    "raw_has_think_close": false,
    "reasoning_chars": 9949,
    "text_chars": 15885,
    "text_sha256": "da8b23f5c947a638d508077cbc6787afe6aeb529f98d038fbbee08dccc0c1c1e",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 45.86109949275851,
    "vllm_decode_tps": 70.51728012998576,
    "vllm_generation_tokens_delta": 3234.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.444526282139122,
    "vllm_prefill_tps": 29.838292484049,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 53.61823017492523
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 42.0066326521337,
    "endpoint": "http://127.0.0.1:19078/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T01:00:07Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.009246834553778172,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8078/metrics",
    "model": "Qwen3.6-27B-HF-CausalLM",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9629,
    "text_chars": 9629,
    "text_sha256": "166f1a5cf5e0080402ebf7099d53ca119066ece8cb52882a4fe8cd5ec401156f",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 28.1066905381158,
    "vllm_decode_tps": 71.15743482100027,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 13.89147546235472,
    "vllm_prefill_tps": 31.02622188463643,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 47.61152879266582
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 40547,
    "content_has_think_close": false,
    "elapsed_seconds": 164.41756441816688,
    "endpoint": "http://127.0.0.1:19078/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T01:02:51Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008683929219841957,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8078/metrics",
    "model": "Qwen3.6-27B-HF-CausalLM",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 40547,
    "qwen_answer_text_sha256": "6b37071096f0f0ba680c6493ff5fd736b9a43e9e66fcc01a39b831b99863482a",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 40547,
    "qwen_visible_text_sha256": "6b37071096f0f0ba680c6493ff5fd736b9a43e9e66fcc01a39b831b99863482a",
    "raw_has_think_close": false,
    "reasoning_chars": 9949,
    "text_chars": 50497,
    "text_sha256": "10fcf1e7cefef0bd9ff1f9498f82bd0f5694e9329aceaffff40ea4d827635d54",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 150.05429869424552,
    "vllm_decode_tps": 66.64254264635402,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.353619427420199,
    "vllm_prefill_tps": 30.027269580287623,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 60.82075254786512
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 29. One new sample.PASS3,894 natural tokens2K 71.36010K 66.825 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-08T22:58:14.831580175Z",
    "FinishedAt": "2026-11-08T23:16:33.728315227Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "1d2c3350fafbf79cf6d1aca4a68e17a670c7f2e384ce7589e2d7f8bce876d262",
  "retest_strict": "PASS",
  "retest_strict_tokens": 3894,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 3894,
      "decode_tps": 70.29589397473099,
      "prefill_seconds": 14.098794215358794,
      "client_total_seconds": 69.50212777592242,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 3894.0,
        "vllm:request_prefill_time_seconds_sum": 14.098794215358794,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 55.39441608637571
      },
      "request_sha256": "d6b5acc7f81647ef18db224f701221c1f68a009b4548f6849cf87c3c8e419a9a",
      "summary_sha256": "7edb828e4d7722b0c1fd383897c084e95cdb225e7c5a037b23b28761ada4bc47"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 71.35954591433956,
      "prefill_seconds": 13.803319579921663,
      "client_total_seconds": 41.83979881927371,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_prefill_time_seconds_sum": 13.803319579921663,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 28.02708417456597
      },
      "request_sha256": "3d137c416021d94f838f557d4f42b0fbfe62d81a216b8c0a16b65a1f529aca97",
      "summary_sha256": "35b66db175a416934d4cd1895d171f23baa26dd6de6eeecc1ae3c60ef2905b15"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 66.82477816892438,
      "prefill_seconds": 14.576330919750035,
      "client_total_seconds": 164.2314986139536,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_prefill_time_seconds_sum": 14.576330919750035,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 149.64509084820747
      },
      "request_sha256": "d03cae7830251b8462de5cfc7ed217442c2137d550fefb2baf4f6c92b612c39a",
      "summary_sha256": "09a50ce3034dc4380f7c301323163264cf522177a86be5b4be1fa6e0bdcbc20c"
    }
  },
  "retest_strict_TPS": 70.29589397473099,
  "retest_strict_prefill_seconds": 14.098794215358794,
  "retest_strict_total_seconds": 69.50212777592242,
  "retest_2K_TPS": 71.35954591433956,
  "retest_2K_prefill_seconds": 13.803319579921663,
  "retest_2K_total_seconds": 41.83979881927371,
  "retest_10K_TPS": 66.82477816892438,
  "retest_10K_prefill_seconds": 14.576330919750035,
  "retest_10K_total_seconds": 164.2314986139536,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33791266816,
    "1": 33666322432,
    "2": 33716637696,
    "3": 33666342912,
    "4": 33716674560,
    "5": 33666342912,
    "6": 33716662272,
    "7": 33666334720
  },
  "retest_sampled_peak_VRAM_GiB": 31.470569610595703,
  "retest_memory_observation_count": 517,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 723,
        "text": "2026-11-08T22:59:57.180783513Z (Worker pid=327) (Worker_TP0 pid=327) INFO 11-08 22:59:57 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 9.563808 seconds",
        "values": [
          "6.84"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 1271,
        "text": "2026-11-08T23:01:39.506595611Z (Worker pid=327) (Worker_TP0 pid=327) INFO 11-08 23:01:39 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
        "values": [
          "22.64"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 1284,
        "text": "2026-11-08T23:01:40.213405262Z (EngineCore_DP0 pid=273) INFO 11-08 23:01:40 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
        "values": [
          "370,864"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 1285,
        "text": "2026-11-08T23:01:40.213475072Z (EngineCore_DP0 pid=273) INFO 11-08 23:01:40 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
        "values": [
          "131,072",
          "11.25"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 6.84,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 22.64,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 370864,
  "retest_logged_context_capacity_x": 11.25,
  "retest_memory_log_source": "historical-recovered/fresh-row-29/result.json"
}
Fresh sampled VRAM: 31.471 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 22.64 GiB minimum/worker; rounded engine report.
Evidence and limits

fresh-cache release-band control

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 4120,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r3.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-29.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 29,
    "original_host": "10.0.0.20",
    "logged_model_load_delta_GiB": 6.84,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 7.221931,
    "logged_available_KV_GiB": 22.64,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 370864,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 11.25,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "7b29916dd30992ee4d28b5e1b69537e1adfb8405d4bb0bc254c1314d926e1efb",
    "source_archive": "historical-recovered/host10.0.0.20-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/gguf-vllm-test/vnext-package-runs/hf_dense27b_dot20_releasebundle_freshcache_20260628T203743Z/launch",
    "source_lines": {
      "load": {
        "number": 723,
        "text": "(Worker pid=327) (Worker_TP0 pid=327) INFO 06-28 20:39:17 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 7.221931 seconds"
      },
      "KV": {
        "number": 1273,
        "text": "(Worker pid=327) (Worker_TP0 pid=327) INFO 06-28 20:40:54 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB"
      },
      "tokens": {
        "number": 1284,
        "text": "(EngineCore_DP0 pid=273) INFO 06-28 20:40:55 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens"
      },
      "capacity": {
        "number": 1285,
        "text": "(EngineCore_DP0 pid=273) INFO 06-28 20:40:55 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x"
      },
      "engine": {
        "number": 57,
        "text": "(EngineCore_DP0 pid=273) INFO 06-28 20:38:44 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9', speculative_config=None, tokenizer='/opt/local-models/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-HF-CausalLM, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 22.64,
  "logged_GPU_KV_tokens": 370864,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-29/result.json",
  "retest_strict_tokens": 3894,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33791266816,
    "1": 33666322432,
    "2": 33716637696,
    "3": 33666342912,
    "4": 33716674560,
    "5": 33666342912,
    "6": 33716662272,
    "7": 33666334720
  },
  "retest_sampled_peak_VRAM_GiB": 31.470569610595703,
  "retest_memory_observation_count": 517,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 723,
        "text": "2026-11-08T22:59:57.180783513Z (Worker pid=327) (Worker_TP0 pid=327) INFO 11-08 22:59:57 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 9.563808 seconds",
        "values": [
          "6.84"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 1271,
        "text": "2026-11-08T23:01:39.506595611Z (Worker pid=327) (Worker_TP0 pid=327) INFO 11-08 23:01:39 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
        "values": [
          "22.64"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 1284,
        "text": "2026-11-08T23:01:40.213405262Z (EngineCore_DP0 pid=273) INFO 11-08 23:01:40 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
        "values": [
          "370,864"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 1285,
        "text": "2026-11-08T23:01:40.213475072Z (EngineCore_DP0 pid=273) INFO 11-08 23:01:40 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
        "values": [
          "131,072",
          "11.25"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 22.64,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 370864,
  "retest_memory_log_source": "historical-recovered/fresh-row-29/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.20 · Validated Profile Results / table 1HF FP16Qwen/Qwen3.6-35B-A3BTP8MTP off PASS2978 tokens 115.549108.805 0.811 / 18.12810K: 0.803 / 92.720Strict: 0.808 / 26.704Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 8.88 GiB (rank 0)Available KV: 19.91 GiB · engine report521,696 grouped KV tokens · 15.82× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-30
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-30 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row30-aaf874ca19-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-30/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-30/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-30/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-30/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-30/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-30/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-moe,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/995ad96eacd98c81ed38be0c5b274b04031597b0,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-moe "$MODEL_DIR/qwen36-moe"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-30
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-30 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 115.297 · valid historical-model sample10K 108.623 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 4123,
    "content_chars": 8534,
    "content_has_think_close": false,
    "elapsed_seconds": 37.004668129608035,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T01:39:58Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.008319173008203506,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 8534,
    "qwen_answer_text_sha256": "ba0a6d00b3670679073d7eb1a571b6b29db4a839be80a8c8f3d86afb09ff8220",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 8534,
    "qwen_visible_text_sha256": "ba0a6d00b3670679073d7eb1a571b6b29db4a839be80a8c8f3d86afb09ff8220",
    "raw_has_think_close": false,
    "reasoning_chars": 13258,
    "text_chars": 21793,
    "text_sha256": "28fd44aa56787b93626bd24f8604c3893ad729899b4d70e8490b500bb3564071",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 36.186177032999694,
    "vllm_decode_tps": 113.93853504447466,
    "vllm_generation_tokens_delta": 4123.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8094438994303346,
    "vllm_prefill_tps": 532.4643255738001,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 111.41837525901552
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 18.165475289337337,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T01:40:16Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.00866091437637806,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 10313,
    "text_chars": 10313,
    "text_sha256": "6af2a531c6bc5cacd9e2bcf5903c638455cd83162d39d986b0f3f6116ea655fc",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 17.346558371558785,
    "vllm_decode_tps": 115.2966460066901,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8107803724706173,
    "vllm_prefill_tps": 531.5866227578412,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 110.0989634537087
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 42964,
    "content_has_think_close": false,
    "elapsed_seconds": 92.88607190269977,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T01:41:49Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.006269831210374832,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 42964,
    "qwen_answer_text_sha256": "1b3fc40a96673d78a8c2338d0b94b336673bd4d367e7b03132e9b010a80191fc",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 42964,
    "qwen_visible_text_sha256": "1b3fc40a96673d78a8c2338d0b94b336673bd4d367e7b03132e9b010a80191fc",
    "raw_has_think_close": false,
    "reasoning_chars": 13258,
    "text_chars": 56223,
    "text_sha256": "d50a79daf4fa5658877f56d67da4aab8062cd4ffee794376866171c17fb4be48",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 92.06168713700026,
    "vllm_decode_tps": 108.62281923118186,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8135625869035721,
    "vllm_prefill_tps": 529.7687073349705,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 107.65876729586782
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 30, 31. One new sample.PASS3,315 natural tokens2K 116.30310K 109.479 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-08T23:18:06.809781202Z",
    "FinishedAt": "2026-11-08T23:38:48.306386482Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 174,
  "retest_historical_manifest_sha256": "9a2772d75889092782cc9a65e91fb732b333c30fc7469365a71fbf2c8f50798e",
  "retest_strict": "PASS",
  "retest_strict_tokens": 3315,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 3315,
      "decode_tps": 115.49001974098105,
      "prefill_seconds": 0.8086845744401217,
      "client_total_seconds": 29.521306382492185,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 3315.0,
        "vllm:request_prefill_time_seconds_sum": 0.8086845744401217,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 28.703778970986605,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "e31c712c0771301f2e4d592bc966448d3143f2660e1b4cab33a91704f19e9370",
      "summary_sha256": "6de45c7cc013914550cba9d9e4fb391546d47711db7c87c3248b42dcf7698c6d"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 116.30302059984058,
      "prefill_seconds": 0.81096709985286,
      "client_total_seconds": 18.01519708149135,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_prefill_time_seconds_sum": 0.81096709985286,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 17.19645792245865,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "bca01af4a8338cd6274c1c9e0b281d8e59066ebae2ece3e3a636620066da3a03",
      "summary_sha256": "0acf3601e108fbbfd1a891c8d6a16ccbe40e590b194451e109627d82f3c5e96d"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 109.47943821033532,
      "prefill_seconds": 0.8086706819012761,
      "client_total_seconds": 92.15940057393163,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_prefill_time_seconds_sum": 0.8086706819012761,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 91.34135289210826,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "46fd7808353544eb86016bdd1746bafb67def6864a3f5352647b66cb9149f95a",
      "summary_sha256": "a8ba495779711e5602e6164c8bec0f7728c656bfa02a9661baafa8ef58f71683"
    }
  },
  "retest_strict_TPS": 115.49001974098105,
  "retest_strict_prefill_seconds": 0.8086845744401217,
  "retest_strict_total_seconds": 29.521306382492185,
  "retest_2K_TPS": 116.30302059984058,
  "retest_2K_prefill_seconds": 0.81096709985286,
  "retest_2K_total_seconds": 18.01519708149135,
  "retest_10K_TPS": 109.47943821033532,
  "retest_10K_prefill_seconds": 0.8086706819012761,
  "retest_10K_total_seconds": 92.15940057393163,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 34334515200,
    "1": 34314481664,
    "2": 34314469376,
    "3": 34314502144,
    "4": 34314551296,
    "5": 34314530816,
    "6": 34314477568,
    "7": 34318667776
  },
  "retest_sampled_peak_VRAM_GiB": 31.97650909423828,
  "retest_memory_observation_count": 588,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 193,
        "text": "2026-11-08T23:23:02.194194967Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:23:02 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 193.073592 seconds",
        "values": [
          "8.88"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 311,
        "text": "2026-11-08T23:24:25.909888257Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:24:25 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
        "values": [
          "19.91"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 315,
        "text": "2026-11-08T23:24:26.013151548Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
        "values": [
          "521,696"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 316,
        "text": "2026-11-08T23:24:26.013251748Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
        "values": [
          "131,072",
          "15.82"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 8.88,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 19.91,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 521696,
  "retest_logged_context_capacity_x": 15.82,
  "retest_memory_log_source": "historical-recovered/fresh-row-30/result.json"
}
Fresh sampled VRAM: 31.977 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 19.91 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 2978,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "995ad96eacd98c81ed38be0c5b274b04031597b0",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r3.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-30.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 30,
    "original_host": "10.0.0.20",
    "logged_model_load_delta_GiB": 8.88,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 189.378052,
    "logged_available_KV_GiB": 19.91,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 521696,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 15.82,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "b89b1bada4eac6d54c85c9f77c9c1aa0aee0fb9ade0f826fbeb8e919bf371f4c",
    "source_archive": "historical-recovered/host10.0.0.20-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/gguf-vllm-test/vnext-contract-runs/20260629T003817Z-hf-moe35b-tp8",
    "source_lines": {
      "load": {
        "number": 193,
        "text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-29 01:42:17 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 189.378052 seconds"
      },
      "KV": {
        "number": 308,
        "text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-29 01:43:40 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB"
      },
      "tokens": {
        "number": 315,
        "text": "(EngineCore_DP0 pid=171) INFO 06-29 01:43:41 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens"
      },
      "capacity": {
        "number": 316,
        "text": "(EngineCore_DP0 pid=171) INFO 06-29 01:43:41 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x"
      },
      "engine": {
        "number": 33,
        "text": "(EngineCore_DP0 pid=171) INFO 06-29 01:38:37 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/995ad96eacd98c81ed38be0c5b274b04031597b0', speculative_config=None, tokenizer='/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/995ad96eacd98c81ed38be0c5b274b04031597b0', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.6-35B-A3B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 19.91,
  "logged_GPU_KV_tokens": 521696,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-30/result.json",
  "retest_strict_tokens": 3315,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 34334515200,
    "1": 34314481664,
    "2": 34314469376,
    "3": 34314502144,
    "4": 34314551296,
    "5": 34314530816,
    "6": 34314477568,
    "7": 34318667776
  },
  "retest_sampled_peak_VRAM_GiB": 31.97650909423828,
  "retest_memory_observation_count": 588,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 193,
        "text": "2026-11-08T23:23:02.194194967Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:23:02 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 193.073592 seconds",
        "values": [
          "8.88"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 311,
        "text": "2026-11-08T23:24:25.909888257Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:24:25 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
        "values": [
          "19.91"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 315,
        "text": "2026-11-08T23:24:26.013151548Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
        "values": [
          "521,696"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 316,
        "text": "2026-11-08T23:24:26.013251748Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
        "values": [
          "131,072",
          "15.82"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 19.91,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 521696,
  "retest_memory_log_source": "historical-recovered/fresh-row-30/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.30 · Validated Profile Results / table 1HF FP16Qwen/Qwen3.6-35B-A3BTP8MTP off PASS3103 tokens 115.532108.673 0.809 / 18.12810K: 0.807 / 92.836Strict: 0.808 / 27.870Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 8.88 GiB (rank 0)Available KV: 19.91 GiB · engine report521,696 grouped KV tokens · 15.82× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-31
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-31 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row31-90a75956f3-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-31/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-31/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-31/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-31/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-31/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-31/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-moe,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/995ad96eacd98c81ed38be0c5b274b04031597b0,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-moe "$MODEL_DIR/qwen36-moe"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-31
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-31 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 115.825 · valid historical-model sample10K 108.889 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3074,
    "content_chars": 6062,
    "content_has_think_close": false,
    "elapsed_seconds": 27.538480462506413,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T02:18:25Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.01051213126629591,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 6062,
    "qwen_answer_text_sha256": "1da5eaf2d1a2df7954628036116f036612a2e11e62d19ab301d81891297acee2",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 6062,
    "qwen_visible_text_sha256": "1da5eaf2d1a2df7954628036116f036612a2e11e62d19ab301d81891297acee2",
    "raw_has_think_close": false,
    "reasoning_chars": 9805,
    "text_chars": 15868,
    "text_sha256": "c3953bbf14c8bf2a03c07aa8de91765d98935a3422c8b9473e825d5678ae9947",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 26.723272138275206,
    "vllm_decode_tps": 115.03082347453893,
    "vllm_generation_tokens_delta": 3074.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8065544962882996,
    "vllm_prefill_tps": 534.3718273017238,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 111.62562161645937
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 200,
    "content_has_think_close": false,
    "elapsed_seconds": 18.08423571381718,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T02:18:44Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.006676186807453632,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 200,
    "qwen_answer_text_sha256": "2f213ad86009aacf37f15220812d39d2ddf2e8288ddb317a100ac0d32c11b69c",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 200,
    "qwen_visible_text_sha256": "2f213ad86009aacf37f15220812d39d2ddf2e8288ddb317a100ac0d32c11b69c",
    "raw_has_think_close": false,
    "reasoning_chars": 9805,
    "text_chars": 10006,
    "text_sha256": "622b7aa36324f0b39addb94e14df1edc548b4e7e33e4a6780113bca38cdf628a",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 17.267360970377922,
    "vllm_decode_tps": 115.82545841434546,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8081611143425107,
    "vllm_prefill_tps": 533.3095002357857,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 110.59355958692294
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 48347,
    "content_has_think_close": false,
    "elapsed_seconds": 92.6585035957396,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T02:20:16Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.005911114625632763,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 48347,
    "qwen_answer_text_sha256": "ae73d5c7cbbdccaf573352ad53b4ff705746df31b9b2da9b40e640fc29123613",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 48347,
    "qwen_visible_text_sha256": "ae73d5c7cbbdccaf573352ad53b4ff705746df31b9b2da9b40e640fc29123613",
    "raw_has_think_close": false,
    "reasoning_chars": 9805,
    "text_chars": 58153,
    "text_sha256": "0dbb2aae366ae7556a2270143f96ce9c0efbad0938d8f1b734817ccb8d9939b1",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 91.83650601096451,
    "vllm_decode_tps": 108.88916003409454,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8104435317218304,
    "vllm_prefill_tps": 531.8075635501928,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 107.9231760921703
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 30, 31. One new sample.PASS3,315 natural tokens2K 116.30310K 109.479 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-08T23:18:06.809781202Z",
    "FinishedAt": "2026-11-08T23:38:48.306386482Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 174,
  "retest_historical_manifest_sha256": "9a2772d75889092782cc9a65e91fb732b333c30fc7469365a71fbf2c8f50798e",
  "retest_strict": "PASS",
  "retest_strict_tokens": 3315,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 3315,
      "decode_tps": 115.49001974098105,
      "prefill_seconds": 0.8086845744401217,
      "client_total_seconds": 29.521306382492185,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 3315.0,
        "vllm:request_prefill_time_seconds_sum": 0.8086845744401217,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 28.703778970986605,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "e31c712c0771301f2e4d592bc966448d3143f2660e1b4cab33a91704f19e9370",
      "summary_sha256": "6de45c7cc013914550cba9d9e4fb391546d47711db7c87c3248b42dcf7698c6d"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 116.30302059984058,
      "prefill_seconds": 0.81096709985286,
      "client_total_seconds": 18.01519708149135,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_prefill_time_seconds_sum": 0.81096709985286,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 17.19645792245865,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "bca01af4a8338cd6274c1c9e0b281d8e59066ebae2ece3e3a636620066da3a03",
      "summary_sha256": "0acf3601e108fbbfd1a891c8d6a16ccbe40e590b194451e109627d82f3c5e96d"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 109.47943821033532,
      "prefill_seconds": 0.8086706819012761,
      "client_total_seconds": 92.15940057393163,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_prefill_time_seconds_sum": 0.8086706819012761,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 91.34135289210826,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "46fd7808353544eb86016bdd1746bafb67def6864a3f5352647b66cb9149f95a",
      "summary_sha256": "a8ba495779711e5602e6164c8bec0f7728c656bfa02a9661baafa8ef58f71683"
    }
  },
  "retest_strict_TPS": 115.49001974098105,
  "retest_strict_prefill_seconds": 0.8086845744401217,
  "retest_strict_total_seconds": 29.521306382492185,
  "retest_2K_TPS": 116.30302059984058,
  "retest_2K_prefill_seconds": 0.81096709985286,
  "retest_2K_total_seconds": 18.01519708149135,
  "retest_10K_TPS": 109.47943821033532,
  "retest_10K_prefill_seconds": 0.8086706819012761,
  "retest_10K_total_seconds": 92.15940057393163,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 34334515200,
    "1": 34314481664,
    "2": 34314469376,
    "3": 34314502144,
    "4": 34314551296,
    "5": 34314530816,
    "6": 34314477568,
    "7": 34318667776
  },
  "retest_sampled_peak_VRAM_GiB": 31.97650909423828,
  "retest_memory_observation_count": 588,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 193,
        "text": "2026-11-08T23:23:02.194194967Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:23:02 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 193.073592 seconds",
        "values": [
          "8.88"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 311,
        "text": "2026-11-08T23:24:25.909888257Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:24:25 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
        "values": [
          "19.91"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 315,
        "text": "2026-11-08T23:24:26.013151548Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
        "values": [
          "521,696"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 316,
        "text": "2026-11-08T23:24:26.013251748Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
        "values": [
          "131,072",
          "15.82"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 8.88,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 19.91,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 521696,
  "retest_logged_context_capacity_x": 15.82,
  "retest_memory_log_source": "historical-recovered/fresh-row-30/result.json"
}
Fresh sampled VRAM: 31.977 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 19.91 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 3103,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "995ad96eacd98c81ed38be0c5b274b04031597b0",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r3.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-31.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 31,
    "original_host": "10.0.0.30",
    "logged_model_load_delta_GiB": 8.88,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 191.364916,
    "logged_available_KV_GiB": 19.91,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 521696,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 15.82,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "72649d16114bbe7ecda6325c3b0eff1b7e054304b6dcfb050476b2307b2ee7ba",
    "source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/gguf-vllm-test/vnext-contract-runs/20260629T003817Z-hf-moe35b-tp8",
    "source_lines": {
      "load": {
        "number": 193,
        "text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-29 01:42:22 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 191.364916 seconds"
      },
      "KV": {
        "number": 308,
        "text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-29 01:43:44 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB"
      },
      "tokens": {
        "number": 315,
        "text": "(EngineCore_DP0 pid=171) INFO 06-29 01:43:45 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens"
      },
      "capacity": {
        "number": 316,
        "text": "(EngineCore_DP0 pid=171) INFO 06-29 01:43:45 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x"
      },
      "engine": {
        "number": 33,
        "text": "(EngineCore_DP0 pid=171) INFO 06-29 01:38:36 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/995ad96eacd98c81ed38be0c5b274b04031597b0', speculative_config=None, tokenizer='/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/995ad96eacd98c81ed38be0c5b274b04031597b0', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.6-35B-A3B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 19.91,
  "logged_GPU_KV_tokens": 521696,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-30/result.json",
  "retest_strict_tokens": 3315,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 34334515200,
    "1": 34314481664,
    "2": 34314469376,
    "3": 34314502144,
    "4": 34314551296,
    "5": 34314530816,
    "6": 34314477568,
    "7": 34318667776
  },
  "retest_sampled_peak_VRAM_GiB": 31.97650909423828,
  "retest_memory_observation_count": 588,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 193,
        "text": "2026-11-08T23:23:02.194194967Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:23:02 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 193.073592 seconds",
        "values": [
          "8.88"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 311,
        "text": "2026-11-08T23:24:25.909888257Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:24:25 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
        "values": [
          "19.91"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 315,
        "text": "2026-11-08T23:24:26.013151548Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
        "values": [
          "521,696"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 316,
        "text": "2026-11-08T23:24:26.013251748Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
        "values": [
          "131,072",
          "15.82"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 19.91,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 521696,
  "retest_memory_log_source": "historical-recovered/fresh-row-30/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.30 · Public-Staged POSIX-Runner Reproduction Audit / table 1GGUF F16Qwen/Qwen3.6-27BTP8MTP off PASS3636 tokens 70.93566.403 12.073 / 40.27610K: 12.556 / 163.163Strict: 13.232 / 65.194Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 23.15 GiB · engine report379,184 grouped KV tokens · 11.50× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-32
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-32 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row32-8e5b34f5cf-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-32/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-32/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-32/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-32/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-32/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-32/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense"
python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-32
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-32 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 70.912 · valid historical-model sample10K 66.452 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3687,
    "content_chars": 5329,
    "content_has_think_close": false,
    "elapsed_seconds": 65.07993041258305,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T02:56:48Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.012216160073876381,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 5329,
    "qwen_answer_text_sha256": "3a7a6b602375e1ddc6c8e18ad6a2710bf65108182452f41ace63f1ab0f80812d",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 5329,
    "qwen_visible_text_sha256": "3a7a6b602375e1ddc6c8e18ad6a2710bf65108182452f41ace63f1ab0f80812d",
    "raw_has_think_close": false,
    "reasoning_chars": 12365,
    "text_chars": 17695,
    "text_sha256": "5b3f450c991bceef669328df77eae3d6b223ed3d3ba745f87d3c95e9ee2d591e",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 52.69167762435973,
    "vllm_decode_tps": 69.973099476633,
    "vllm_generation_tokens_delta": 3687.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 12.377928133122623,
    "vllm_prefill_tps": 34.8200438203118,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 56.65341030062206
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 42.01967437751591,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T02:57:30Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008349246345460415,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9610,
    "text_chars": 9610,
    "text_sha256": "b6e619acbecd7b98e7bd3fafcd526951e3260b60cf5909be4ad3a5f325565bff",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 28.204140285030007,
    "vllm_decode_tps": 70.91157467620262,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 13.807229696772993,
    "vllm_prefill_tps": 31.215530520270317,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 47.59675151290962
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 45133,
    "content_has_think_close": false,
    "elapsed_seconds": 165.0049143144861,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T03:00:15Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.009004133753478527,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 45133,
    "qwen_answer_text_sha256": "79555b8c674ed6168687a63258550a0f08ef0033b17b0f5b24d96f2e305de681",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 45133,
    "qwen_visible_text_sha256": "79555b8c674ed6168687a63258550a0f08ef0033b17b0f5b24d96f2e305de681",
    "raw_has_think_close": false,
    "reasoning_chars": 12365,
    "text_chars": 57499,
    "text_sha256": "54c178bd283aaf13a7da9ffef6cbdb53f7833574997cb0e5421dce9f6dd6cf24",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 150.48369926866144,
    "vllm_decode_tps": 66.45238021526045,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.511613338254392,
    "vllm_prefill_tps": 29.70035033002369,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 60.60425558562944
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 27, 28, 32, 33, 34. One new sample.PASS4,508 natural tokens2K 71.02510K 66.531 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-08T22:18:52.716405342Z",
    "FinishedAt": "2026-11-08T22:40:00.260467584Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "9a55da176fe75043aa7dc1ca12883e4f952738b9ebf2616b6f0edce4cb417ed7",
  "retest_strict": "PASS",
  "retest_strict_tokens": 4508,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 4508,
      "decode_tps": 69.61732325093088,
      "prefill_seconds": 14.042354583740234,
      "client_total_seconds": 78.80580217391253,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 64.75399784836918,
        "vllm:request_prefill_time_seconds_sum": 14.042354583740234,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 4508.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
      "summary_sha256": "754f0c5eeaf987089a936b03a08af821069b38f7e14fa59e17115de3fc4d2fae"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 71.02517985930231,
      "prefill_seconds": 13.268245195038617,
      "client_total_seconds": 41.43482113070786,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 28.15902760066092,
        "vllm:request_prefill_time_seconds_sum": 13.268245195038617,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
      "summary_sha256": "aff3ec02e75684eb8bd8ecdd4936b31fc9fbb3754970d3b0a39d80e61d360495"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 66.53116656725801,
      "prefill_seconds": 12.96059922594577,
      "client_total_seconds": 163.27578648924828,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 150.3054961450398,
        "vllm:request_prefill_time_seconds_sum": 12.96059922594577,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
      "summary_sha256": "29705fa055020f3834638960e4af19073f76c5672f7f2d3c621b46c7bf61f6ef"
    }
  },
  "retest_strict_TPS": 69.61732325093088,
  "retest_strict_prefill_seconds": 14.042354583740234,
  "retest_strict_total_seconds": 78.80580217391253,
  "retest_2K_TPS": 71.02517985930231,
  "retest_2K_prefill_seconds": 13.268245195038617,
  "retest_2K_total_seconds": 41.43482113070786,
  "retest_10K_TPS": 66.53116656725801,
  "retest_10K_prefill_seconds": 12.96059922594577,
  "retest_10K_total_seconds": 163.27578648924828,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33758752768,
    "1": 33634103296,
    "2": 33684492288,
    "3": 33634070528,
    "4": 33684193280,
    "5": 33633898496,
    "6": 33684754432,
    "7": 33634291712
  },
  "retest_sampled_peak_VRAM_GiB": 31.440288543701172,
  "retest_memory_observation_count": 601,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 6.75,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_logged_context_capacity_x": 11.32,
  "retest_memory_log_source": "historical-recovered/fresh-row-27/result.json"
}
Fresh sampled VRAM: 31.440 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 22.79 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid; ai-info 10K gate cleared

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 3636,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r2.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-32.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 32,
    "original_host": "10.0.0.30",
    "logged_model_load_delta_GiB": 6.75,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 109.957728,
    "logged_available_KV_GiB": 23.15,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 379184,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 11.5,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "759f71b82d879600a80aa25cd5340f4bcf248dd73784a58ffaa097e640117f39",
    "source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/vnext-release-repro/localaiservers/qwen36-gfx906/vnext-launch-runs/repro-gguf-dense27b-tp8",
    "source_lines": {
      "load": {
        "number": 4589,
        "text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 12:36:09 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 109.957728 seconds"
      },
      "KV": {
        "number": 5245,
        "text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 12:37:30 [gpu_worker.py:453] Available KV cache memory: 23.15 GiB"
      },
      "tokens": {
        "number": 5252,
        "text": "(EngineCore_DP0 pid=200) INFO 06-29 12:37:31 [kv_cache_utils.py:1314] GPU KV cache size: 379,184 tokens"
      },
      "capacity": {
        "number": 5253,
        "text": "(EngineCore_DP0 pid=200) INFO 06-29 12:37:31 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.50x"
      },
      "engine": {
        "number": 64,
        "text": "(EngineCore_DP0 pid=200) INFO 06-29 12:33:52 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 23.15,
  "logged_GPU_KV_tokens": 379184,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-27/result.json",
  "retest_strict_tokens": 4508,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33758752768,
    "1": 33634103296,
    "2": 33684492288,
    "3": 33634070528,
    "4": 33684193280,
    "5": 33633898496,
    "6": 33684754432,
    "7": 33634291712
  },
  "retest_sampled_peak_VRAM_GiB": 31.440288543701172,
  "retest_memory_observation_count": 601,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_memory_log_source": "historical-recovered/fresh-row-27/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.30 · Clean-Room GGUF Package-Path Reproduction Audit / table 1GGUF F16Qwen/Qwen3.6-27BTP8MTP off PASS4210 tokens 70.83766.309 11.616 / 39.85710K: 13.530 / 164.349Strict: 14.289 / 74.803Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 22.79 GiB · engine report373,360 grouped KV tokens · 11.32× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-33
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-33 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row33-6830e5d368-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-33/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-33/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-33/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-33/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-33/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-33/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense"
python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-33
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-33 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 70.883 · valid historical-model sample10K 66.394 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3742,
    "content_chars": 6468,
    "content_has_think_close": false,
    "elapsed_seconds": 66.08779009897262,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T03:34:15Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.012077721767127514,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 6468,
    "qwen_answer_text_sha256": "a3e745fc71d97304f1caba5a41e98d3560a4affeb33e477aa907ba6e5fef5c87",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 6468,
    "qwen_visible_text_sha256": "a3e745fc71d97304f1caba5a41e98d3560a4affeb33e477aa907ba6e5fef5c87",
    "raw_has_think_close": false,
    "reasoning_chars": 12127,
    "text_chars": 18596,
    "text_sha256": "9a94117e6de620eee1f2549f35ad6e94911474e61cd6dda14a52b9981ae2d03e",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 53.53867237456143,
    "vllm_decode_tps": 69.89340291856001,
    "vllm_generation_tokens_delta": 3742.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 12.540770625695586,
    "vllm_prefill_tps": 34.36790392425299,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 56.62165423289244
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 41.21531672030687,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T03:34:57Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.006134975701570511,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9940,
    "text_chars": 9940,
    "text_sha256": "f4724c0e635226e325d2779aeb7bb6a3d34abdbcbbbd9d5f951f8b11442e0e74",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 28.215577918104827,
    "vllm_decode_tps": 70.88282954206933,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 12.991800364106894,
    "vllm_prefill_tps": 33.17477084936939,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 48.525649179704004
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 44552,
    "content_has_think_close": false,
    "elapsed_seconds": 164.95538353640586,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T03:37:42Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008456559851765633,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 44552,
    "qwen_answer_text_sha256": "8c459b3f86193f648e1132dd7bf503eda77fc453896ca4d7f7e99d9c81e92cfd",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 44552,
    "qwen_visible_text_sha256": "8c459b3f86193f648e1132dd7bf503eda77fc453896ca4d7f7e99d9c81e92cfd",
    "raw_has_think_close": false,
    "reasoning_chars": 12127,
    "text_chars": 56680,
    "text_sha256": "1d1b28e8252e3f6248432dbef2a6bc6a6421cef72bf73d7a6f2cfa02605c97fe",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 150.61556784249842,
    "vllm_decode_tps": 66.39419910734055,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.329687963239849,
    "vllm_prefill_tps": 30.077416975558044,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 60.62245308770409
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 27, 28, 32, 33, 34. One new sample.PASS4,508 natural tokens2K 71.02510K 66.531 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-08T22:18:52.716405342Z",
    "FinishedAt": "2026-11-08T22:40:00.260467584Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "9a55da176fe75043aa7dc1ca12883e4f952738b9ebf2616b6f0edce4cb417ed7",
  "retest_strict": "PASS",
  "retest_strict_tokens": 4508,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 4508,
      "decode_tps": 69.61732325093088,
      "prefill_seconds": 14.042354583740234,
      "client_total_seconds": 78.80580217391253,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 64.75399784836918,
        "vllm:request_prefill_time_seconds_sum": 14.042354583740234,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 4508.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
      "summary_sha256": "754f0c5eeaf987089a936b03a08af821069b38f7e14fa59e17115de3fc4d2fae"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 71.02517985930231,
      "prefill_seconds": 13.268245195038617,
      "client_total_seconds": 41.43482113070786,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 28.15902760066092,
        "vllm:request_prefill_time_seconds_sum": 13.268245195038617,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
      "summary_sha256": "aff3ec02e75684eb8bd8ecdd4936b31fc9fbb3754970d3b0a39d80e61d360495"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 66.53116656725801,
      "prefill_seconds": 12.96059922594577,
      "client_total_seconds": 163.27578648924828,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 150.3054961450398,
        "vllm:request_prefill_time_seconds_sum": 12.96059922594577,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
      "summary_sha256": "29705fa055020f3834638960e4af19073f76c5672f7f2d3c621b46c7bf61f6ef"
    }
  },
  "retest_strict_TPS": 69.61732325093088,
  "retest_strict_prefill_seconds": 14.042354583740234,
  "retest_strict_total_seconds": 78.80580217391253,
  "retest_2K_TPS": 71.02517985930231,
  "retest_2K_prefill_seconds": 13.268245195038617,
  "retest_2K_total_seconds": 41.43482113070786,
  "retest_10K_TPS": 66.53116656725801,
  "retest_10K_prefill_seconds": 12.96059922594577,
  "retest_10K_total_seconds": 163.27578648924828,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33758752768,
    "1": 33634103296,
    "2": 33684492288,
    "3": 33634070528,
    "4": 33684193280,
    "5": 33633898496,
    "6": 33684754432,
    "7": 33634291712
  },
  "retest_sampled_peak_VRAM_GiB": 31.440288543701172,
  "retest_memory_observation_count": 601,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 6.75,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_logged_context_capacity_x": 11.32,
  "retest_memory_log_source": "historical-recovered/fresh-row-27/result.json"
}
Fresh sampled VRAM: 31.440 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 22.79 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid; ai-info 10K gate cleared

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 4210,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r2.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-33.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 33,
    "original_host": "10.0.0.30",
    "logged_model_load_delta_GiB": 6.75,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 114.586086,
    "logged_available_KV_GiB": 22.79,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 373360,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 11.32,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "37540bd9d6d3a34812dcc6e2d956c59bc7047f62aa91288bafefb2fc8773f312",
    "source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/vnext-release-repro/cleanroom-20260629T144538Z/localaiservers/qwen36-gfx906/vnext-launch-runs/gguf-dense27b-tp8",
    "source_lines": {
      "load": {
        "number": 4589,
        "text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 14:59:15 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 114.586086 seconds"
      },
      "KV": {
        "number": 5168,
        "text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 15:02:04 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB"
      },
      "tokens": {
        "number": 5175,
        "text": "(EngineCore_DP0 pid=200) INFO 06-29 15:02:04 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens"
      },
      "capacity": {
        "number": 5176,
        "text": "(EngineCore_DP0 pid=200) INFO 06-29 15:02:04 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x"
      },
      "engine": {
        "number": 64,
        "text": "(EngineCore_DP0 pid=200) INFO 06-29 14:56:55 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 22.79,
  "logged_GPU_KV_tokens": 373360,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-27/result.json",
  "retest_strict_tokens": 4508,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33758752768,
    "1": 33634103296,
    "2": 33684492288,
    "3": 33634070528,
    "4": 33684193280,
    "5": 33633898496,
    "6": 33684754432,
    "7": 33634291712
  },
  "retest_sampled_peak_VRAM_GiB": 31.440288543701172,
  "retest_memory_observation_count": 601,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_memory_log_source": "historical-recovered/fresh-row-27/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.30 · Stand-Alone GGUF Release-Package Audit / table 1GGUF F16Qwen/Qwen3.6-27BTP8MTP off PASS3715 tokens 71.12966.530 13.592 / 41.71810K: 12.328 / 162.646Strict: 13.505 / 66.534Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 22.79 GiB · engine report373,360 grouped KV tokens · 11.32× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-34
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-34 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row34-15f69466a8-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-34/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-34/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-34/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-34/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-34/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-34/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense"
python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-34
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-34 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 70.896 · valid historical-model sample10K 66.395 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 4044,
    "content_chars": 7548,
    "content_has_think_close": false,
    "elapsed_seconds": 72.31937592290342,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T04:11:40Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.011942341923713684,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 7548,
    "qwen_answer_text_sha256": "b8dcb06babe50c57ac2d999a7a4366bd9156506cbb59b91ba497f9e84fb74165",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 7548,
    "qwen_visible_text_sha256": "b8dcb06babe50c57ac2d999a7a4366bd9156506cbb59b91ba497f9e84fb74165",
    "raw_has_think_close": false,
    "reasoning_chars": 12371,
    "text_chars": 19920,
    "text_sha256": "ce311899a1f84ab85c42383d67628ec0138a96d7e3d4f788a18abcc7387ecd9c",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 57.98553304374218,
    "vllm_decode_tps": 69.74153358130472,
    "vllm_generation_tokens_delta": 4044.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.324300050735474,
    "vllm_prefill_tps": 30.08873023278164,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 55.91862413623611
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 41.197517121210694,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T04:12:21Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008826046250760555,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9647,
    "text_chars": 9647,
    "text_sha256": "1d259b96408168939c15694a69070d10a6bea9eaa997e3817f3dba8b5d0a375b",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 28.21035666577518,
    "vllm_decode_tps": 70.89594873596197,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 12.978673229925334,
    "vllm_prefill_tps": 33.20832510107657,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 48.546614935934876
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 39469,
    "content_has_think_close": false,
    "elapsed_seconds": 165.68056911695749,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T04:15:07Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008547734469175339,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 39469,
    "qwen_answer_text_sha256": "8cf441537e4d89e9f0d90733dad51c162387787efbc9ee5fa4d325d5f4a62c34",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 39469,
    "qwen_visible_text_sha256": "8cf441537e4d89e9f0d90733dad51c162387787efbc9ee5fa4d325d5f4a62c34",
    "raw_has_think_close": false,
    "reasoning_chars": 12371,
    "text_chars": 51841,
    "text_sha256": "d76a00c8f065f38b2b3551bad9418362baf93c16fed14699be81777846b1f433",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 150.61303294822574,
    "vllm_decode_tps": 66.3953165556235,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.058221222832799,
    "vllm_prefill_tps": 28.62223855142161,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 60.35710797770609
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 27, 28, 32, 33, 34. One new sample.PASS4,508 natural tokens2K 71.02510K 66.531 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-08T22:18:52.716405342Z",
    "FinishedAt": "2026-11-08T22:40:00.260467584Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "9a55da176fe75043aa7dc1ca12883e4f952738b9ebf2616b6f0edce4cb417ed7",
  "retest_strict": "PASS",
  "retest_strict_tokens": 4508,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 4508,
      "decode_tps": 69.61732325093088,
      "prefill_seconds": 14.042354583740234,
      "client_total_seconds": 78.80580217391253,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 64.75399784836918,
        "vllm:request_prefill_time_seconds_sum": 14.042354583740234,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 4508.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
      "summary_sha256": "754f0c5eeaf987089a936b03a08af821069b38f7e14fa59e17115de3fc4d2fae"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 71.02517985930231,
      "prefill_seconds": 13.268245195038617,
      "client_total_seconds": 41.43482113070786,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 28.15902760066092,
        "vllm:request_prefill_time_seconds_sum": 13.268245195038617,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2000.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
      "summary_sha256": "aff3ec02e75684eb8bd8ecdd4936b31fc9fbb3754970d3b0a39d80e61d360495"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 66.53116656725801,
      "prefill_seconds": 12.96059922594577,
      "client_total_seconds": 163.27578648924828,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_decode_time_seconds_sum": 150.3054961450398,
        "vllm:request_prefill_time_seconds_sum": 12.96059922594577,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 10000.0,
        "vllm:request_prefill_time_seconds_count": 1.0
      },
      "request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
      "summary_sha256": "29705fa055020f3834638960e4af19073f76c5672f7f2d3c621b46c7bf61f6ef"
    }
  },
  "retest_strict_TPS": 69.61732325093088,
  "retest_strict_prefill_seconds": 14.042354583740234,
  "retest_strict_total_seconds": 78.80580217391253,
  "retest_2K_TPS": 71.02517985930231,
  "retest_2K_prefill_seconds": 13.268245195038617,
  "retest_2K_total_seconds": 41.43482113070786,
  "retest_10K_TPS": 66.53116656725801,
  "retest_10K_prefill_seconds": 12.96059922594577,
  "retest_10K_total_seconds": 163.27578648924828,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33758752768,
    "1": 33634103296,
    "2": 33684492288,
    "3": 33634070528,
    "4": 33684193280,
    "5": 33633898496,
    "6": 33684754432,
    "7": 33634291712
  },
  "retest_sampled_peak_VRAM_GiB": 31.440288543701172,
  "retest_memory_observation_count": 601,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 6.75,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_logged_context_capacity_x": 11.32,
  "retest_memory_log_source": "historical-recovered/fresh-row-27/result.json"
}
Fresh sampled VRAM: 31.440 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 22.79 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid; ai-info 10K gate cleared

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 3715,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r2.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-34.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 34,
    "original_host": "10.0.0.30",
    "logged_model_load_delta_GiB": 6.75,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 112.330588,
    "logged_available_KV_GiB": 22.79,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 373360,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 11.32,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "440fc7e6d449bf85ebc64565dfb0544de1e59b96415d3a09f31df4c5871a8a7c",
    "source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/vnext-standalone-audit/20260629T171707Z/localaiservers/vnext-launch-runs/gguf-dense27b-tp8",
    "source_lines": {
      "load": {
        "number": 4589,
        "text": "(Worker pid=251) (Worker_TP0 pid=251) INFO 06-29 17:28:29 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 112.330588 seconds"
      },
      "KV": {
        "number": 5169,
        "text": "(Worker pid=251) (Worker_TP0 pid=251) INFO 06-29 17:31:23 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB"
      },
      "tokens": {
        "number": 5175,
        "text": "(EngineCore_DP0 pid=224) INFO 06-29 17:31:23 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens"
      },
      "capacity": {
        "number": 5176,
        "text": "(EngineCore_DP0 pid=224) INFO 06-29 17:31:23 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x"
      },
      "engine": {
        "number": 64,
        "text": "(EngineCore_DP0 pid=224) INFO 06-29 17:26:12 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 22.79,
  "logged_GPU_KV_tokens": 373360,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-27/result.json",
  "retest_strict_tokens": 4508,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33758752768,
    "1": 33634103296,
    "2": 33684492288,
    "3": 33634070528,
    "4": 33684193280,
    "5": 33633898496,
    "6": 33684754432,
    "7": 33634291712
  },
  "retest_sampled_peak_VRAM_GiB": 31.440288543701172,
  "retest_memory_observation_count": 601,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_memory_log_source": "historical-recovered/fresh-row-27/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.30 · Public-Staged HF Reproduction Audit / table 1HF FP16Qwen/Qwen3.6-27BTP8MTP off PASS4884 tokens 71.34666.729 14.065 / 42.10610K: 13.218 / 163.087Strict: 13.993 / 84.091Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.84 GiB (rank 0)Available KV: 22.64 GiB · engine report370,864 grouped KV tokens · 11.25× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-35
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-35 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row35-99603ed262-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-35/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-35/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-35/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-35/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-35/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-35/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/opt/local-models/qwen36-27b-hf,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-35
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-35 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 71.295 · valid historical-model sample10K 66.800 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3718,
    "content_chars": 5402,
    "content_has_think_close": false,
    "elapsed_seconds": 68.07331505976617,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T04:41:48Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.0126217445358634,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 5402,
    "qwen_answer_text_sha256": "44119bdba6f8fe6fa4fec4357ee8a88c27eda314f12159987fe0552df4205ef4",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 5402,
    "qwen_visible_text_sha256": "44119bdba6f8fe6fa4fec4357ee8a88c27eda314f12159987fe0552df4205ef4",
    "raw_has_think_close": false,
    "reasoning_chars": 12461,
    "text_chars": 17864,
    "text_sha256": "9399dfc9caf27d0063a4ea1cfe02070c9d0382ef098359845c7cc888a3123158",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 52.871985126286745,
    "vllm_decode_tps": 70.32079448349472,
    "vllm_generation_tokens_delta": 3718.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.192512446083128,
    "vllm_prefill_tps": 28.36923790779047,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 54.61758394953611
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 42.9475625436753,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T04:42:31Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.006040042266249657,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9610,
    "text_chars": 9610,
    "text_sha256": "b6e619acbecd7b98e7bd3fafcd526951e3260b60cf5909be4ad3a5f325565bff",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 28.05252104625106,
    "vllm_decode_tps": 71.29484001464746,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.887143962085247,
    "vllm_prefill_tps": 28.95115417018038,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 46.56841696117469
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 41015,
    "content_has_think_close": false,
    "elapsed_seconds": 164.69132177624851,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T04:45:16Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008324465714395046,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 41015,
    "qwen_answer_text_sha256": "ceaec15a92ffffd97f2dbd1c9884a52f22187f067808c2022d82e8ed942b9d31",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 41015,
    "qwen_visible_text_sha256": "ceaec15a92ffffd97f2dbd1c9884a52f22187f067808c2022d82e8ed942b9d31",
    "raw_has_think_close": false,
    "reasoning_chars": 12461,
    "text_chars": 53477,
    "text_sha256": "b1abee2192f7316f13106288d0a3b7898999e3bfa2fdf13496f9212608eb5782",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 149.7005802281201,
    "vllm_decode_tps": 66.80000828828837,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.980803821235895,
    "vllm_prefill_tps": 28.77015179846625,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 60.719653544259685
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 35, 38, 41, 44. One new sample.PASS3,609 natural tokens2K 70.95110K 66.469 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-08T23:40:06.929336115Z",
    "FinishedAt": "2026-11-08T23:58:45.910937408Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "a96218fee0c78b5145db76c43cdf273ff01d169be52e0662ddcc70106f936f79",
  "retest_strict": "PASS",
  "retest_strict_tokens": 3609,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 3609,
      "decode_tps": 70.06009773846488,
      "prefill_seconds": 12.782165881246328,
      "client_total_seconds": 64.30384199228138,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 3609.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_sum": 12.782165881246328,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 51.512917002663016,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "ee97c35547770515d3bd63d45b2557687a8278bbeb38ce72abbaad2fee0ab250",
      "summary_sha256": "6ae03f5709d05f854c0ff4b3e846b625d2d4a691aac551f35ae21d6a4b36c17e"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 70.95081153473755,
      "prefill_seconds": 15.072314956225455,
      "client_total_seconds": 43.269196966663,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 2000.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_sum": 15.072314956225455,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 28.188542974181473,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "d476546705501abd0468f2fc6527e9ace49f732079e53b75abf7bcfb98635f44",
      "summary_sha256": "f70df1fbfc18c6af1f765a73663863bb438fbe1dd644049da3c9e283ae692d0e"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 66.46936099788869,
      "prefill_seconds": 14.515854343771935,
      "client_total_seconds": 164.97092120815068,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 10000.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_sum": 14.515854343771935,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 150.4452555263415,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "ee5df402413359190aaa7ed74f4a298f749233768156b8fc14f2ea4bef0e8017",
      "summary_sha256": "1dfc4acf114048076b24fd3ed43f4a13dd3800ec8892ff4352974a55fd56aba2"
    }
  },
  "retest_strict_TPS": 70.06009773846488,
  "retest_strict_prefill_seconds": 12.782165881246328,
  "retest_strict_total_seconds": 64.30384199228138,
  "retest_2K_TPS": 70.95081153473755,
  "retest_2K_prefill_seconds": 15.072314956225455,
  "retest_2K_total_seconds": 43.269196966663,
  "retest_10K_TPS": 66.46936099788869,
  "retest_10K_prefill_seconds": 14.515854343771935,
  "retest_10K_total_seconds": 164.97092120815068,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33776762880,
    "1": 33666486272,
    "2": 33716842496,
    "3": 33666523136,
    "4": 33716817920,
    "5": 33666486272,
    "6": 33716846592,
    "7": 33666510848
  },
  "retest_sampled_peak_VRAM_GiB": 31.457061767578125,
  "retest_memory_observation_count": 527,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 712,
        "text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
        "values": [
          "6.84"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 1267,
        "text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
        "values": [
          "22.64"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 1273,
        "text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
        "values": [
          "370,864"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 1274,
        "text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
        "values": [
          "131,072",
          "11.25"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 6.84,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 22.64,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 370864,
  "retest_logged_context_capacity_x": 11.25,
  "retest_memory_log_source": "historical-recovered/fresh-row-35/result.json"
}
Fresh sampled VRAM: 31.457 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 22.64 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid; ai-info 10K gate cleared

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 4884,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r3.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-35.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 35,
    "original_host": "10.0.0.30",
    "logged_model_load_delta_GiB": 6.84,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 22.097343,
    "logged_available_KV_GiB": 22.64,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 370864,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 11.25,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "102a7ff5f97bc9e718827f1860435d52aba59065ead3318be73d7a2872aef11a",
    "source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/vnext-release-repro/localaiservers/qwen36-gfx906/vnext-launch-runs/repro-hf-dense27b-tp8-public",
    "source_lines": {
      "load": {
        "number": 712,
        "text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-29 13:41:17 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.097343 seconds"
      },
      "KV": {
        "number": 1271,
        "text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-29 13:42:58 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB"
      },
      "tokens": {
        "number": 1273,
        "text": "(EngineCore_DP0 pid=184) INFO 06-29 13:42:58 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens"
      },
      "capacity": {
        "number": 1274,
        "text": "(EngineCore_DP0 pid=184) INFO 06-29 13:42:58 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x"
      },
      "engine": {
        "number": 55,
        "text": "(EngineCore_DP0 pid=184) INFO 06-29 13:40:27 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-27b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 22.64,
  "logged_GPU_KV_tokens": 370864,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-35/result.json",
  "retest_strict_tokens": 3609,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33776762880,
    "1": 33666486272,
    "2": 33716842496,
    "3": 33666523136,
    "4": 33716817920,
    "5": 33666486272,
    "6": 33716846592,
    "7": 33666510848
  },
  "retest_sampled_peak_VRAM_GiB": 31.457061767578125,
  "retest_memory_observation_count": 527,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 712,
        "text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
        "values": [
          "6.84"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 1267,
        "text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
        "values": [
          "22.64"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 1273,
        "text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
        "values": [
          "370,864"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 1274,
        "text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
        "values": [
          "131,072",
          "11.25"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 22.64,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 370864,
  "retest_memory_log_source": "historical-recovered/fresh-row-35/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.30 · Public-Staged HF Reproduction Audit / table 1HF FP16Qwen/Qwen3.6-35B-A3BTP8MTP off PASS2975 tokens 116.258109.242 0.817 / 18.03010K: 0.818 / 92.369Strict: 0.808 / 26.577Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 8.88 GiB (rank 0)Available KV: 19.91 GiB · engine report521,696 grouped KV tokens · 15.82× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-36
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-36 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row36-f3436e8210-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-36/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-36/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-36/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-36/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-36/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-36/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-moe,dst=/opt/local-models/qwen36-35b-a3b-hf,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-moe "$MODEL_DIR/qwen36-moe"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-36
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-36 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 116.158 · valid historical-model sample10K 109.274 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3867,
    "content_chars": 7448,
    "content_has_think_close": false,
    "elapsed_seconds": 34.548473133705556,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T05:24:29Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.010292066261172295,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 7448,
    "qwen_answer_text_sha256": "df3adaa80cdc4fb582fc7654e356001d375fc9f61acfe39fbbc3a45bbf20995e",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 7448,
    "qwen_visible_text_sha256": "df3adaa80cdc4fb582fc7654e356001d375fc9f61acfe39fbbc3a45bbf20995e",
    "raw_has_think_close": false,
    "reasoning_chars": 12244,
    "text_chars": 19693,
    "text_sha256": "419dea0c3413266f4ff53d4a698d31273de2d306b49681fb788af33368af5c96",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 33.72320320736617,
    "vllm_decode_tps": 114.66882241943523,
    "vllm_generation_tokens_delta": 3867.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8162200124934316,
    "vllm_prefill_tps": 528.0439016477416,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 111.9296932467718
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 18.036245052702725,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T05:24:47Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008652694523334503,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 10030,
    "text_chars": 10030,
    "text_sha256": "0ab784d0ec5e96d9efb3934f80cfd8b0cf2b3536f46f7c75264e5114bda0c04e",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 17.217981002293527,
    "vllm_decode_tps": 116.15763774704998,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8099861172959208,
    "vllm_prefill_tps": 532.1078853040863,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 110.88782582826467
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 43387,
    "content_has_think_close": false,
    "elapsed_seconds": 92.34119913075119,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T05:26:20Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.00704532116651535,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 43387,
    "qwen_answer_text_sha256": "d35733135a89d8effc0a7c67690e9d4414d78fd965a979b0150343275ef63dbf",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 43387,
    "qwen_visible_text_sha256": "d35733135a89d8effc0a7c67690e9d4414d78fd965a979b0150343275ef63dbf",
    "raw_has_think_close": false,
    "reasoning_chars": 12244,
    "text_chars": 55632,
    "text_sha256": "af45e17100f84637c61eadb247d364e6be4ccd3fde6062efc0b81cb3cceddba6",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 91.51303980406374,
    "vllm_decode_tps": 109.27404467615487,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8177437912672758,
    "vllm_prefill_tps": 527.0599478744678,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 108.29402362254824
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 36, 39, 42, 45. One new sample.PASS2,911 natural tokens2K 117.36710K 110.169 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-09T00:00:14.441833328Z",
    "FinishedAt": "2026-11-09T00:20:09.78005595Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "d3e8150b92459eb9bfeb0628cfbc7f1ba14417d265985bd295006c8f223de819",
  "retest_strict": "PASS",
  "retest_strict_tokens": 2911,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 2911,
      "decode_tps": 116.74599429536063,
      "prefill_seconds": 0.8083759220317006,
      "client_total_seconds": 25.750953457318246,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 0.8083759220317006,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 24.93447434809059,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2911.0
      },
      "request_sha256": "e31c712c0771301f2e4d592bc966448d3143f2660e1b4cab33a91704f19e9370",
      "summary_sha256": "eebfaae9c73243c913cb60f1c6f4ec103cac551b4f8d5c5529c5364a6f3465ca"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 117.36665638338557,
      "prefill_seconds": 0.8091103108599782,
      "client_total_seconds": 17.857541773468256,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 0.8091103108599782,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 17.04061495512724,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2000.0
      },
      "request_sha256": "bca01af4a8338cd6274c1c9e0b281d8e59066ebae2ece3e3a636620066da3a03",
      "summary_sha256": "2c37396cb2c1b23f8ae028e9beb941f96a022ffb8a5da29d3363d63003b69672"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 110.16866681199451,
      "prefill_seconds": 0.8098680507391691,
      "client_total_seconds": 91.58954442292452,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 0.8098680507391691,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 90.76991026010364,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 10000.0
      },
      "request_sha256": "46fd7808353544eb86016bdd1746bafb67def6864a3f5352647b66cb9149f95a",
      "summary_sha256": "b44fdcf5c45447a4a3ba6eda737cc6adcf2f183f3569935f415e850a380531e5"
    }
  },
  "retest_strict_TPS": 116.74599429536063,
  "retest_strict_prefill_seconds": 0.8083759220317006,
  "retest_strict_total_seconds": 25.750953457318246,
  "retest_2K_TPS": 117.36665638338557,
  "retest_2K_prefill_seconds": 0.8091103108599782,
  "retest_2K_total_seconds": 17.857541773468256,
  "retest_10K_TPS": 110.16866681199451,
  "retest_10K_prefill_seconds": 0.8098680507391691,
  "retest_10K_total_seconds": 91.58954442292452,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 34334547968,
    "1": 34314465280,
    "2": 34314543104,
    "3": 34314469376,
    "4": 34334240768,
    "5": 34314461184,
    "6": 34314457088,
    "7": 34318581760
  },
  "retest_sampled_peak_VRAM_GiB": 31.976539611816406,
  "retest_memory_observation_count": 565,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 193,
        "text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
        "values": [
          "8.88"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 313,
        "text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
        "values": [
          "19.91"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 315,
        "text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
        "values": [
          "521,696"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 316,
        "text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
        "values": [
          "131,072",
          "15.82"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 8.88,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 19.91,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 521696,
  "retest_logged_context_capacity_x": 15.82,
  "retest_memory_log_source": "historical-recovered/fresh-row-36/result.json"
}
Fresh sampled VRAM: 31.977 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 19.91 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 2975,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "995ad96eacd98c81ed38be0c5b274b04031597b0",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r3.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-36.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 36,
    "original_host": "10.0.0.30",
    "logged_model_load_delta_GiB": 8.88,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 108.136458,
    "logged_available_KV_GiB": 19.91,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 521696,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 15.82,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "4eb4f84cc3c3a759f1a904f06f6727f7d5a570c87d16ff334c5f48045c2ef969",
    "source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/vnext-release-repro/localaiservers/qwen36-gfx906/vnext-launch-runs/repro-hf-moe35b-tp8-public",
    "source_lines": {
      "load": {
        "number": 193,
        "text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-29 14:15:32 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 108.136458 seconds"
      },
      "KV": {
        "number": 313,
        "text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-29 14:17:00 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB"
      },
      "tokens": {
        "number": 315,
        "text": "(EngineCore_DP0 pid=171) INFO 06-29 14:17:00 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens"
      },
      "capacity": {
        "number": 316,
        "text": "(EngineCore_DP0 pid=171) INFO 06-29 14:17:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x"
      },
      "engine": {
        "number": 33,
        "text": "(EngineCore_DP0 pid=171) INFO 06-29 14:13:13 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-35b-a3b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-35b-a3b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.6-35B-A3B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 19.91,
  "logged_GPU_KV_tokens": 521696,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-36/result.json",
  "retest_strict_tokens": 2911,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 34334547968,
    "1": 34314465280,
    "2": 34314543104,
    "3": 34314469376,
    "4": 34334240768,
    "5": 34314461184,
    "6": 34314457088,
    "7": 34318581760
  },
  "retest_sampled_peak_VRAM_GiB": 31.976539611816406,
  "retest_memory_observation_count": 565,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 193,
        "text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
        "values": [
          "8.88"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 313,
        "text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
        "values": [
          "19.91"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 315,
        "text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
        "values": [
          "521,696"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 316,
        "text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
        "values": [
          "131,072",
          "15.82"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 19.91,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 521696,
  "retest_memory_log_source": "historical-recovered/fresh-row-36/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.20 · Full vNext Release-Readiness Serving Replay / table 1GGUF F16Qwen/Qwen3.6-27BTP8MTP off PASS3701 tokens 70.88166.446 13.934 / 42.15910K: 14.571 / 165.078Strict: 14.468 / 67.418Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 22.79 GiB · engine report373,360 grouped KV tokens · 11.32× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-37
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-37 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row37-2c5900fca7-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-37/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-37/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-37/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-37/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-37/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-37/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense"
python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-37
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-37 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 70.841 · valid historical-model sample10K 66.373 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3878,
    "content_chars": 6406,
    "content_has_think_close": false,
    "elapsed_seconds": 69.53517793025821,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T06:02:00Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.009362303651869297,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 6406,
    "qwen_answer_text_sha256": "d01a470e93c380e6a60db3467c7434ffff73248597d6ea2708468e7141a34d17",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 6406,
    "qwen_visible_text_sha256": "d01a470e93c380e6a60db3467c7434ffff73248597d6ea2708468e7141a34d17",
    "raw_has_think_close": false,
    "reasoning_chars": 12683,
    "text_chars": 19090,
    "text_sha256": "2d71822f464cf3ae4264986cd1e5f82bfc75faa6829d04e192e6eafead517f2a",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 55.5404448620975,
    "vllm_decode_tps": 69.82299132872927,
    "vllm_generation_tokens_delta": 3878.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 13.984859772957861,
    "vllm_prefill_tps": 30.819043379570587,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 55.77033259179293
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 42.994658844545484,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T06:02:43Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008095500990748405,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9905,
    "text_chars": 9905,
    "text_sha256": "5ab7bffe8c721a17390b8d131cd339b67b68be1e9b06f97e8ee51276791519bb",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 28.23240473214537,
    "vllm_decode_tps": 70.84058262039589,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.754096891731024,
    "vllm_prefill_tps": 29.212225130604583,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 46.51740596968895
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 40852,
    "content_has_think_close": false,
    "elapsed_seconds": 165.7533283000812,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T06:05:29Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.00877947174012661,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 40852,
    "qwen_answer_text_sha256": "10f12fb0806376aa6c74232eb8a7892839336925fbd86df0e95736cbf71d0808",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 40852,
    "qwen_visible_text_sha256": "10f12fb0806376aa6c74232eb8a7892839336925fbd86df0e95736cbf71d0808",
    "raw_has_think_close": false,
    "reasoning_chars": 12683,
    "text_chars": 53536,
    "text_sha256": "d198d0c8b309b50972d4e85140d9d7b39c69b4141296698daa7c5fcb0112302d",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 150.66444395296276,
    "vllm_decode_tps": 66.37266058023609,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.079772244207561,
    "vllm_prefill_tps": 28.581333525481835,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 60.33061358439764
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 37, 40, 43. One new sample.PASS4,263 natural tokens2K 70.87610K 66.474 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-09T00:22:27.410864084Z",
    "FinishedAt": "2026-11-09T00:43:40.972906165Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "ae0dba47e379e10ae120c82dbe767b915856ebc0a56771c0751407ceb9b17f98",
  "retest_strict": "PASS",
  "retest_strict_tokens": 4263,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 4263,
      "decode_tps": 69.65298485444228,
      "prefill_seconds": 13.264510760083795,
      "client_total_seconds": 74.47932942770422,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 13.264510760083795,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 61.20340727549046,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 4263.0
      },
      "request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
      "summary_sha256": "2a4ec12838f2054993cfc40a0ce46452682e1e06ac8c785eefd538bcc5266ae6"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 70.8759028403918,
      "prefill_seconds": 15.0672946870327,
      "client_total_seconds": 43.29480273462832,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 15.0672946870327,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 28.218335426412523,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2000.0
      },
      "request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
      "summary_sha256": "304d5e8b0f2afd6fcf019e0b454f8917c037eece212906cfcf2c1686136c1ee7"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 66.47442948531119,
      "prefill_seconds": 14.876342931762338,
      "client_total_seconds": 165.31903470959514,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 14.876342931762338,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 150.43378450069577,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 10000.0
      },
      "request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
      "summary_sha256": "895a7e44c83b5069acf38a1a85397205ac0d8a1cefc38010e259fa29cda426f9"
    }
  },
  "retest_strict_TPS": 69.65298485444228,
  "retest_strict_prefill_seconds": 13.264510760083795,
  "retest_strict_total_seconds": 74.47932942770422,
  "retest_2K_TPS": 70.8759028403918,
  "retest_2K_prefill_seconds": 15.0672946870327,
  "retest_2K_total_seconds": 43.29480273462832,
  "retest_10K_TPS": 66.47442948531119,
  "retest_10K_prefill_seconds": 14.876342931762338,
  "retest_10K_total_seconds": 165.31903470959514,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33744068608,
    "1": 33634144256,
    "2": 33684496384,
    "3": 33634095104,
    "4": 33684230144,
    "5": 33633964032,
    "6": 33684738048,
    "7": 33634250752
  },
  "retest_sampled_peak_VRAM_GiB": 31.426612854003906,
  "retest_memory_observation_count": 604,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-09T00:25:47.689313425Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:25:47 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 105.144041 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-09T00:28:42.161027038Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:28:42 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-09T00:28:42.250244890Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-09T00:28:42.250362411Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 6.75,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_logged_context_capacity_x": 11.32,
  "retest_memory_log_source": "historical-recovered/fresh-row-37/result.json"
}
Fresh sampled VRAM: 31.427 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 22.79 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid; ai-info 10K gate cleared

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 3701,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r3.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-37.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 37,
    "original_host": "10.0.0.20",
    "logged_model_load_delta_GiB": 6.75,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 106.384137,
    "logged_available_KV_GiB": 22.79,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 373360,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 11.32,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "8db635ce1dee530b6536026905839b582cf7335807fb11229461c2333087d67b",
    "source_archive": "historical-recovered/host10.0.0.20-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/vnext-repeat-20260630T045153Z/runtime/vnext-launch-runs/20260630T050006Z-2928686/gguf-dense27b-tp8",
    "source_lines": {
      "load": {
        "number": 4596,
        "text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-30 05:05:39 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 106.384137 seconds"
      },
      "KV": {
        "number": 5170,
        "text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-30 05:08:26 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB"
      },
      "tokens": {
        "number": 5174,
        "text": "(EngineCore_DP0 pid=200) INFO 06-30 05:08:26 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens"
      },
      "capacity": {
        "number": 5175,
        "text": "(EngineCore_DP0 pid=200) INFO 06-30 05:08:26 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x"
      },
      "engine": {
        "number": 63,
        "text": "(EngineCore_DP0 pid=200) INFO 06-30 05:03:28 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=6a9e13bd6fc8f0983b9b99948120bc37f49c13e9, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 22.79,
  "logged_GPU_KV_tokens": 373360,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-37/result.json",
  "retest_strict_tokens": 4263,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33744068608,
    "1": 33634144256,
    "2": 33684496384,
    "3": 33634095104,
    "4": 33684230144,
    "5": 33633964032,
    "6": 33684738048,
    "7": 33634250752
  },
  "retest_sampled_peak_VRAM_GiB": 31.426612854003906,
  "retest_memory_observation_count": 604,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-09T00:25:47.689313425Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:25:47 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 105.144041 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-09T00:28:42.161027038Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:28:42 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-09T00:28:42.250244890Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-09T00:28:42.250362411Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_memory_log_source": "historical-recovered/fresh-row-37/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.20 · Full vNext Release-Readiness Serving Replay / table 1HF FP16Qwen/Qwen3.6-27BTP8MTP off PASS4237 tokens 71.39566.876 14.888 / 42.91010K: 14.015 / 163.554Strict: 13.858 / 74.265Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.84 GiB (rank 0)Available KV: 22.64 GiB · engine report370,864 grouped KV tokens · 11.25× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-38
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-38 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row38-f33ad375e1-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-38/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-38/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-38/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-38/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-38/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-38/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/opt/local-models/qwen36-27b-hf,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-38
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-38 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 71.286 · valid historical-model sample10K 66.781 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3588,
    "content_chars": 5883,
    "content_has_think_close": false,
    "elapsed_seconds": 64.89045448042452,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T06:32:14Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.01263996958732605,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 5883,
    "qwen_answer_text_sha256": "3d0e920f50ef42aedafc11bbfb924b76222895f37114bb681663302f9b91f75c",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 5883,
    "qwen_visible_text_sha256": "3d0e920f50ef42aedafc11bbfb924b76222895f37114bb681663302f9b91f75c",
    "raw_has_think_close": false,
    "reasoning_chars": 11124,
    "text_chars": 17008,
    "text_sha256": "fc08f3a103846a02d7b39738392e17893d685b555aca4052a1d533e1812c9962",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 50.96286449301988,
    "vllm_decode_tps": 70.40420580148962,
    "vllm_generation_tokens_delta": 3588.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 13.919675651937723,
    "vllm_prefill_tps": 30.96336515140003,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 55.29318647448201
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 42.615525607950985,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T06:32:56Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008263365365564823,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9427,
    "text_chars": 9427,
    "text_sha256": "54cbb97b2b6cddbb147bbe1694b9bcc497925aeaf06256baf5c70d59ca44ec82",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 28.056015537120402,
    "vllm_decode_tps": 71.28595995229033,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.55124822538346,
    "vllm_prefill_tps": 29.61945211326653,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 46.93125267066635
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 38346,
    "content_has_think_close": false,
    "elapsed_seconds": 163.270943348296,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T06:35:40Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008678438141942024,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 38346,
    "qwen_answer_text_sha256": "518a9ba4dd4d45e51d323b328845c047caa5881d47e582a24689ee5960efb863",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 38346,
    "qwen_visible_text_sha256": "518a9ba4dd4d45e51d323b328845c047caa5881d47e582a24689ee5960efb863",
    "raw_has_think_close": false,
    "reasoning_chars": 11124,
    "text_chars": 49471,
    "text_sha256": "8c34f28f2fca20d1d2be75fb967201d2860e51eb043cda3b63d8047c14c5db74",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 149.74291642103344,
    "vllm_decode_tps": 66.78112219934941,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 13.51911969203502,
    "vllm_prefill_tps": 31.880774031014,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 61.247885232509546
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 35, 38, 41, 44. One new sample.PASS3,609 natural tokens2K 70.95110K 66.469 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-08T23:40:06.929336115Z",
    "FinishedAt": "2026-11-08T23:58:45.910937408Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "a96218fee0c78b5145db76c43cdf273ff01d169be52e0662ddcc70106f936f79",
  "retest_strict": "PASS",
  "retest_strict_tokens": 3609,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 3609,
      "decode_tps": 70.06009773846488,
      "prefill_seconds": 12.782165881246328,
      "client_total_seconds": 64.30384199228138,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 3609.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_sum": 12.782165881246328,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 51.512917002663016,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "ee97c35547770515d3bd63d45b2557687a8278bbeb38ce72abbaad2fee0ab250",
      "summary_sha256": "6ae03f5709d05f854c0ff4b3e846b625d2d4a691aac551f35ae21d6a4b36c17e"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 70.95081153473755,
      "prefill_seconds": 15.072314956225455,
      "client_total_seconds": 43.269196966663,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 2000.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_sum": 15.072314956225455,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 28.188542974181473,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "d476546705501abd0468f2fc6527e9ace49f732079e53b75abf7bcfb98635f44",
      "summary_sha256": "f70df1fbfc18c6af1f765a73663863bb438fbe1dd644049da3c9e283ae692d0e"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 66.46936099788869,
      "prefill_seconds": 14.515854343771935,
      "client_total_seconds": 164.97092120815068,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 10000.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_sum": 14.515854343771935,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 150.4452555263415,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "ee5df402413359190aaa7ed74f4a298f749233768156b8fc14f2ea4bef0e8017",
      "summary_sha256": "1dfc4acf114048076b24fd3ed43f4a13dd3800ec8892ff4352974a55fd56aba2"
    }
  },
  "retest_strict_TPS": 70.06009773846488,
  "retest_strict_prefill_seconds": 12.782165881246328,
  "retest_strict_total_seconds": 64.30384199228138,
  "retest_2K_TPS": 70.95081153473755,
  "retest_2K_prefill_seconds": 15.072314956225455,
  "retest_2K_total_seconds": 43.269196966663,
  "retest_10K_TPS": 66.46936099788869,
  "retest_10K_prefill_seconds": 14.515854343771935,
  "retest_10K_total_seconds": 164.97092120815068,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33776762880,
    "1": 33666486272,
    "2": 33716842496,
    "3": 33666523136,
    "4": 33716817920,
    "5": 33666486272,
    "6": 33716846592,
    "7": 33666510848
  },
  "retest_sampled_peak_VRAM_GiB": 31.457061767578125,
  "retest_memory_observation_count": 527,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 712,
        "text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
        "values": [
          "6.84"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 1267,
        "text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
        "values": [
          "22.64"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 1273,
        "text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
        "values": [
          "370,864"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 1274,
        "text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
        "values": [
          "131,072",
          "11.25"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 6.84,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 22.64,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 370864,
  "retest_logged_context_capacity_x": 11.25,
  "retest_memory_log_source": "historical-recovered/fresh-row-35/result.json"
}
Fresh sampled VRAM: 31.457 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 22.64 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid; ai-info 10K gate cleared

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 4237,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r3.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-38.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 38,
    "original_host": "10.0.0.20",
    "logged_model_load_delta_GiB": 6.84,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 23.229114,
    "logged_available_KV_GiB": 22.64,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 370864,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 11.25,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "9e63d9b84eed3eab33372a7e7d9ccd320c5f0004a5d35eecda83a3d621f402db",
    "source_archive": "historical-recovered/host10.0.0.20-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/vnext-repeat-20260630T045153Z/runtime/vnext-launch-runs/20260630T050006Z-2928686/hf-dense27b-tp8",
    "source_lines": {
      "load": {
        "number": 712,
        "text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-30 05:39:43 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 23.229114 seconds"
      },
      "KV": {
        "number": 1271,
        "text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-30 05:41:25 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB"
      },
      "tokens": {
        "number": 1273,
        "text": "(EngineCore_DP0 pid=184) INFO 06-30 05:41:25 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens"
      },
      "capacity": {
        "number": 1274,
        "text": "(EngineCore_DP0 pid=184) INFO 06-30 05:41:25 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x"
      },
      "engine": {
        "number": 55,
        "text": "(EngineCore_DP0 pid=184) INFO 06-30 05:38:53 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-27b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 22.64,
  "logged_GPU_KV_tokens": 370864,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-35/result.json",
  "retest_strict_tokens": 3609,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33776762880,
    "1": 33666486272,
    "2": 33716842496,
    "3": 33666523136,
    "4": 33716817920,
    "5": 33666486272,
    "6": 33716846592,
    "7": 33666510848
  },
  "retest_sampled_peak_VRAM_GiB": 31.457061767578125,
  "retest_memory_observation_count": 527,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 712,
        "text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
        "values": [
          "6.84"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 1267,
        "text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
        "values": [
          "22.64"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 1273,
        "text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
        "values": [
          "370,864"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 1274,
        "text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
        "values": [
          "131,072",
          "11.25"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 22.64,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 370864,
  "retest_memory_log_source": "historical-recovered/fresh-row-35/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.20 · Full vNext Release-Readiness Serving Replay / table 1HF FP16Qwen/Qwen3.6-35B-A3BTP8MTP off PASS3310 tokens 116.542109.672 0.810 / 17.98010K: 0.810 / 92.001Strict: 0.809 / 29.432Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 8.88 GiB (rank 0)Available KV: 19.91 GiB · engine report521,696 grouped KV tokens · 15.82× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-39
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-39 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row39-a344df5fa1-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-39/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-39/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-39/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-39/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-39/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-39/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-moe,dst=/opt/local-models/qwen36-35b-a3b-hf,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-moe "$MODEL_DIR/qwen36-moe"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-39
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-39 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 117.028 · valid historical-model sample10K 110.021 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3268,
    "content_chars": 5228,
    "content_has_think_close": false,
    "elapsed_seconds": 28.94538085255772,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T07:12:14Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.008697621524333954,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 5228,
    "qwen_answer_text_sha256": "facfcedef1b47165f2c0333e2266078fe5937046756c294e8c636f14b2e55856",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 5228,
    "qwen_visible_text_sha256": "facfcedef1b47165f2c0333e2266078fe5937046756c294e8c636f14b2e55856",
    "raw_has_think_close": false,
    "reasoning_chars": 10118,
    "text_chars": 15347,
    "text_sha256": "c5ac798c99358ba9bfb1c1e5a6634c07750fbffbc3d4fa406686e880ae4b8fa6",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 28.128748228773475,
    "vllm_decode_tps": 116.18007219592856,
    "vllm_generation_tokens_delta": 3268.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8081940831616521,
    "vllm_prefill_tps": 533.2877448371432,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 112.90229749080078
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 17.911016230471432,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T07:12:32Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008739661425352097,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9152,
    "text_chars": 9152,
    "text_sha256": "2e2cda4a7e91f091f07b9fb76f6795b4b46d34c06d67b86dd9a0f10cd7b071a0",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 17.08993177022785,
    "vllm_decode_tps": 117.02796868295134,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.812127098441124,
    "vllm_prefill_tps": 530.7051086305376,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 111.66312253111941
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 46155,
    "content_has_think_close": false,
    "elapsed_seconds": 91.70880929846317,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T07:14:04Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.0056197261437773705,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 46155,
    "qwen_answer_text_sha256": "99f0313665a258aa889dc09836d4da910585d581ad15d09867b8075d647ac554",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 46155,
    "qwen_visible_text_sha256": "99f0313665a258aa889dc09836d4da910585d581ad15d09867b8075d647ac554",
    "raw_has_think_close": false,
    "reasoning_chars": 10118,
    "text_chars": 56274,
    "text_sha256": "7024979e63cc90d99ec03dc2fe1d81b8cafbb572aa23120bb1f334686519c3f5",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 90.89162033330649,
    "vllm_decode_tps": 110.02114345997177,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8079116530716419,
    "vllm_prefill_tps": 533.4741717876681,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 109.04077892294234
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 36, 39, 42, 45. One new sample.PASS2,911 natural tokens2K 117.36710K 110.169 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-09T00:00:14.441833328Z",
    "FinishedAt": "2026-11-09T00:20:09.78005595Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "d3e8150b92459eb9bfeb0628cfbc7f1ba14417d265985bd295006c8f223de819",
  "retest_strict": "PASS",
  "retest_strict_tokens": 2911,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 2911,
      "decode_tps": 116.74599429536063,
      "prefill_seconds": 0.8083759220317006,
      "client_total_seconds": 25.750953457318246,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 0.8083759220317006,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 24.93447434809059,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2911.0
      },
      "request_sha256": "e31c712c0771301f2e4d592bc966448d3143f2660e1b4cab33a91704f19e9370",
      "summary_sha256": "eebfaae9c73243c913cb60f1c6f4ec103cac551b4f8d5c5529c5364a6f3465ca"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 117.36665638338557,
      "prefill_seconds": 0.8091103108599782,
      "client_total_seconds": 17.857541773468256,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 0.8091103108599782,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 17.04061495512724,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2000.0
      },
      "request_sha256": "bca01af4a8338cd6274c1c9e0b281d8e59066ebae2ece3e3a636620066da3a03",
      "summary_sha256": "2c37396cb2c1b23f8ae028e9beb941f96a022ffb8a5da29d3363d63003b69672"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 110.16866681199451,
      "prefill_seconds": 0.8098680507391691,
      "client_total_seconds": 91.58954442292452,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 0.8098680507391691,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 90.76991026010364,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 10000.0
      },
      "request_sha256": "46fd7808353544eb86016bdd1746bafb67def6864a3f5352647b66cb9149f95a",
      "summary_sha256": "b44fdcf5c45447a4a3ba6eda737cc6adcf2f183f3569935f415e850a380531e5"
    }
  },
  "retest_strict_TPS": 116.74599429536063,
  "retest_strict_prefill_seconds": 0.8083759220317006,
  "retest_strict_total_seconds": 25.750953457318246,
  "retest_2K_TPS": 117.36665638338557,
  "retest_2K_prefill_seconds": 0.8091103108599782,
  "retest_2K_total_seconds": 17.857541773468256,
  "retest_10K_TPS": 110.16866681199451,
  "retest_10K_prefill_seconds": 0.8098680507391691,
  "retest_10K_total_seconds": 91.58954442292452,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 34334547968,
    "1": 34314465280,
    "2": 34314543104,
    "3": 34314469376,
    "4": 34334240768,
    "5": 34314461184,
    "6": 34314457088,
    "7": 34318581760
  },
  "retest_sampled_peak_VRAM_GiB": 31.976539611816406,
  "retest_memory_observation_count": 565,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 193,
        "text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
        "values": [
          "8.88"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 313,
        "text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
        "values": [
          "19.91"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 315,
        "text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
        "values": [
          "521,696"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 316,
        "text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
        "values": [
          "131,072",
          "15.82"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 8.88,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 19.91,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 521696,
  "retest_logged_context_capacity_x": 15.82,
  "retest_memory_log_source": "historical-recovered/fresh-row-36/result.json"
}
Fresh sampled VRAM: 31.977 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 19.91 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 3310,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "995ad96eacd98c81ed38be0c5b274b04031597b0",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r3.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-39.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 39,
    "original_host": "10.0.0.20",
    "logged_model_load_delta_GiB": 8.88,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 115.559999,
    "logged_available_KV_GiB": 19.91,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 521696,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 15.82,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "6e661b86c7832729f1ccff5efa87579d419dd3771f3ed7e6ea42e1105ddfb44a",
    "source_archive": "historical-recovered/host10.0.0.20-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/vnext-repeat-20260630T045153Z/runtime/vnext-launch-runs/20260630T050006Z-2928686/hf-moe35b-tp8",
    "source_lines": {
      "load": {
        "number": 193,
        "text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-30 06:13:45 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 115.559999 seconds"
      },
      "KV": {
        "number": 313,
        "text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-30 06:15:07 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB"
      },
      "tokens": {
        "number": 315,
        "text": "(EngineCore_DP0 pid=171) INFO 06-30 06:15:07 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens"
      },
      "capacity": {
        "number": 316,
        "text": "(EngineCore_DP0 pid=171) INFO 06-30 06:15:07 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x"
      },
      "engine": {
        "number": 33,
        "text": "(EngineCore_DP0 pid=171) INFO 06-30 06:11:18 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-35b-a3b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-35b-a3b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.6-35B-A3B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 19.91,
  "logged_GPU_KV_tokens": 521696,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-36/result.json",
  "retest_strict_tokens": 2911,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 34334547968,
    "1": 34314465280,
    "2": 34314543104,
    "3": 34314469376,
    "4": 34334240768,
    "5": 34314461184,
    "6": 34314457088,
    "7": 34318581760
  },
  "retest_sampled_peak_VRAM_GiB": 31.976539611816406,
  "retest_memory_observation_count": 565,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 193,
        "text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
        "values": [
          "8.88"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 313,
        "text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
        "values": [
          "19.91"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 315,
        "text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
        "values": [
          "521,696"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 316,
        "text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
        "values": [
          "131,072",
          "15.82"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 19.91,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 521696,
  "retest_memory_log_source": "historical-recovered/fresh-row-36/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.30 · Full vNext Release-Readiness Serving Replay / table 2GGUF F16Qwen/Qwen3.6-27BTP8MTP off PASS4315 tokens 71.16166.638 14.232 / 42.34610K: 13.878 / 163.951Strict: 14.445 / 76.208Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 22.79 GiB · engine report373,360 grouped KV tokens · 11.32× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-40
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-40 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row40-4ed2ea0727-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-40/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-40/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-40/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-40/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-40/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-40/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense"
python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-40
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-40 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 70.949 · valid historical-model sample10K 66.461 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3907,
    "content_chars": 6521,
    "content_has_think_close": false,
    "elapsed_seconds": 69.24951494298875,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T07:50:37Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.012738149613142014,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 6521,
    "qwen_answer_text_sha256": "f71a7aa77008154118acaa8bce77c983cf68537ad16c8aaf250141d295344e11",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 6521,
    "qwen_visible_text_sha256": "f71a7aa77008154118acaa8bce77c983cf68537ad16c8aaf250141d295344e11",
    "raw_has_think_close": false,
    "reasoning_chars": 12901,
    "text_chars": 19423,
    "text_sha256": "6a78b9fc249f729fd22f6baf6d2d6fe6bda8d6b3238969514e27d0790adb833a",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 55.90619322191924,
    "vllm_decode_tps": 69.88492284729871,
    "vllm_generation_tokens_delta": 3907.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 13.3340172059834,
    "vllm_prefill_tps": 32.323342121277335,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 56.419167747478475
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 41.90379103273153,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T07:51:19Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.005990299396216869,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9874,
    "text_chars": 9874,
    "text_sha256": "ce16880512a707739f7b919b99fd9aec38787d685976d559c94585be4e3f91e7",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 28.189258734695613,
    "vllm_decode_tps": 70.94901000494846,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 13.706564854830503,
    "vllm_prefill_tps": 31.444786098108736,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 47.72837852398073
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 40712,
    "content_has_think_close": false,
    "elapsed_seconds": 164.8181800954044,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T07:54:04Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008725512772798538,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 40712,
    "qwen_answer_text_sha256": "259e6e860384fbf4cd65398f1acb30ababd88f54788727ae13a546228ad57c72",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 40712,
    "qwen_visible_text_sha256": "259e6e860384fbf4cd65398f1acb30ababd88f54788727ae13a546228ad57c72",
    "raw_has_think_close": false,
    "reasoning_chars": 12901,
    "text_chars": 53614,
    "text_sha256": "821c84d987e2c88b9fd53463cde3f56f1d66c21f7557970817da935d3f5067f6",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 150.4640983948484,
    "vllm_decode_tps": 66.46103692960672,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.344401407055557,
    "vllm_prefill_tps": 30.046565748502044,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 60.6729184499643
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 37, 40, 43. One new sample.PASS4,263 natural tokens2K 70.87610K 66.474 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-09T00:22:27.410864084Z",
    "FinishedAt": "2026-11-09T00:43:40.972906165Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "ae0dba47e379e10ae120c82dbe767b915856ebc0a56771c0751407ceb9b17f98",
  "retest_strict": "PASS",
  "retest_strict_tokens": 4263,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 4263,
      "decode_tps": 69.65298485444228,
      "prefill_seconds": 13.264510760083795,
      "client_total_seconds": 74.47932942770422,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 13.264510760083795,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 61.20340727549046,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 4263.0
      },
      "request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
      "summary_sha256": "2a4ec12838f2054993cfc40a0ce46452682e1e06ac8c785eefd538bcc5266ae6"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 70.8759028403918,
      "prefill_seconds": 15.0672946870327,
      "client_total_seconds": 43.29480273462832,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 15.0672946870327,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 28.218335426412523,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2000.0
      },
      "request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
      "summary_sha256": "304d5e8b0f2afd6fcf019e0b454f8917c037eece212906cfcf2c1686136c1ee7"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 66.47442948531119,
      "prefill_seconds": 14.876342931762338,
      "client_total_seconds": 165.31903470959514,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 14.876342931762338,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 150.43378450069577,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 10000.0
      },
      "request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
      "summary_sha256": "895a7e44c83b5069acf38a1a85397205ac0d8a1cefc38010e259fa29cda426f9"
    }
  },
  "retest_strict_TPS": 69.65298485444228,
  "retest_strict_prefill_seconds": 13.264510760083795,
  "retest_strict_total_seconds": 74.47932942770422,
  "retest_2K_TPS": 70.8759028403918,
  "retest_2K_prefill_seconds": 15.0672946870327,
  "retest_2K_total_seconds": 43.29480273462832,
  "retest_10K_TPS": 66.47442948531119,
  "retest_10K_prefill_seconds": 14.876342931762338,
  "retest_10K_total_seconds": 165.31903470959514,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33744068608,
    "1": 33634144256,
    "2": 33684496384,
    "3": 33634095104,
    "4": 33684230144,
    "5": 33633964032,
    "6": 33684738048,
    "7": 33634250752
  },
  "retest_sampled_peak_VRAM_GiB": 31.426612854003906,
  "retest_memory_observation_count": 604,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-09T00:25:47.689313425Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:25:47 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 105.144041 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-09T00:28:42.161027038Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:28:42 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-09T00:28:42.250244890Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-09T00:28:42.250362411Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 6.75,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_logged_context_capacity_x": 11.32,
  "retest_memory_log_source": "historical-recovered/fresh-row-37/result.json"
}
Fresh sampled VRAM: 31.427 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 22.79 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid; ai-info 10K gate cleared

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 4315,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r3.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-40.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 40,
    "original_host": "10.0.0.30",
    "logged_model_load_delta_GiB": 6.75,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 115.568181,
    "logged_available_KV_GiB": 22.79,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 373360,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 11.32,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "8e5636e5ce75ea4e080735843d07126b395cbb4f4b8ea2928b3f6e92369a61ae",
    "source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/vnext-release-repro/public-staging-20260629/runtime-full-current-20260630T111241Z/vnext-launch-runs/20260630T111242Z-2834957/gguf-dense27b-tp8",
    "source_lines": {
      "load": {
        "number": 4588,
        "text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-30 11:18:48 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 115.568181 seconds"
      },
      "KV": {
        "number": 5168,
        "text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-30 11:21:37 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB"
      },
      "tokens": {
        "number": 5174,
        "text": "(EngineCore_DP0 pid=200) INFO 06-30 11:21:37 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens"
      },
      "capacity": {
        "number": 5175,
        "text": "(EngineCore_DP0 pid=200) INFO 06-30 11:21:37 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x"
      },
      "engine": {
        "number": 63,
        "text": "(EngineCore_DP0 pid=200) INFO 06-30 11:16:27 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=6a9e13bd6fc8f0983b9b99948120bc37f49c13e9, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 22.79,
  "logged_GPU_KV_tokens": 373360,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-37/result.json",
  "retest_strict_tokens": 4263,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33744068608,
    "1": 33634144256,
    "2": 33684496384,
    "3": 33634095104,
    "4": 33684230144,
    "5": 33633964032,
    "6": 33684738048,
    "7": 33634250752
  },
  "retest_sampled_peak_VRAM_GiB": 31.426612854003906,
  "retest_memory_observation_count": 604,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-09T00:25:47.689313425Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:25:47 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 105.144041 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-09T00:28:42.161027038Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:28:42 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-09T00:28:42.250244890Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-09T00:28:42.250362411Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_memory_log_source": "historical-recovered/fresh-row-37/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.30 · Full vNext Release-Readiness Serving Replay / table 2HF FP16Qwen/Qwen3.6-27BTP8MTP off PASS3518 tokens 71.32666.810 14.080 / 42.12810K: 11.670 / 161.357Strict: 13.255 / 63.184Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.84 GiB (rank 0)Available KV: 22.64 GiB · engine report370,864 grouped KV tokens · 11.25× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-41
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-41 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row41-ae19602e82-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-41/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-41/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-41/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-41/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-41/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-41/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/opt/local-models/qwen36-27b-hf,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-41
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-41 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 71.149 · valid historical-model sample10K 66.668 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3920,
    "content_chars": 6472,
    "content_has_think_close": false,
    "elapsed_seconds": 70.2467966740951,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T08:20:47Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.014053838327527046,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 6472,
    "qwen_answer_text_sha256": "d79b6fa83b0fa444add4ed1d95c2f6f62c5c0cc2099f90436c3859434f444eee",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 6472,
    "qwen_visible_text_sha256": "d79b6fa83b0fa444add4ed1d95c2f6f62c5c0cc2099f90436c3859434f444eee",
    "raw_has_think_close": false,
    "reasoning_chars": 12896,
    "text_chars": 19369,
    "text_sha256": "9e353abcb76fb81d15044b5756245d235d27d4868bca1f06c1b821426adc19c9",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 55.93259923905134,
    "vllm_decode_tps": 70.08435247656276,
    "vllm_generation_tokens_delta": 3920.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.302407649345696,
    "vllm_prefill_tps": 30.134786433647577,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 55.803256313402514
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 41.84833710733801,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T08:21:29Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008913260884582996,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9935,
    "text_chars": 9935,
    "text_sha256": "83610c81afefe692098576ea3890219b4f89e9edc8bb008987daa4e2e22f54b7",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 28.109985790215433,
    "vllm_decode_tps": 71.14909324131224,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 13.728937095962465,
    "vllm_prefill_tps": 31.393544670457594,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 47.79162418975316
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 39962,
    "content_has_think_close": false,
    "elapsed_seconds": 164.7553375950083,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T08:24:14Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008609971031546593,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 39962,
    "qwen_answer_text_sha256": "d8c1f94bd6b8093da4fd146802c26246aae4ff3246e4039fbe101a47424784c6",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 39962,
    "qwen_visible_text_sha256": "d8c1f94bd6b8093da4fd146802c26246aae4ff3246e4039fbe101a47424784c6",
    "raw_has_think_close": false,
    "reasoning_chars": 12896,
    "text_chars": 52859,
    "text_sha256": "008e0ab44ea086b2f133d8b151934ce3e8f74780a735fe7804fd9c4647331b54",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 149.99608603771776,
    "vllm_decode_tps": 66.66840625084988,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.749934704042971,
    "vllm_prefill_tps": 29.22046833752169,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 60.696060874102905
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 35, 38, 41, 44. One new sample.PASS3,609 natural tokens2K 70.95110K 66.469 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-08T23:40:06.929336115Z",
    "FinishedAt": "2026-11-08T23:58:45.910937408Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "a96218fee0c78b5145db76c43cdf273ff01d169be52e0662ddcc70106f936f79",
  "retest_strict": "PASS",
  "retest_strict_tokens": 3609,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 3609,
      "decode_tps": 70.06009773846488,
      "prefill_seconds": 12.782165881246328,
      "client_total_seconds": 64.30384199228138,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 3609.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_sum": 12.782165881246328,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 51.512917002663016,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "ee97c35547770515d3bd63d45b2557687a8278bbeb38ce72abbaad2fee0ab250",
      "summary_sha256": "6ae03f5709d05f854c0ff4b3e846b625d2d4a691aac551f35ae21d6a4b36c17e"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 70.95081153473755,
      "prefill_seconds": 15.072314956225455,
      "client_total_seconds": 43.269196966663,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 2000.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_sum": 15.072314956225455,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 28.188542974181473,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "d476546705501abd0468f2fc6527e9ace49f732079e53b75abf7bcfb98635f44",
      "summary_sha256": "f70df1fbfc18c6af1f765a73663863bb438fbe1dd644049da3c9e283ae692d0e"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 66.46936099788869,
      "prefill_seconds": 14.515854343771935,
      "client_total_seconds": 164.97092120815068,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 10000.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_sum": 14.515854343771935,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 150.4452555263415,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "ee5df402413359190aaa7ed74f4a298f749233768156b8fc14f2ea4bef0e8017",
      "summary_sha256": "1dfc4acf114048076b24fd3ed43f4a13dd3800ec8892ff4352974a55fd56aba2"
    }
  },
  "retest_strict_TPS": 70.06009773846488,
  "retest_strict_prefill_seconds": 12.782165881246328,
  "retest_strict_total_seconds": 64.30384199228138,
  "retest_2K_TPS": 70.95081153473755,
  "retest_2K_prefill_seconds": 15.072314956225455,
  "retest_2K_total_seconds": 43.269196966663,
  "retest_10K_TPS": 66.46936099788869,
  "retest_10K_prefill_seconds": 14.515854343771935,
  "retest_10K_total_seconds": 164.97092120815068,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33776762880,
    "1": 33666486272,
    "2": 33716842496,
    "3": 33666523136,
    "4": 33716817920,
    "5": 33666486272,
    "6": 33716846592,
    "7": 33666510848
  },
  "retest_sampled_peak_VRAM_GiB": 31.457061767578125,
  "retest_memory_observation_count": 527,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 712,
        "text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
        "values": [
          "6.84"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 1267,
        "text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
        "values": [
          "22.64"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 1273,
        "text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
        "values": [
          "370,864"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 1274,
        "text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
        "values": [
          "131,072",
          "11.25"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 6.84,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 22.64,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 370864,
  "retest_logged_context_capacity_x": 11.25,
  "retest_memory_log_source": "historical-recovered/fresh-row-35/result.json"
}
Fresh sampled VRAM: 31.457 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 22.64 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid; ai-info 10K gate cleared

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 3518,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r3.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-41.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 41,
    "original_host": "10.0.0.30",
    "logged_model_load_delta_GiB": 6.84,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 24.431786,
    "logged_available_KV_GiB": 22.64,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 370864,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 11.25,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "31b012b6fd4429cc642f68c73f43900399682da4ad3cc8254abae19f76ce393f",
    "source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/vnext-release-repro/public-staging-20260629/runtime-full-current-20260630T111241Z/vnext-launch-runs/20260630T111242Z-2834957/hf-dense27b-tp8",
    "source_lines": {
      "load": {
        "number": 712,
        "text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-30 11:53:09 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 24.431786 seconds"
      },
      "KV": {
        "number": 1270,
        "text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-30 11:54:49 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB"
      },
      "tokens": {
        "number": 1273,
        "text": "(EngineCore_DP0 pid=184) INFO 06-30 11:54:49 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens"
      },
      "capacity": {
        "number": 1274,
        "text": "(EngineCore_DP0 pid=184) INFO 06-30 11:54:49 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x"
      },
      "engine": {
        "number": 55,
        "text": "(EngineCore_DP0 pid=184) INFO 06-30 11:52:18 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-27b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 22.64,
  "logged_GPU_KV_tokens": 370864,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-35/result.json",
  "retest_strict_tokens": 3609,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33776762880,
    "1": 33666486272,
    "2": 33716842496,
    "3": 33666523136,
    "4": 33716817920,
    "5": 33666486272,
    "6": 33716846592,
    "7": 33666510848
  },
  "retest_sampled_peak_VRAM_GiB": 31.457061767578125,
  "retest_memory_observation_count": 527,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 712,
        "text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
        "values": [
          "6.84"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 1267,
        "text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
        "values": [
          "22.64"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 1273,
        "text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
        "values": [
          "370,864"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 1274,
        "text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
        "values": [
          "131,072",
          "11.25"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 22.64,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 370864,
  "retest_memory_log_source": "historical-recovered/fresh-row-35/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.30 · Full vNext Release-Readiness Serving Replay / table 2HF FP16Qwen/Qwen3.6-35B-A3BTP8MTP off PASS3657 tokens 115.895109.001 0.813 / 18.07810K: 0.811 / 92.562Strict: 0.811 / 32.717Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 8.88 GiB (rank 0)Available KV: 19.91 GiB · engine report521,696 grouped KV tokens · 15.82× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-42
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-42 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row42-acca00650b-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-42/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-42/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-42/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-42/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-42/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-42/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-moe,dst=/opt/local-models/qwen36-35b-a3b-hf,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-moe "$MODEL_DIR/qwen36-moe"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-42
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-42 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 117.029 · valid historical-model sample10K 110.049 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 4563,
    "content_chars": 9559,
    "content_has_think_close": false,
    "elapsed_seconds": 40.47988819144666,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T09:03:15Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.01213598158210516,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 9559,
    "qwen_answer_text_sha256": "c0bf1c311f22fd6f7b5cc4f84da6468132661431aef27df3dfbb257bd7c5ebf4",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 9559,
    "qwen_visible_text_sha256": "c0bf1c311f22fd6f7b5cc4f84da6468132661431aef27df3dfbb257bd7c5ebf4",
    "raw_has_think_close": false,
    "reasoning_chars": 14042,
    "text_chars": 23602,
    "text_sha256": "66143a28ce4a77ff43265dad18efd33c26b176e6beae677a61343a6964631afc",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 39.66698031872511,
    "vllm_decode_tps": 115.03270385938605,
    "vllm_generation_tokens_delta": 4563.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8039803570136428,
    "vllm_prefill_tps": 536.0827490872222,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 112.72264336352971
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 17.905847259797156,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T09:03:33Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.005839169956743717,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 10261,
    "text_chars": 10261,
    "text_sha256": "c4fec0daf03f83f76b3d525966214f007fa4c81c71478fe6125229c7da45b99b",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 17.089729534462094,
    "vllm_decode_tps": 117.02935356390067,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8079051412642002,
    "vllm_prefill_tps": 533.478471650244,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 111.69535688436655
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 42369,
    "content_has_think_close": false,
    "elapsed_seconds": 91.69410692714155,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T09:05:05Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.006687283515930176,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 42369,
    "qwen_answer_text_sha256": "4b64ba6d64efec639f826c39f2d980a8d2fbc330e9c841ed671d422667c1738a",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 42369,
    "qwen_visible_text_sha256": "4b64ba6d64efec639f826c39f2d980a8d2fbc330e9c841ed671d422667c1738a",
    "raw_has_think_close": false,
    "reasoning_chars": 14042,
    "text_chars": 56412,
    "text_sha256": "24d73cff4cbffbb79f61919dbac79b0311246bb8408bdcad000d2265afb850c0",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 90.86887793801725,
    "vllm_decode_tps": 110.04867922789934,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8149959305301309,
    "vllm_prefill_tps": 528.836996424813,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 109.0582626857996
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 36, 39, 42, 45. One new sample.PASS2,911 natural tokens2K 117.36710K 110.169 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-09T00:00:14.441833328Z",
    "FinishedAt": "2026-11-09T00:20:09.78005595Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "d3e8150b92459eb9bfeb0628cfbc7f1ba14417d265985bd295006c8f223de819",
  "retest_strict": "PASS",
  "retest_strict_tokens": 2911,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 2911,
      "decode_tps": 116.74599429536063,
      "prefill_seconds": 0.8083759220317006,
      "client_total_seconds": 25.750953457318246,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 0.8083759220317006,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 24.93447434809059,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2911.0
      },
      "request_sha256": "e31c712c0771301f2e4d592bc966448d3143f2660e1b4cab33a91704f19e9370",
      "summary_sha256": "eebfaae9c73243c913cb60f1c6f4ec103cac551b4f8d5c5529c5364a6f3465ca"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 117.36665638338557,
      "prefill_seconds": 0.8091103108599782,
      "client_total_seconds": 17.857541773468256,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 0.8091103108599782,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 17.04061495512724,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2000.0
      },
      "request_sha256": "bca01af4a8338cd6274c1c9e0b281d8e59066ebae2ece3e3a636620066da3a03",
      "summary_sha256": "2c37396cb2c1b23f8ae028e9beb941f96a022ffb8a5da29d3363d63003b69672"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 110.16866681199451,
      "prefill_seconds": 0.8098680507391691,
      "client_total_seconds": 91.58954442292452,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 0.8098680507391691,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 90.76991026010364,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 10000.0
      },
      "request_sha256": "46fd7808353544eb86016bdd1746bafb67def6864a3f5352647b66cb9149f95a",
      "summary_sha256": "b44fdcf5c45447a4a3ba6eda737cc6adcf2f183f3569935f415e850a380531e5"
    }
  },
  "retest_strict_TPS": 116.74599429536063,
  "retest_strict_prefill_seconds": 0.8083759220317006,
  "retest_strict_total_seconds": 25.750953457318246,
  "retest_2K_TPS": 117.36665638338557,
  "retest_2K_prefill_seconds": 0.8091103108599782,
  "retest_2K_total_seconds": 17.857541773468256,
  "retest_10K_TPS": 110.16866681199451,
  "retest_10K_prefill_seconds": 0.8098680507391691,
  "retest_10K_total_seconds": 91.58954442292452,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 34334547968,
    "1": 34314465280,
    "2": 34314543104,
    "3": 34314469376,
    "4": 34334240768,
    "5": 34314461184,
    "6": 34314457088,
    "7": 34318581760
  },
  "retest_sampled_peak_VRAM_GiB": 31.976539611816406,
  "retest_memory_observation_count": 565,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 193,
        "text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
        "values": [
          "8.88"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 313,
        "text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
        "values": [
          "19.91"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 315,
        "text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
        "values": [
          "521,696"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 316,
        "text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
        "values": [
          "131,072",
          "15.82"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 8.88,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 19.91,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 521696,
  "retest_logged_context_capacity_x": 15.82,
  "retest_memory_log_source": "historical-recovered/fresh-row-36/result.json"
}
Fresh sampled VRAM: 31.977 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 19.91 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 3657,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "995ad96eacd98c81ed38be0c5b274b04031597b0",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r3.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-42.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 42,
    "original_host": "10.0.0.30",
    "logged_model_load_delta_GiB": 8.88,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 110.381889,
    "logged_available_KV_GiB": 19.91,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 521696,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 15.82,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "537cbbf045673c0c5d8d0ae112aca2b44ca8cf1d8a8055c940cdd6d0e50761ed",
    "source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/vnext-release-repro/public-staging-20260629/runtime-full-current-20260630T111241Z/vnext-launch-runs/20260630T111242Z-2834957/hf-moe35b-tp8",
    "source_lines": {
      "load": {
        "number": 193,
        "text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-30 12:26:46 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 110.381889 seconds"
      },
      "KV": {
        "number": 311,
        "text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-30 12:28:16 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB"
      },
      "tokens": {
        "number": 315,
        "text": "(EngineCore_DP0 pid=171) INFO 06-30 12:28:16 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens"
      },
      "capacity": {
        "number": 316,
        "text": "(EngineCore_DP0 pid=171) INFO 06-30 12:28:16 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x"
      },
      "engine": {
        "number": 33,
        "text": "(EngineCore_DP0 pid=171) INFO 06-30 12:24:26 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-35b-a3b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-35b-a3b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.6-35B-A3B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 19.91,
  "logged_GPU_KV_tokens": 521696,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-36/result.json",
  "retest_strict_tokens": 2911,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 34334547968,
    "1": 34314465280,
    "2": 34314543104,
    "3": 34314469376,
    "4": 34334240768,
    "5": 34314461184,
    "6": 34314457088,
    "7": 34318581760
  },
  "retest_sampled_peak_VRAM_GiB": 31.976539611816406,
  "retest_memory_observation_count": 565,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 193,
        "text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
        "values": [
          "8.88"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 313,
        "text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
        "values": [
          "19.91"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 315,
        "text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
        "values": [
          "521,696"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 316,
        "text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
        "values": [
          "131,072",
          "15.82"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 19.91,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 521696,
  "retest_memory_log_source": "historical-recovered/fresh-row-36/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.30 · Full vNext Release-Readiness Serving Replay / table 3GGUF F16Qwen/Qwen3.6-27BTP8MTP off PASS3742 tokens 71.08166.543 12.165 / 40.31110K: 12.343 / 162.632Strict: 14.023 / 67.416Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 22.79 GiB · engine report373,360 grouped KV tokens · 11.32× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-43
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-43 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row43-5690e3a7ec-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-43/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-43/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-43/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-43/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-43/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-43/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense"
python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-43
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-43 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 70.836 · valid historical-model sample10K 66.352 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3649,
    "content_chars": 6475,
    "content_has_think_close": false,
    "elapsed_seconds": 66.57352643553168,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T09:41:13Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.012507816776633263,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 6475,
    "qwen_answer_text_sha256": "925a28033b3d31f0cc40bce6a370158e0957324d09afef3a046da393cd2d5a6c",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 6475,
    "qwen_visible_text_sha256": "925a28033b3d31f0cc40bce6a370158e0957324d09afef3a046da393cd2d5a6c",
    "raw_has_think_close": false,
    "reasoning_chars": 11161,
    "text_chars": 17637,
    "text_sha256": "f364c8a84832f94e723a28a513e06a4560a7ef6759d7a9117df82e9d93eadfa5",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 52.22083283774555,
    "vllm_decode_tps": 69.87632716118766,
    "vllm_generation_tokens_delta": 3649.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.343986123800278,
    "vllm_prefill_tps": 30.047435648648786,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 54.811577444881344
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 43.17953444086015,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T09:41:56Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008289745077490807,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9518,
    "text_chars": 9518,
    "text_sha256": "2e522d4f025ae541c3176110880f955f89516600fa7c9c2e05d1f38f19146aee",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 28.23441171180457,
    "vllm_decode_tps": 70.83554707689613,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.937016525305808,
    "vllm_prefill_tps": 28.854490404413347,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 46.31823908938281
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 39271,
    "content_has_think_close": false,
    "elapsed_seconds": 164.38277336861938,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T09:44:41Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008772118017077446,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 39271,
    "qwen_answer_text_sha256": "b7d029235f4bef11afa9b73a697e4e253bf00a0aa9615344c274272fd62549e9",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 39271,
    "qwen_visible_text_sha256": "b7d029235f4bef11afa9b73a697e4e253bf00a0aa9615344c274272fd62549e9",
    "raw_has_think_close": false,
    "reasoning_chars": 11161,
    "text_chars": 50433,
    "text_sha256": "75f602e176c6e4d050f56384f17751b47349c8769a376212726722bb6d890950",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 150.71104567032307,
    "vllm_decode_tps": 66.35213733354865,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 13.661930229514837,
    "vllm_prefill_tps": 31.547518744377726,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 60.833625051303564
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 37, 40, 43. One new sample.PASS4,263 natural tokens2K 70.87610K 66.474 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-09T00:22:27.410864084Z",
    "FinishedAt": "2026-11-09T00:43:40.972906165Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "ae0dba47e379e10ae120c82dbe767b915856ebc0a56771c0751407ceb9b17f98",
  "retest_strict": "PASS",
  "retest_strict_tokens": 4263,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 4263,
      "decode_tps": 69.65298485444228,
      "prefill_seconds": 13.264510760083795,
      "client_total_seconds": 74.47932942770422,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 13.264510760083795,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 61.20340727549046,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 4263.0
      },
      "request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
      "summary_sha256": "2a4ec12838f2054993cfc40a0ce46452682e1e06ac8c785eefd538bcc5266ae6"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 70.8759028403918,
      "prefill_seconds": 15.0672946870327,
      "client_total_seconds": 43.29480273462832,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 15.0672946870327,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 28.218335426412523,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2000.0
      },
      "request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
      "summary_sha256": "304d5e8b0f2afd6fcf019e0b454f8917c037eece212906cfcf2c1686136c1ee7"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 66.47442948531119,
      "prefill_seconds": 14.876342931762338,
      "client_total_seconds": 165.31903470959514,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 14.876342931762338,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 150.43378450069577,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 10000.0
      },
      "request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
      "summary_sha256": "895a7e44c83b5069acf38a1a85397205ac0d8a1cefc38010e259fa29cda426f9"
    }
  },
  "retest_strict_TPS": 69.65298485444228,
  "retest_strict_prefill_seconds": 13.264510760083795,
  "retest_strict_total_seconds": 74.47932942770422,
  "retest_2K_TPS": 70.8759028403918,
  "retest_2K_prefill_seconds": 15.0672946870327,
  "retest_2K_total_seconds": 43.29480273462832,
  "retest_10K_TPS": 66.47442948531119,
  "retest_10K_prefill_seconds": 14.876342931762338,
  "retest_10K_total_seconds": 165.31903470959514,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33744068608,
    "1": 33634144256,
    "2": 33684496384,
    "3": 33634095104,
    "4": 33684230144,
    "5": 33633964032,
    "6": 33684738048,
    "7": 33634250752
  },
  "retest_sampled_peak_VRAM_GiB": 31.426612854003906,
  "retest_memory_observation_count": 604,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-09T00:25:47.689313425Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:25:47 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 105.144041 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-09T00:28:42.161027038Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:28:42 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-09T00:28:42.250244890Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-09T00:28:42.250362411Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 6.75,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_logged_context_capacity_x": 11.32,
  "retest_memory_log_source": "historical-recovered/fresh-row-37/result.json"
}
Fresh sampled VRAM: 31.427 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 22.79 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid; ai-info 10K gate cleared

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 3742,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r3.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-43.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 43,
    "original_host": "10.0.0.30",
    "logged_model_load_delta_GiB": 6.75,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 108.890125,
    "logged_available_KV_GiB": 22.79,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 373360,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 11.32,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "a9babfca90bf4dc23965294f0abf27b401f6228b3748e1864c5814dd2d3dc3c4",
    "source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/vnext-release-repro/public-staging-20260629/runtime-full-current-proof/vnext-launch-runs/20260630T130329Z-2951956/gguf-dense27b-tp8",
    "source_lines": {
      "load": {
        "number": 4588,
        "text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-30 13:09:21 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 108.890125 seconds"
      },
      "KV": {
        "number": 5163,
        "text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-30 13:12:06 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB"
      },
      "tokens": {
        "number": 5174,
        "text": "(EngineCore_DP0 pid=200) INFO 06-30 13:12:08 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens"
      },
      "capacity": {
        "number": 5175,
        "text": "(EngineCore_DP0 pid=200) INFO 06-30 13:12:08 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x"
      },
      "engine": {
        "number": 63,
        "text": "(EngineCore_DP0 pid=200) INFO 06-30 13:07:07 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=6a9e13bd6fc8f0983b9b99948120bc37f49c13e9, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 22.79,
  "logged_GPU_KV_tokens": 373360,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-37/result.json",
  "retest_strict_tokens": 4263,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33744068608,
    "1": 33634144256,
    "2": 33684496384,
    "3": 33634095104,
    "4": 33684230144,
    "5": 33633964032,
    "6": 33684738048,
    "7": 33634250752
  },
  "retest_sampled_peak_VRAM_GiB": 31.426612854003906,
  "retest_memory_observation_count": 604,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 4589,
        "text": "2026-11-09T00:25:47.689313425Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:25:47 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 105.144041 seconds",
        "values": [
          "6.75"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 5172,
        "text": "2026-11-09T00:28:42.161027038Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:28:42 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
        "values": [
          "22.79"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 5175,
        "text": "2026-11-09T00:28:42.250244890Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
        "values": [
          "373,360"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 5176,
        "text": "2026-11-09T00:28:42.250362411Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
        "values": [
          "131,072",
          "11.32"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 22.79,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 373360,
  "retest_memory_log_source": "historical-recovered/fresh-row-37/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.30 · Full vNext Release-Readiness Serving Replay / table 3HF FP16Qwen/Qwen3.6-27BTP8MTP off PASS3742 tokens 71.25466.796 14.435 / 42.51210K: 13.400 / 163.117Strict: 14.229 / 67.462Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.84 GiB (rank 0)Available KV: 22.64 GiB · engine report370,864 grouped KV tokens · 11.25× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-44
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-44 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row44-fb2c570fcb-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-44/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-44/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-44/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-44/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-44/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-44/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/opt/local-models/qwen36-27b-hf,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-44
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-44 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 71.545 · valid historical-model sample10K 67.124 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3547,
    "content_chars": 6457,
    "content_has_think_close": false,
    "elapsed_seconds": 64.16236412711442,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T10:11:23Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.013237979263067245,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 6457,
    "qwen_answer_text_sha256": "3ad006f79090660185525516d2562a6210d7ac8433bbec5df068b059fa1ec551",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 6457,
    "qwen_visible_text_sha256": "3ad006f79090660185525516d2562a6210d7ac8433bbec5df068b059fa1ec551",
    "raw_has_think_close": false,
    "reasoning_chars": 11482,
    "text_chars": 17940,
    "text_sha256": "fec7940db9e37cea727ec405d4b1b0ceaa254e277632e0b30b189fde4134c4c5",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 50.13813655637205,
    "vllm_decode_tps": 70.74455182457739,
    "vllm_generation_tokens_delta": 3547.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.016393848694861,
    "vllm_prefill_tps": 30.7497067114829,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 55.28162885290367
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 42.49045191332698,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T10:12:05Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008615121245384216,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9954,
    "text_chars": 9954,
    "text_sha256": "901ca3921ce1cca130da42852733cff33e53103148af2ea7b9fdac37416bf1a1",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 27.954272541217506,
    "vllm_decode_tps": 71.54541392737288,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.52750407718122,
    "vllm_prefill_tps": 29.66786295224549,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 47.06939818102305
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 41009,
    "content_has_think_close": false,
    "elapsed_seconds": 163.0703025655821,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T10:14:48Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.00643538311123848,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen3.6-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 41009,
    "qwen_answer_text_sha256": "baab8ed253ef12664fea03f14001f648e3b72046331a0e03aacf6420570d88e6",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 41009,
    "qwen_visible_text_sha256": "baab8ed253ef12664fea03f14001f648e3b72046331a0e03aacf6420570d88e6",
    "raw_has_think_close": false,
    "reasoning_chars": 11482,
    "text_chars": 52492,
    "text_sha256": "1690d20b0f764e652f782470928e6a6525e42a044bd9c10a0fbd3bdb8daa4486",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 148.97759592998773,
    "vllm_decode_tps": 67.1241869462004,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.08258059155196,
    "vllm_prefill_tps": 30.605186116140807,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 61.323244285870466
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 35, 38, 41, 44. One new sample.PASS3,609 natural tokens2K 70.95110K 66.469 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-08T23:40:06.929336115Z",
    "FinishedAt": "2026-11-08T23:58:45.910937408Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "a96218fee0c78b5145db76c43cdf273ff01d169be52e0662ddcc70106f936f79",
  "retest_strict": "PASS",
  "retest_strict_tokens": 3609,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 3609,
      "decode_tps": 70.06009773846488,
      "prefill_seconds": 12.782165881246328,
      "client_total_seconds": 64.30384199228138,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 3609.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_sum": 12.782165881246328,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 51.512917002663016,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "ee97c35547770515d3bd63d45b2557687a8278bbeb38ce72abbaad2fee0ab250",
      "summary_sha256": "6ae03f5709d05f854c0ff4b3e846b625d2d4a691aac551f35ae21d6a4b36c17e"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 70.95081153473755,
      "prefill_seconds": 15.072314956225455,
      "client_total_seconds": 43.269196966663,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 2000.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_sum": 15.072314956225455,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 28.188542974181473,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "d476546705501abd0468f2fc6527e9ace49f732079e53b75abf7bcfb98635f44",
      "summary_sha256": "f70df1fbfc18c6af1f765a73663863bb438fbe1dd644049da3c9e283ae692d0e"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 66.46936099788869,
      "prefill_seconds": 14.515854343771935,
      "client_total_seconds": 164.97092120815068,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:generation_tokens_total": 10000.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_prefill_time_seconds_sum": 14.515854343771935,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:request_decode_time_seconds_sum": 150.4452555263415,
        "vllm:request_decode_time_seconds_count": 1.0
      },
      "request_sha256": "ee5df402413359190aaa7ed74f4a298f749233768156b8fc14f2ea4bef0e8017",
      "summary_sha256": "1dfc4acf114048076b24fd3ed43f4a13dd3800ec8892ff4352974a55fd56aba2"
    }
  },
  "retest_strict_TPS": 70.06009773846488,
  "retest_strict_prefill_seconds": 12.782165881246328,
  "retest_strict_total_seconds": 64.30384199228138,
  "retest_2K_TPS": 70.95081153473755,
  "retest_2K_prefill_seconds": 15.072314956225455,
  "retest_2K_total_seconds": 43.269196966663,
  "retest_10K_TPS": 66.46936099788869,
  "retest_10K_prefill_seconds": 14.515854343771935,
  "retest_10K_total_seconds": 164.97092120815068,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33776762880,
    "1": 33666486272,
    "2": 33716842496,
    "3": 33666523136,
    "4": 33716817920,
    "5": 33666486272,
    "6": 33716846592,
    "7": 33666510848
  },
  "retest_sampled_peak_VRAM_GiB": 31.457061767578125,
  "retest_memory_observation_count": 527,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 712,
        "text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
        "values": [
          "6.84"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 1267,
        "text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
        "values": [
          "22.64"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 1273,
        "text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
        "values": [
          "370,864"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 1274,
        "text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
        "values": [
          "131,072",
          "11.25"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 6.84,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 22.64,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 370864,
  "retest_logged_context_capacity_x": 11.25,
  "retest_memory_log_source": "historical-recovered/fresh-row-35/result.json"
}
Fresh sampled VRAM: 31.457 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 22.64 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid; ai-info 10K gate cleared

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 3742,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r3.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-44.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 44,
    "original_host": "10.0.0.30",
    "logged_model_load_delta_GiB": 6.84,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 23.129179,
    "logged_available_KV_GiB": 22.64,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 370864,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 11.25,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "b5013067f3d140ab979a82911c5da3b6999e4ab982bb451aba02c56a7eb79a3a",
    "source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/vnext-release-repro/public-staging-20260629/runtime-full-current-proof/vnext-launch-runs/20260630T130329Z-2951956/hf-dense27b-tp8",
    "source_lines": {
      "load": {
        "number": 712,
        "text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-30 13:43:37 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 23.129179 seconds"
      },
      "KV": {
        "number": 1269,
        "text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-30 13:45:17 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB"
      },
      "tokens": {
        "number": 1273,
        "text": "(EngineCore_DP0 pid=184) INFO 06-30 13:45:17 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens"
      },
      "capacity": {
        "number": 1274,
        "text": "(EngineCore_DP0 pid=184) INFO 06-30 13:45:17 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x"
      },
      "engine": {
        "number": 55,
        "text": "(EngineCore_DP0 pid=184) INFO 06-30 13:42:48 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-27b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 22.64,
  "logged_GPU_KV_tokens": 370864,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-35/result.json",
  "retest_strict_tokens": 3609,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 33776762880,
    "1": 33666486272,
    "2": 33716842496,
    "3": 33666523136,
    "4": 33716817920,
    "5": 33666486272,
    "6": 33716846592,
    "7": 33666510848
  },
  "retest_sampled_peak_VRAM_GiB": 31.457061767578125,
  "retest_memory_observation_count": 527,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 712,
        "text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
        "values": [
          "6.84"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 1267,
        "text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
        "values": [
          "22.64"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 1273,
        "text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
        "values": [
          "370,864"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 1274,
        "text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
        "values": [
          "131,072",
          "11.25"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 22.64,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 370864,
  "retest_memory_log_source": "historical-recovered/fresh-row-35/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50Historical vNext Qwen3.6Host 10.0.0.30 · Full vNext Release-Readiness Serving Replay / table 3HF FP16Qwen/Qwen3.6-35B-A3BTP8MTP off PASS3719 tokens 116.704109.754 0.811 / 17.95610K: 0.806 / 91.928Strict: 0.811 / 33.017Recovered original raw result — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 8.88 GiB (rank 0)Available KV: 19.91 GiB · engine report521,696 grouped KV tokens · 15.82× reported context capacityOriginal memory log reports · rounded, not physical usage Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1e
Runtime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-45
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-45 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained historical ports
The launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling.
Retained lab Docker command · not portable
docker \
  run \
  -d \
  --name \
  historical-row45-ff4f548663-reproduction \
  --network \
  host \
  --ipc \
  host \
  --privileged \
  --device \
  /dev/kfd \
  --device \
  /dev/dri \
  --group-add \
  video \
  --cap-add \
  SYS_PTRACE \
  --cap-add \
  IPC_LOCK \
  --security-opt \
  seccomp=unconfined \
  --security-opt \
  label=disable \
  --ulimit \
  memlock=-1:-1 \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-45/hf,dst=/root/.cache/huggingface \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-45/launch,dst=/runwork \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-45/cache/vllm,dst=/root/.cache/vllm \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-45/cache/triton,dst=/root/.triton \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-45/cache/inductor,dst=/tmp/torchinductor_root \
  --mount \
  type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-45/overlay,dst=/opt/vllm_patch_bundle,readonly \
  --mount \
  type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-moe,dst=/opt/local-models/qwen36-35b-a3b-hf,readonly \
  -e \
  HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e \
  LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \
  -e \
  VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e \
  HF_HUB_OFFLINE=1 \
  --entrypoint \
  /runwork/container_entrypoint.sh \
  sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Historical launch provenance
Recovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv.
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-model-collection
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-moe "$MODEL_DIR/qwen36-moe"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-historical-r6.tar.gz.sha256
mkdir -p launch-tools-historical-r6
tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-model-collection
RUN_DIR=/absolute/path/to/new-runs/historical-row-45
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-historical-r6/portable/launch.py --config historical-row-45 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-historical-r6.tar.gz · SHA-256
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 116.069 · valid historical-model sample10K 109.160 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 3506,
    "content_chars": 6574,
    "content_has_think_close": false,
    "elapsed_seconds": 31.27900091558695,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-10T10:53:44Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.011204360984265804,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 6574,
    "qwen_answer_text_sha256": "4cc902e2dbd5bf26afa3078ea66d42bfcc2b84060bb1c08945302077e2c2901e",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 6574,
    "qwen_visible_text_sha256": "4cc902e2dbd5bf26afa3078ea66d42bfcc2b84060bb1c08945302077e2c2901e",
    "raw_has_think_close": false,
    "reasoning_chars": 11741,
    "text_chars": 18316,
    "text_sha256": "74a69c47113596ca51a504d2471f356d634a7eba1608ff1fc76943610f355128",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 30.453829997219145,
    "vllm_decode_tps": 115.12509265074857,
    "vllm_generation_tokens_delta": 3506.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.815642450004816,
    "vllm_prefill_tps": 528.4178134640431,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 112.08797907138046
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 18.055767958052456,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T10:54:03Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008598048239946365,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 10281,
    "text_chars": 10281,
    "text_sha256": "1952df1c990e0bd3a5ce8f3b98d46928f2d5a3c9c4516df616d4e9743568620b",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 17.23110121395439,
    "vllm_decode_tps": 116.06919228008047,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8160559171810746,
    "vllm_prefill_tps": 528.1500825198543,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 110.76792771409349
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 48989,
    "content_has_think_close": false,
    "elapsed_seconds": 92.42294359579682,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-10T10:55:35Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008693818934261799,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.6-35B-A3B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 48989,
    "qwen_answer_text_sha256": "e8d83b5df9c7028076f75b0d2e0919b919b8ba383facde99c261cc4b7ebb1d39",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 48989,
    "qwen_visible_text_sha256": "e8d83b5df9c7028076f75b0d2e0919b919b8ba383facde99c261cc4b7ebb1d39",
    "raw_has_think_close": false,
    "reasoning_chars": 11741,
    "text_chars": 60731,
    "text_sha256": "73352886b27a6286ec20b2c30a86c9440ba82d3da48a921453598027c9484fad",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 91.60833835601807,
    "vllm_decode_tps": 109.16036879892894,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 0.8050807006657124,
    "vllm_prefill_tps": 535.3500582533042,
    "vllm_prompt_tokens_delta": 431.0,
    "wall_completion_tps": 108.19824181032443
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Historical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 36, 39, 42, 45. One new sample.PASS2,911 natural tokens2K 117.36710K 110.169 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 137,
    "Error": "",
    "StartedAt": "2026-11-09T00:00:14.441833328Z",
    "FinishedAt": "2026-11-09T00:20:09.78005595Z"
  },
  "retest_historical_limits": [
    "One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
    "Historical published measurements remain separate from this fresh run.",
    "Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
    "Fresh root-SSD caches and the current host state differ from the historical runs.",
    "Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
  ],
  "retest_historical_current_Qwen38_official_eligible": false,
  "retest_historical_verified_regular_files": 173,
  "retest_historical_manifest_sha256": "d3e8150b92459eb9bfeb0628cfbc7f1ba14417d265985bd295006c8f223de819",
  "retest_strict": "PASS",
  "retest_strict_tokens": 2911,
  "retest_historical_2K_eligible": true,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 2911,
      "decode_tps": 116.74599429536063,
      "prefill_seconds": 0.8083759220317006,
      "client_total_seconds": 25.750953457318246,
      "strict_pass": true,
      "finish_reason": "stop",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 0.8083759220317006,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 24.93447434809059,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2911.0
      },
      "request_sha256": "e31c712c0771301f2e4d592bc966448d3143f2660e1b4cab33a91704f19e9370",
      "summary_sha256": "eebfaae9c73243c913cb60f1c6f4ec103cac551b4f8d5c5529c5364a6f3465ca"
    },
    "2K": {
      "completion_tokens": 2000,
      "decode_tps": 117.36665638338557,
      "prefill_seconds": 0.8091103108599782,
      "client_total_seconds": 17.857541773468256,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 0.8091103108599782,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 17.04061495512724,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 2000.0
      },
      "request_sha256": "bca01af4a8338cd6274c1c9e0b281d8e59066ebae2ece3e3a636620066da3a03",
      "summary_sha256": "2c37396cb2c1b23f8ae028e9beb941f96a022ffb8a5da29d3363d63003b69672"
    },
    "10K": {
      "completion_tokens": 10000,
      "decode_tps": 110.16866681199451,
      "prefill_seconds": 0.8098680507391691,
      "client_total_seconds": 91.58954442292452,
      "strict_pass": null,
      "finish_reason": "length",
      "raw_counter_deltas": {
        "vllm:request_prefill_time_seconds_sum": 0.8098680507391691,
        "vllm:request_prefill_time_seconds_count": 1.0,
        "vllm:prompt_tokens_total": 431.0,
        "vllm:request_decode_time_seconds_sum": 90.76991026010364,
        "vllm:request_decode_time_seconds_count": 1.0,
        "vllm:generation_tokens_total": 10000.0
      },
      "request_sha256": "46fd7808353544eb86016bdd1746bafb67def6864a3f5352647b66cb9149f95a",
      "summary_sha256": "b44fdcf5c45447a4a3ba6eda737cc6adcf2f183f3569935f415e850a380531e5"
    }
  },
  "retest_strict_TPS": 116.74599429536063,
  "retest_strict_prefill_seconds": 0.8083759220317006,
  "retest_strict_total_seconds": 25.750953457318246,
  "retest_2K_TPS": 117.36665638338557,
  "retest_2K_prefill_seconds": 0.8091103108599782,
  "retest_2K_total_seconds": 17.857541773468256,
  "retest_10K_TPS": 110.16866681199451,
  "retest_10K_prefill_seconds": 0.8098680507391691,
  "retest_10K_total_seconds": 91.58954442292452,
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 34334547968,
    "1": 34314465280,
    "2": 34314543104,
    "3": 34314469376,
    "4": 34334240768,
    "5": 34314461184,
    "6": 34314457088,
    "7": 34318581760
  },
  "retest_sampled_peak_VRAM_GiB": 31.976539611816406,
  "retest_memory_observation_count": 565,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 193,
        "text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
        "values": [
          "8.88"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 313,
        "text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
        "values": [
          "19.91"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 315,
        "text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
        "values": [
          "521,696"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 316,
        "text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
        "values": [
          "131,072",
          "15.82"
        ]
      }
    ]
  },
  "retest_logged_model_load_delta_GiB": 8.88,
  "retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
  "retest_KV_budget_GiB": 19.91,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 521696,
  "retest_logged_context_capacity_x": 15.82,
  "retest_memory_log_source": "historical-recovered/fresh-row-36/result.json"
}
Fresh sampled VRAM: 31.977 GiB max/card. Logged load/KV budgets and source lines are retained in the result.Fresh logged KV available: 19.91 GiB minimum/worker; rounded engine report.
Evidence and limits

strict-valid

Pinned published result table
Recovered original result / scorer replay
Original raw evidence
{
  "strict_tokens": 3719,
  "source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
  "source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
  "model_revision": "995ad96eacd98c81ed38be0c5b274b04031597b0",
  "comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
  "historical_reported_VRAM_capacity_bytes": 34342961152,
  "historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
  "historical_result_source_copy": "historical-published-results.md",
  "historical_source_download": "historical-reproduction-r3.tar.gz",
  "strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
  "historical_raw_source": "historical-recovered/row-45.json",
  "historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
  "historical_memory_reports": {
    "row": 45,
    "original_host": "10.0.0.30",
    "logged_model_load_delta_GiB": 8.88,
    "model_load_report_rank": 0,
    "logged_model_load_seconds": 116.951902,
    "logged_available_KV_GiB": 19.91,
    "KV_memory_report_rank": 0,
    "logged_GPU_KV_tokens": 521696,
    "logged_context_tokens": 131072,
    "logged_context_capacity_x": 15.82,
    "engine_FP16_TP8_context131072_MTPoff": true,
    "source_log_sha256": "5fb63afa234d5088372f07d9632c05c1e4329772bf04a0e3c8fa2fff842b58a7",
    "source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
    "original_launch_path": "/usr/share/ollama/vnext-release-repro/public-staging-20260629/runtime-full-current-proof/vnext-launch-runs/20260630T130329Z-2951956/hf-moe35b-tp8",
    "source_lines": {
      "load": {
        "number": 193,
        "text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-30 14:17:32 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 116.951902 seconds"
      },
      "KV": {
        "number": 307,
        "text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-30 14:18:54 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB"
      },
      "tokens": {
        "number": 315,
        "text": "(EngineCore_DP0 pid=171) INFO 06-30 14:18:55 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens"
      },
      "capacity": {
        "number": 316,
        "text": "(EngineCore_DP0 pid=171) INFO 06-30 14:18:55 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x"
      },
      "engine": {
        "number": 33,
        "text": "(EngineCore_DP0 pid=171) INFO 06-30 14:15:05 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-35b-a3b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-35b-a3b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.6-35B-A3B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
      }
    },
    "limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
  },
  "logged_available_KV_GiB": 19.91,
  "logged_GPU_KV_tokens": 521696,
  "historical_memory_source": "historical-recovered/memory-reports.json",
  "historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
  "historical_binding_source": "historical-recovered/launch-bindings.json",
  "historical_fresh_result_source": "historical-recovered/fresh-row-36/result.json",
  "retest_strict_tokens": 2911,
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 34334547968,
    "1": 34314465280,
    "2": 34314543104,
    "3": 34314469376,
    "4": 34334240768,
    "5": 34314461184,
    "6": 34314457088,
    "7": 34318581760
  },
  "retest_sampled_peak_VRAM_GiB": 31.976539611816406,
  "retest_memory_observation_count": 565,
  "retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
  "retest_historical_memory_log_observations": {
    "model_load_delta_GiB": [
      {
        "line": 193,
        "text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
        "values": [
          "8.88"
        ]
      }
    ],
    "available_KV_GiB": [
      {
        "line": 313,
        "text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
        "values": [
          "19.91"
        ]
      }
    ],
    "grouped_GPU_KV_tokens": [
      {
        "line": 315,
        "text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
        "values": [
          "521,696"
        ]
      }
    ],
    "context_capacity": [
      {
        "line": 316,
        "text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
        "values": [
          "131,072",
          "15.82"
        ]
      }
    ]
  },
  "retest_KV_budget_GiB": 19.91,
  "retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
  "retest_logged_GPU_KV_tokens": 521696,
  "retest_memory_log_source": "historical-recovered/fresh-row-36/result.json",
  "sampled_peak_VRAM_GiB": null
}
MI50New Qwen3.8 TP88-bit RTNQwen/Qwen3.8-27BTP8MTP off PASS17039 tokens 59.39556.660 15.686 / 49.36710K: 15.894 / 192.394Strict: 15.589 / 329.176 2.96 / 28.64packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:2ab2203bcd66050fcae7dbf0b0c8ddbb0958e08da57ce2a0c9d7adc6511e9052
Runtime image config: sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp8-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-r2/portable/launch.py --config mi50-w8-tp8-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-w8-tp8-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-15 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD128:/dev/dri/renderD128:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \
  --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \
  -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \
  -e MI50_RTN_TP=8 \
  -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e HF_HUB_OFFLINE=1 \
  -e MI50_RTN_BITS=8 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp8-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp8-mtp0/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/usr/share/ollama/vnext-offload/models/Qwen3.8-27B-1d4bf0f2ff60,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp8-mtp0/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp8-mtp0,dst=/runwork \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp8-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp8-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-r2/portable/launch.py --config mi50-w8-tp8-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 58.721 · eligible10K 55.999 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 11995,
    "content_chars": 19449,
    "content_has_think_close": false,
    "elapsed_seconds": 231.85842335037887,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T22:31:49Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.012737645767629147,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 19449,
    "qwen_answer_text_sha256": "dbbbe18db3bceb6753e547e32d4d94d1551fdf9990d6039eec034a61a9913764",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 19449,
    "qwen_visible_text_sha256": "dbbbe18db3bceb6753e547e32d4d94d1551fdf9990d6039eec034a61a9913764",
    "raw_has_think_close": false,
    "reasoning_chars": 37407,
    "text_chars": 56857,
    "text_sha256": "39538347312b140fcaf3edd5d34f8fff007518c4ba34cdfe7ceb6d41df4b81fb",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 216.69863117672503,
    "vllm_decode_tps": 55.35337226111813,
    "vllm_generation_tokens_delta": 11995.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.147749536670744,
    "vllm_prefill_tps": 31.22576055637361,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 51.73415667488364
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 48.91990517452359,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T22:32:38Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.008235195651650429,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 9537,
    "text_chars": 9537,
    "text_sha256": "06336e353aafc8293a9164528c60db07e93500459fdef68e30cfa9d2a6a86cae",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 34.05907580256462,
    "vllm_decode_tps": 58.72149942041004,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.851454346440732,
    "vllm_prefill_tps": 31.848732721139744,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 40.88315365422164
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 9817,
    "content_has_think_close": false,
    "elapsed_seconds": 193.18140136823058,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T22:35:52Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.008163412101566792,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 9817,
    "qwen_answer_text_sha256": "0c8abe5c6cf14820798890db3fbdc8867f8704107e61298d5fe5f1455d634785",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length"
    ],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 9817,
    "qwen_visible_text_sha256": "0c8abe5c6cf14820798890db3fbdc8867f8704107e61298d5fe5f1455d634785",
    "raw_has_think_close": false,
    "reasoning_chars": 37407,
    "text_chars": 47225,
    "text_sha256": "1f9bb7602a5e39f2106774666188a115847e2d11885444f49d21544ae12b5e62",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 178.5761641394347,
    "vllm_decode_tps": 55.99851496525518,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 14.595634530298412,
    "vllm_prefill_tps": 32.406950106767944,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 51.764817571328265
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completedVerification · Raw retestPASS17,039 natural tokens2K 59.39510K 56.660 · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T16:44:04.393426541Z",
      "FinishedAt": "2026-10-26T17:08:54.166149865Z"
    },
    "devices_after": [
      {
        "card": 0,
        "render": 128,
        "pci": "0000:06:00.0",
        "actual_card": "card1",
        "busy_percent": 0,
        "vram_bytes": 17765081088,
        "dpm": "auto"
      },
      {
        "card": 1,
        "render": 129,
        "pci": "0000:09:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 20133625856,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:45:00.0",
        "actual_card": "card3",
        "busy_percent": 0,
        "vram_bytes": 20133601280,
        "dpm": "auto"
      },
      {
        "card": 3,
        "render": 131,
        "pci": "0000:48:00.0",
        "actual_card": "card4",
        "busy_percent": 0,
        "vram_bytes": 20133044224,
        "dpm": "auto"
      },
      {
        "card": 4,
        "render": 132,
        "pci": "0000:89:00.0",
        "actual_card": "card5",
        "busy_percent": 0,
        "vram_bytes": 20133081088,
        "dpm": "auto"
      },
      {
        "card": 5,
        "render": 133,
        "pci": "0000:8c:00.0",
        "actual_card": "card6",
        "busy_percent": 0,
        "vram_bytes": 20132954112,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:c5:00.0",
        "actual_card": "card7",
        "busy_percent": 0,
        "vram_bytes": 20133105664,
        "dpm": "auto"
      },
      {
        "card": 7,
        "render": 135,
        "pci": "0000:c8:00.0",
        "actual_card": "card8",
        "busy_percent": 0,
        "vram_bytes": 20081528832,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T17:08:54.294353+00:00"
  },
  "retest_strict": "PASS",
  "retest_official_2K_eligible": true,
  "retest_2K_TPS": 59.39478915627555,
  "retest_10K_TPS": 56.66006826564521,
  "retest_strict_tokens": 17039,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 17039,
      "elapsed_seconds": 329.1763630043715,
      "qwen_gate_valid": true,
      "vllm_decode_seconds_delta": 313.5765574136749,
      "vllm_decode_tps": 54.337607825453276,
      "vllm_generation_tokens_delta": 17039.0,
      "vllm_prefill_seconds_delta": 15.588659266009927,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 49.36692126002163,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 33.67298762081191,
      "vllm_decode_tps": 59.39478915627555,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 15.685871358029544,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 192.3944482197985,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 176.49113928200677,
      "vllm_decode_tps": 56.66006826564521,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 15.894322982057929,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 15.685871358029544,
  "retest_2K_total_seconds": 49.36692126002163,
  "retest_10K_prefill_seconds": 15.894322982057929,
  "retest_10K_total_seconds": 192.3944482197985,
  "retest_strict_prefill_seconds": 15.588659266009927,
  "retest_strict_total_seconds": 329.1763630043715,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": "Complete record: chart-details/row-46-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30757179392,
    "1": 30703603712,
    "2": 30703562752,
    "3": 30703595520,
    "4": 30703566848,
    "5": 30703587328,
    "6": 30703579136,
    "7": 30653231104
  },
  "retest_memory_observation_count": 24,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w8-tp8-mtp0/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 6.95,
  "retest_logged_model_load_seconds": 17.205276,
  "retest_logged_available_KV_GiB": 22.52,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 22.52,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 368800,
  "retest_logged_context_capacity_x": 11.14,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-w8-tp8-mtp0",
    "image": "sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7",
    "container": "c61c4fe370d271c8db3d197080e59e6d5b1f4dacc57b36c94291766d55ea0e1f",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 6.95,
    "retest_logged_model_load_seconds": 17.205276,
    "retest_logged_available_KV_GiB": 22.52,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 22.52,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 368800,
    "retest_logged_context_capacity_x": 11.14,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w8-tp8-mtp0/service-final.log",
    "source_log_sha256": "dcee58767022051e10ffee4de33d2fdcc25ce21dcdc0a9f7092717ef867728eb",
    "sealed_index_sha256": "21dc9b4b2d45cc245fe333cd4281f9c38eca3601458bec384de9e343d1c62f9a",
    "source_lines": {
      "model_load": {
        "line": 196,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:45:52 [gpu_model_runner.py:4584] Model loading took 6.95 GiB memory and 17.205276 seconds"
      },
      "automatic_KV_budget": {
        "line": 236,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:46:57 [gpu_worker.py:453] Available KV cache memory: 22.52 GiB"
      },
      "grouped_KV_tokens": {
        "line": 243,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:46:57 [kv_cache_utils.py:1314] GPU KV cache size: 368,800 tokens"
      },
      "context_capacity": {
        "line": 244,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:46:57 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.14x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w8-tp8-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.644855499267578
}
Complete retest loaded admission
Retest KV budget: 22.520 GiB (automatic, rounded)368,800 grouped KV tokens · 11.14× reported capacityRetest sampled VRAM: 28.645 GiB max/cardRetest memory reports and limits
Evidence and limits

TP8 GPU engineering passed; full model test pending

{
  "retest_strict_tokens": 17039,
  "retest_MTP_accounting_by_tier": {},
  "strict_tokens": 17039,
  "source_revision": "research-tp8-shapes-20261026",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "source": "retest-evidence/mi50-w8-tp8-mtp0/result.json",
  "comparison_limit": "One first full-model sample; TP8-specific padding/dispatch compatibility and fresh root-SSD caches. No isolated TP attribution or repeatability. Failed strict makes fixed speeds diagnostic.",
  "source_revision_type": "Preparation label, not a Git commit; exact source hashes are in the linked compatibility receipt.",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 30757179392,
    "1": 30703603712,
    "2": 30703562752,
    "3": 30703595520,
    "4": 30703566848,
    "5": 30703587328,
    "6": 30703579136,
    "7": 30653231104
  },
  "retest_memory_observation_count": 24,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w8-tp8-mtp0/runtime-observations.jsonl",
  "sampled_peak_VRAM_bytes_by_PCI": {
    "0000:06:00.0": 30757179392,
    "0000:09:00.0": 30703603712,
    "0000:45:00.0": 30703562752,
    "0000:48:00.0": 30703595520,
    "0000:89:00.0": 30703566848,
    "0000:8c:00.0": 30703587328,
    "0000:c5:00.0": 30703579136,
    "0000:c8:00.0": 30653231104
  },
  "retest_logged_available_KV_GiB": 22.52,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 22.52,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 368800,
  "retest_memory_log_report": {
    "profile": "mi50-w8-tp8-mtp0",
    "image": "sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7",
    "container": "c61c4fe370d271c8db3d197080e59e6d5b1f4dacc57b36c94291766d55ea0e1f",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 6.95,
    "retest_logged_model_load_seconds": 17.205276,
    "retest_logged_available_KV_GiB": 22.52,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 22.52,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 368800,
    "retest_logged_context_capacity_x": 11.14,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w8-tp8-mtp0/service-final.log",
    "source_log_sha256": "dcee58767022051e10ffee4de33d2fdcc25ce21dcdc0a9f7092717ef867728eb",
    "sealed_index_sha256": "21dc9b4b2d45cc245fe333cd4281f9c38eca3601458bec384de9e343d1c62f9a",
    "source_lines": {
      "model_load": {
        "line": 196,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:45:52 [gpu_model_runner.py:4584] Model loading took 6.95 GiB memory and 17.205276 seconds"
      },
      "automatic_KV_budget": {
        "line": 236,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:46:57 [gpu_worker.py:453] Available KV cache memory: 22.52 GiB"
      },
      "grouped_KV_tokens": {
        "line": 243,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:46:57 [kv_cache_utils.py:1314] GPU KV cache size: 368,800 tokens"
      },
      "context_capacity": {
        "line": 244,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:46:57 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.14x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w8-tp8-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 28.644855499267578,
  "sampled_peak_VRAM_GiB": 28.644855499267578
}
MI50New Qwen3.8 TP84-bit RTNQwen/Qwen3.8-27BTP8MTP off FAIL14034 tokens 66.438*62.951 15.101 / 45.21210K: 15.105 / 173.970Strict: 14.373 / 243.245 1.48 / 27.28packed target projections / VRAM GiB Docker Hub · pinned runtime
Immutable docker pull
docker pull joe2gaan/localaiservers@sha256:2ab2203bcd66050fcae7dbf0b0c8ddbb0958e08da57ce2a0c9d7adc6511e9052
Runtime image config: sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7
Registry verification
Serve on your host · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp8-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Verifies model hashes, exact registry image, selected GPUs and peer access.
# Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh.
python3 launch-tools-r2/portable/launch.py --config mi50-w4-tp8-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serve
Download serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command.
Retained lab Docker command · not portable
docker run -d \
  --name serve-mi50-w4-tp8-mtp0-20261028 \
  --network host \
  --ipc host \
  --cpuset-cpus 0-15 \
  --security-opt label=disable \
  --shm-size 67108864 \
  --device /dev/kfd:/dev/kfd:rwm \
  --device /dev/dri/renderD128:/dev/dri/renderD128:rwm \
  --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \
  --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \
  --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \
  --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \
  --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
  --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
  --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
  --user root \
  --workdir /tmp \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \
  -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \
  -e MI50_RTN_TP=8 \
  -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e HF_HUB_OFFLINE=1 \
  -e MI50_RTN_BITS=4 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \
  -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e ROCM_VERSION=7.2.1 \
  -e AMDGPU_VERSION=not-installed \
  -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \
  -e TORCH_VERSION=2.11.0+rocm7.2 \
  -e TORCHVISION_VERSION=0.26.0+rocm7.2 \
  -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \
  -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \
  -e AMDSMI_VERSION=7.0.2 \
  -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \
  -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \
  -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \
  -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \
  -e SOURCE_DATE_EPOCH=1764000000 \
  -e VLLM_TARGET_DEVICE=rocm \
  -e PYTORCH_ROCM_ARCH=gfx906 \
  -e GPU_ARCHS=gfx906 \
  -e AMDGPU_TARGETS=gfx906 \
  -e CMAKE_HIP_ARCHITECTURES=gfx906 \
  -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \
  -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \
  -e PYTHONPATH=/opt/qwen36-python \
  -e PYTHONUNBUFFERED=1 \
  -e PYTHONHASHSEED=0 \
  -e PIP_NO_CACHE_DIR=1 \
  -e PIP_NO_COMPILE=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  -e PYTHONDONTWRITEBYTECODE=1 \
  -e LANG=C.UTF-8 \
  -e LC_ALL=C.UTF-8 \
  -e TZ=Etc/UTC \
  -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \
  -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \
  -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \
  -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \
  -e VLLM_GFX906_POSTED_GEMMA=1 \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp8-mtp0/cache/vllm,dst=/root/.cache/vllm \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp8-mtp0/cache/.triton,dst=/root/.triton \
  --mount type=bind,src=/usr/share/ollama/vnext-offload/models/Qwen3.8-27B-1d4bf0f2ff60,dst=/opt/local-models/Qwen3.8-27B,readonly \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp8-mtp0/cache/huggingface,dst=/root/.cache/huggingface \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp8-mtp0,dst=/runwork \
  --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp8-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
  --entrypoint /opt/vnext/profile/container_entrypoint.sh \
  sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7
Historical host paths and device addresses are retained as evidence. Use the serving command above.
Retained launch source
Pinned model acquisition · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR"
Host requirements and package guide
Run the complete benchmark · copy
#!/usr/bin/env bash
set -euo pipefail
# Run in the downloaded release directory on your Linux Docker host.
sha256sum -c launch-tools-r2.tar.gz.sha256
mkdir -p launch-tools-r2
tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2

# Edit these host choices; numerical/runtime settings remain pinned.
MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint
RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp8-mtp0
GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135

# Starts a fresh cohort, runs the original ladder, and stops only that new cohort.
python3 launch-tools-r2/portable/launch.py --config mi50-w4-tp8-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmark
Download benchmark script
launch-tools-r2.tar.gz · SHA-256
Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service.
All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 66.644 · eligible10K 63.164 · diagnostic
Fresh tier results and counters
{
  "strict": {
    "completion_tokens": 10166,
    "content_chars": 6892,
    "content_has_think_close": false,
    "elapsed_seconds": 176.45749805495143,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "stop",
    "finished_at_utc": "2026-11-09T23:04:03Z",
    "force_length": false,
    "has_reasoning_field": true,
    "max_tokens": null,
    "metrics_settle_elapsed_seconds": 0.008709434419870377,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 6892,
    "qwen_answer_text_sha256": "a53b93bd66e91cf5575d86b684573e1e556920dbc610ff1a106bf3d18e67e57d",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [],
    "qwen_gate_parser_split_answer": true,
    "qwen_gate_valid": true,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 6892,
    "qwen_visible_text_sha256": "a53b93bd66e91cf5575d86b684573e1e556920dbc610ff1a106bf3d18e67e57d",
    "raw_has_think_close": false,
    "reasoning_chars": 37872,
    "text_chars": 44765,
    "text_sha256": "673608635cffe4379f984e99f0e44039895dc4e93c9285fe75c20544c2e233a1",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 160.9127823319286,
    "vllm_decode_tps": 63.177081725115656,
    "vllm_generation_tokens_delta": 10166.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.530331411398947,
    "vllm_prefill_tps": 30.456529707590633,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 57.61160682916495
  },
  "2K": {
    "completion_tokens": 2000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 45.10825101658702,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T23:04:48Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 2000,
    "metrics_settle_elapsed_seconds": 0.006011459976434708,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 10262,
    "text_chars": 10262,
    "text_sha256": "baa1fafcb2abb3014a4dae9a3ac255ba88f4bb44538598a342507151ca6eac69",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 30.0101278424263,
    "vllm_decode_tps": 66.64416794561383,
    "vllm_generation_tokens_delta": 2000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 15.089444040320814,
    "vllm_prefill_tps": 31.346416656311987,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 44.337786434339215
  },
  "10K": {
    "completion_tokens": 10000,
    "content_chars": 0,
    "content_has_think_close": false,
    "elapsed_seconds": 174.96906033623964,
    "endpoint": "http://127.0.0.1:19080/v1",
    "finish_reason": "length",
    "finished_at_utc": "2026-11-09T23:07:43Z",
    "force_length": true,
    "has_reasoning_field": true,
    "max_tokens": 10000,
    "metrics_settle_elapsed_seconds": 0.00648938212543726,
    "metrics_settle_timeout_seconds": 30.0,
    "metrics_settled": true,
    "metrics_url": "http://127.0.0.1:8001/metrics",
    "model": "Qwen/Qwen3.8-27B",
    "prompt_repeat": 32,
    "qwen_answer_text_chars": 0,
    "qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "qwen_gate_close_in_content": false,
    "qwen_gate_close_in_raw": false,
    "qwen_gate_invalid_reasons": [
      "client_max_tokens_cap",
      "finish_reason_length",
      "missing_think_close_or_parser_answer"
    ],
    "qwen_gate_parser_split_answer": false,
    "qwen_gate_valid": false,
    "qwen_visible_parse_complete": true,
    "qwen_visible_text_chars": 0,
    "qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "raw_has_think_close": false,
    "reasoning_chars": 18023,
    "text_chars": 18023,
    "text_sha256": "9552134caca7dee33fcf050196c55d90b694ce6c7dfe160f2d8697da6cf218bb",
    "vllm_decode_request_count_delta": 1.0,
    "vllm_decode_seconds_delta": 158.31779999658465,
    "vllm_decode_tps": 63.164091468020196,
    "vllm_generation_tokens_delta": 10000.0,
    "vllm_prefill_request_count_delta": 1.0,
    "vllm_prefill_seconds_delta": 16.642732575535774,
    "vllm_prefill_tps": 28.42081358053504,
    "vllm_prompt_tokens_delta": 473.0,
    "wall_completion_tps": 57.15296167666963
  }
}
Exact-command evidence and failures
Earlier exported-image retest · separate evidence
Retest completed; strict failed; fixed speeds diagnostic onlyVerification · Raw retestFAIL14,034 natural tokens2K 66.438*10K 62.951* · diagnostic
Retest latency / counters / KV
{
  "retest_full_ladder_completed": true,
  "retest_closure": {
    "stop_command_exit": 0,
    "container_state": {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 0,
      "Error": "",
      "StartedAt": "2026-10-26T16:21:11.087699539Z",
      "FinishedAt": "2026-10-26T16:43:25.499908697Z"
    },
    "devices_after": [
      {
        "card": 0,
        "render": 128,
        "pci": "0000:06:00.0",
        "actual_card": "card1",
        "busy_percent": 0,
        "vram_bytes": 15957090304,
        "dpm": "auto"
      },
      {
        "card": 1,
        "render": 129,
        "pci": "0000:09:00.0",
        "actual_card": "card2",
        "busy_percent": 0,
        "vram_bytes": 17210978304,
        "dpm": "auto"
      },
      {
        "card": 2,
        "render": 130,
        "pci": "0000:45:00.0",
        "actual_card": "card3",
        "busy_percent": 0,
        "vram_bytes": 17213030400,
        "dpm": "auto"
      },
      {
        "card": 3,
        "render": 131,
        "pci": "0000:48:00.0",
        "actual_card": "card4",
        "busy_percent": 0,
        "vram_bytes": 17212928000,
        "dpm": "auto"
      },
      {
        "card": 4,
        "render": 132,
        "pci": "0000:89:00.0",
        "actual_card": "card5",
        "busy_percent": 0,
        "vram_bytes": 18720759808,
        "dpm": "auto"
      },
      {
        "card": 5,
        "render": 133,
        "pci": "0000:8c:00.0",
        "actual_card": "card6",
        "busy_percent": 0,
        "vram_bytes": 18721447936,
        "dpm": "auto"
      },
      {
        "card": 6,
        "render": 134,
        "pci": "0000:c5:00.0",
        "actual_card": "card7",
        "busy_percent": 0,
        "vram_bytes": 17212538880,
        "dpm": "auto"
      },
      {
        "card": 7,
        "render": 135,
        "pci": "0000:c8:00.0",
        "actual_card": "card8",
        "busy_percent": 0,
        "vram_bytes": 17162596352,
        "dpm": "auto"
      }
    ],
    "finished": "2026-10-26T16:43:25.625278+00:00"
  },
  "retest_strict": "FAIL",
  "retest_official_2K_eligible": false,
  "retest_2K_TPS": 66.43828553575366,
  "retest_10K_TPS": 62.950553642009105,
  "retest_strict_tokens": 14034,
  "retest_tier_metrics": {
    "strict": {
      "completion_tokens": 14034,
      "elapsed_seconds": 243.24463842017576,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 228.86323131993413,
      "vllm_decode_tps": 61.32046602270283,
      "vllm_generation_tokens_delta": 14034.0,
      "vllm_prefill_seconds_delta": 14.372599754948169,
      "vllm_prompt_tokens_delta": 473.0
    },
    "2K": {
      "completion_tokens": 2000,
      "elapsed_seconds": 45.21246994892135,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 30.103124785237014,
      "vllm_decode_tps": 66.43828553575366,
      "vllm_generation_tokens_delta": 2000.0,
      "vllm_prefill_seconds_delta": 15.10064309882,
      "vllm_prompt_tokens_delta": 473.0
    },
    "10K": {
      "completion_tokens": 10000,
      "elapsed_seconds": 173.97005397314206,
      "qwen_gate_valid": false,
      "vllm_decode_seconds_delta": 158.8548379871063,
      "vllm_decode_tps": 62.950553642009105,
      "vllm_generation_tokens_delta": 10000.0,
      "vllm_prefill_seconds_delta": 15.105305118951946,
      "vllm_prompt_tokens_delta": 473.0
    }
  },
  "retest_2K_prefill_seconds": 15.10064309882,
  "retest_2K_total_seconds": 45.21246994892135,
  "retest_10K_prefill_seconds": 15.105305118951946,
  "retest_10K_total_seconds": 173.97005397314206,
  "retest_strict_prefill_seconds": 14.372599754948169,
  "retest_strict_total_seconds": 243.24463842017576,
  "retest_MTP_accounting_by_tier": {},
  "retest_loaded_admission": "Complete record: chart-details/row-47-retest-loaded-admission.json",
  "retest_audit_exit_codes": "{\"strict\": 1, \"fixed\": 1}",
  "retest_diagnostic_counter_audit": "retest-evidence/mi50-w4-tp8-mtp0/local-diagnostic-fixed-audit.json",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 29295050752,
    "1": 29241323520,
    "2": 29241430016,
    "3": 29241384960,
    "4": 29241339904,
    "5": 29241348096,
    "6": 29241323520,
    "7": 29191086080
  },
  "retest_memory_observation_count": 17,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w4-tp8-mtp0/runtime-observations.jsonl",
  "retest_logged_model_load_delta_GiB": 6.94,
  "retest_logged_model_load_seconds": 16.603594,
  "retest_logged_available_KV_GiB": 22.54,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 22.54,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 368800,
  "retest_logged_context_capacity_x": 11.15,
  "retest_logged_context_tokens": 131072,
  "retest_memory_log_report": {
    "profile": "mi50-w4-tp8-mtp0",
    "image": "sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7",
    "container": "5e1221480e5b4d81ce3a4f5f513c0a5aff435c64a9b4f7378fa2e683791bd435",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 6.94,
    "retest_logged_model_load_seconds": 16.603594,
    "retest_logged_available_KV_GiB": 22.54,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 22.54,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 368800,
    "retest_logged_context_capacity_x": 11.15,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w4-tp8-mtp0/service-final.log",
    "source_log_sha256": "3d3066110fccc8799355e5183fb3c1bfc2e9bb090c058a66fc4a945252e9f2fe",
    "sealed_index_sha256": "65edaba46ff0169318729d56558f624b0a3c954f850ea69f7848b5e0aa401997",
    "source_lines": {
      "model_load": {
        "line": 196,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:22:57 [gpu_model_runner.py:4584] Model loading took 6.94 GiB memory and 16.603594 seconds"
      },
      "automatic_KV_budget": {
        "line": 237,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:24:03 [gpu_worker.py:453] Available KV cache memory: 22.54 GiB"
      },
      "grouped_KV_tokens": {
        "line": 243,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:24:03 [kv_cache_utils.py:1314] GPU KV cache size: 368,800 tokens"
      },
      "context_capacity": {
        "line": 244,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:24:03 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.15x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w4-tp8-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 27.28314208984375
}
Complete retest loaded admission
Retest KV budget: 22.540 GiB (automatic, rounded)368,800 grouped KV tokens · 11.15× reported capacityRetest sampled VRAM: 27.283 GiB max/cardRetest memory reports and limits
Evidence and limits

TP8 GPU engineering passed; full model test pending

{
  "retest_strict_tokens": 14034,
  "retest_MTP_accounting_by_tier": {},
  "retest_diagnostic_counter_audit": "retest-evidence/mi50-w4-tp8-mtp0/local-diagnostic-fixed-audit.json",
  "strict_tokens": 14034,
  "source_revision": "research-tp8-shapes-20261026",
  "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "source": "retest-evidence/mi50-w4-tp8-mtp0/result.json",
  "comparison_limit": "One first full-model sample; TP8-specific padding/dispatch compatibility and fresh root-SSD caches. No isolated TP attribution or repeatability. Failed strict makes fixed speeds diagnostic.",
  "source_revision_type": "Preparation label, not a Git commit; exact source hashes are in the linked compatibility receipt.",
  "retest_sampled_peak_VRAM_bytes_by_rank": {
    "0": 29295050752,
    "1": 29241323520,
    "2": 29241430016,
    "3": 29241384960,
    "4": 29241339904,
    "5": 29241348096,
    "6": 29241323520,
    "7": 29191086080
  },
  "retest_memory_observation_count": 17,
  "retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
  "retest_memory_observation_source": "retest-evidence/mi50-w4-tp8-mtp0/runtime-observations.jsonl",
  "sampled_peak_VRAM_bytes_by_PCI": {
    "0000:06:00.0": 29295050752,
    "0000:09:00.0": 29241323520,
    "0000:45:00.0": 29241430016,
    "0000:48:00.0": 29241384960,
    "0000:89:00.0": 29241339904,
    "0000:8c:00.0": 29241348096,
    "0000:c5:00.0": 29241323520,
    "0000:c8:00.0": 29191086080
  },
  "retest_logged_available_KV_GiB": 22.54,
  "retest_explicit_KV_budget_bytes": null,
  "retest_KV_budget_GiB": 22.54,
  "retest_KV_budget_scope": "Automatic available-memory report; rounded",
  "retest_logged_GPU_KV_tokens": 368800,
  "retest_memory_log_report": {
    "profile": "mi50-w4-tp8-mtp0",
    "image": "sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7",
    "container": "5e1221480e5b4d81ce3a4f5f513c0a5aff435c64a9b4f7378fa2e683791bd435",
    "model_load_report_rank": 0,
    "retest_logged_model_load_delta_GiB": 6.94,
    "retest_logged_model_load_seconds": 16.603594,
    "retest_logged_available_KV_GiB": 22.54,
    "retest_explicit_KV_budget_bytes": null,
    "retest_KV_budget_GiB": 22.54,
    "retest_KV_budget_scope": "Automatic available-memory report; rounded",
    "retest_logged_GPU_KV_tokens": 368800,
    "retest_logged_context_capacity_x": 11.15,
    "retest_logged_context_tokens": 131072,
    "source_log": "retest-evidence/mi50-w4-tp8-mtp0/service-final.log",
    "source_log_sha256": "3d3066110fccc8799355e5183fb3c1bfc2e9bb090c058a66fc4a945252e9f2fe",
    "sealed_index_sha256": "65edaba46ff0169318729d56558f624b0a3c954f850ea69f7848b5e0aa401997",
    "source_lines": {
      "model_load": {
        "line": 196,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:22:57 [gpu_model_runner.py:4584] Model loading took 6.94 GiB memory and 16.603594 seconds"
      },
      "automatic_KV_budget": {
        "line": 237,
        "text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:24:03 [gpu_worker.py:453] Available KV cache memory: 22.54 GiB"
      },
      "grouped_KV_tokens": {
        "line": 243,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:24:03 [kv_cache_utils.py:1314] GPU KV cache size: 368,800 tokens"
      },
      "context_capacity": {
        "line": 244,
        "text": "(EngineCore_DP0 pid=178) INFO 10-26 16:24:03 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.15x"
      }
    },
    "limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
  },
  "retest_memory_log_source": "retest-evidence/mi50-w4-tp8-mtp0/memory-log-report.json",
  "retest_sampled_peak_VRAM_GiB": 27.28314208984375,
  "sampled_peak_VRAM_GiB": 27.28314208984375
}
V620Unmeasured TP8FP16Qwen/Qwen3.8-27BTP8MTP off NOT RUNNot reported —— — / —10K: — / —Strict: — / — — / —FP16 parameter extents / VRAM GiB —
Image identityNot built
No validated eight-V620 serving configurationNo validated eight-V620 profile availableHardware unavailable; no result invented
Evidence and limits

Eight V620s unavailable on recorded hardware

{
  "sampled_peak_VRAM_GiB": null
}
V620Unmeasured TP88-bit RTNQwen/Qwen3.8-27BTP8MTP off NOT RUNNot reported —— — / —10K: — / —Strict: — / — — / —packed target projections / VRAM GiB —
Image identityNot built
No validated eight-V620 serving configurationNo validated eight-V620 profile availableHardware unavailable; no result invented
Evidence and limits

Eight V620s unavailable on recorded hardware

{
  "sampled_peak_VRAM_GiB": null
}
V620Unmeasured TP84-bit RTNQwen/Qwen3.8-27BTP8MTP off NOT RUNNot reported —— — / —10K: — / —Strict: — / — — / —packed target projections / VRAM GiB —
Image identityNot built
No validated eight-V620 serving configurationNo validated eight-V620 profile availableHardware unavailable; no result invented
Evidence and limits

Eight V620s unavailable on recorded hardware

{
  "sampled_peak_VRAM_GiB": null
}

Portable commands use your selected devices and a fresh output directory. The checkpoint is read-only and checked against its full manifest. To repeat a command, choose a new output directory. Registry images share common layers; exact image IDs remain distinct.

Memory distinguishes packed projection tensors from sampled physical VRAM; these are not interchangeable. One recorded speed sample per entry does not establish repeatability or isolated TP/MTP causality. Generated: 2026-11-10T11:06:19.167713+00:00.