MI50 / V620 — results, images and reproduction
Qwen3.8-27B · FP16, local RTN8 and RTN4 · TP2 / TP4 / TP8 · native MTP2
47 of 48 executable configurations have all four commands independently verified; 1 capacity-blocked configuration(s) have a reviewed failure rather than a benchmark pass. 191 of 192 command executions verified. Published on LocalAIServers.com for review.
Docker Hub validation is shown separately from recorded and earlier exported-image results. A strict failure is retained even when the script itself completed successfully. The three unmeasured V620 TP8 placeholders remain unmeasured. The original services on the MI50 test host were restored and verified with real inference.
Validation coverage · Portable launch guide · Final command review · Restored-service verificationEarlier current-model validation · separate campaign
29 of 29 current-profile retests captured and checked · 28 full ladders · 1 without a full ladder · 0 running · 20 exact runtime IDs published to Docker Hub · Download guide
Earlier historical-model validation · separate campaign
6 of 6 distinct historical configurations retested · 6 complete fresh ladders · covers 19 preserved original rows · Fresh historical evidence
Serve / docker run leaves a service running. Benchmark reproduction runs the measurement sequence. Current command-validation coverage · All current commands (JSON) · Pinned model inputs
| GPU / record | Weights / model | Configuration | Recorded strict | Recorded 2K TPS | Recorded 10K TPS | Prefill / total seconds | Memory per rank | Docker Hub image | Portable serving command | Benchmark reproduction | Docker Hub command validation | Evidence |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MI50Current Qwen3.8 | FP16Qwen/Qwen3.8-27B | TP4MTP off | PASS11170 tokens | 53.555 | 50.624 | 11.320 / 48.67610K: 11.950 / 209.497Strict: 11.748 / 233.402 | 12.53 / 31.14retained FP16 parameter extents; projection buffers separate / VRAM GiBSeparate packed-buffer extents: 9.381 GiB/rank; includes alternate views, not additive physical storage. | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:2c4733c2e813c2c9fc1cad8b8aeee0a1d8e7237f7a7a85bb0e9ecd4aa5a43fccRuntime image config: sha256:6bc353a76b3d75e16560d2c5e60a8c1ff301bd85314c95ed93bb31f88c08c8b4 Exact compatibility changeReplaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately. | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp4-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-fp16-tp4-mtp0-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-7,16-23 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \ --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \ --user root \ --workdir /tmp \ -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \ -e HIP_VISIBLE_DEVICES=0,1,2,3 \ -e VLLM_GFX906_ROWPAR_MUTABLE_AR=0 \ -e TRANSFORMERS_OFFLINE=1 \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e VLLM_GFX906_QWEN_OUTPUT_FIXED3=1 \ -e VLLM_GFX906_QWEN_QKV_FIXED3=1 \ -e GFX906_MODEL_IDENTITY_DIR=/runwork \ -e VLLM_GFX906_QWEN_MLP_FIXED3_STRICT=1 \ -e VLLM_GFX906_QWEN_GDN_FIXED3=1 \ -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \ -e VLLM_GFX906_PERSISTENT_AR=0 \ -e VLLM_GFX906_PERSISTENT_AR_KIND_ROUTE=0 \ -e VLLM_GFX906_QWEN_MLP_FIXED3=1 \ -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \ -e HF_HUB_OFFLINE=1 \ -e VLLM_GFX906_QWEN_MLP_FIXED3_EXT_PATH=/opt/vllm_patch_bundle/native/fixed3/v756f96_fixed3_core.so \ -e VLLM_GFX906_ROWPAR_BOUNDARY_CUT=0 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp0/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp0/cache/.triton,dst=/root/.triton \ --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp0/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp0,dst=/runwork \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04Historical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp4-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 55.211 · eligible10K 52.887 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 6447,
"content_chars": 14192,
"content_has_think_close": false,
"elapsed_seconds": 129.0551943834871,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T13:10:56Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.008790487423539162,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 14192,
"qwen_answer_text_sha256": "48ad4353316f3ec81d778cb0bd02f955629b0e42062d8e44609031916a6f0d4a",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 14192,
"qwen_visible_text_sha256": "48ad4353316f3ec81d778cb0bd02f955629b0e42062d8e44609031916a6f0d4a",
"raw_has_think_close": false,
"reasoning_chars": 18865,
"text_chars": 33058,
"text_sha256": "3456fd7908c8bc4e5aa7ac66385d5f572f6a256afd8173ad9877fa582f015fa1",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 119.21210407651961,
"vllm_decode_tps": 54.080078947871044,
"vllm_generation_tokens_delta": 6447.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 9.833608750253916,
"vllm_prefill_tps": 48.10034769664662,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 49.9553701096506
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 45.91352475900203,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T13:11:42Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.009055406786501408,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 10184,
"text_chars": 10184,
"text_sha256": "7dbe29445b58f7f49d4a0db3484a13f4e26bf4636101d736a3329a6e0b19c4de",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 36.224582785740495,
"vllm_decode_tps": 55.21112587630087,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 9.681173008866608,
"vllm_prefill_tps": 48.857715853936064,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 43.560149443936346
},
"10K": {
"completion_tokens": 10000,
"content_chars": 33642,
"content_has_think_close": false,
"elapsed_seconds": 198.85716584604234,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T13:15:01Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.005759751424193382,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 33642,
"qwen_answer_text_sha256": "c9106d931f3a861571d0779d42737ee59d71c0701c09bf861c4f5c0e233c2698",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 33642,
"qwen_visible_text_sha256": "c9106d931f3a861571d0779d42737ee59d71c0701c09bf861c4f5c0e233c2698",
"raw_has_think_close": false,
"reasoning_chars": 18865,
"text_chars": 52508,
"text_sha256": "e6fcb8784626552d714b997d261b8220f5355bb612b4596f16ad87c1274b5f64",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 189.0812345771119,
"vllm_decode_tps": 52.88732127418894,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 9.767040423117578,
"vllm_prefill_tps": 48.428180852047845,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 50.287350508365
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS6,668 natural tokens2K 53.74910K 51.418 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-10-26T20:29:42.542233503Z",
"FinishedAt": "2026-10-26T20:57:24.923491054Z"
},
"devices_after": [
{
"card": 1,
"render": 129,
"pci": "0000:06:00.0",
"actual_card": "card0",
"busy_percent": 0,
"vram_bytes": 31794159616,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:0a:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 31561097216,
"dpm": "auto"
},
{
"card": 3,
"render": 131,
"pci": "0000:0d:00.0",
"actual_card": "card3",
"busy_percent": 0,
"vram_bytes": 31559000064,
"dpm": "auto"
},
{
"card": 4,
"render": 132,
"pci": "0000:10:00.0",
"actual_card": "card4",
"busy_percent": 0,
"vram_bytes": 31504474112,
"dpm": "auto"
}
],
"finished": "2026-10-26T20:57:25.071040+00:00"
},
"retest_closure_note": "The reproduction runner issued Docker stop with a 60-second grace period. Final exit 137, PID 0 and OOM false are retained; this stop outcome is separate from any measurement verdict.",
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 53.74919605459173,
"retest_10K_TPS": 51.41767637840134,
"retest_strict_tokens": 6668,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 6668,
"elapsed_seconds": 143.07483081799,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 127.12655968707986,
"vllm_decode_tps": 52.45166719223098,
"vllm_generation_tokens_delta": 6668.0,
"vllm_prefill_seconds_delta": 15.934438918950036,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 53.29153924190905,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 37.20985887805,
"vllm_decode_tps": 53.74919605459173,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 16.070583492983133,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 210.65881087898742,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 194.4856458780123,
"vllm_decode_tps": 51.41767637840134,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 16.16023535700515,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 16.070583492983133,
"retest_2K_total_seconds": 53.29153924190905,
"retest_10K_prefill_seconds": 16.16023535700515,
"retest_10K_total_seconds": 210.65881087898742,
"retest_strict_prefill_seconds": 15.934438918950036,
"retest_strict_total_seconds": 143.07483081799,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": null,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33421369344,
"1": 33440952320,
"2": 33438003200,
"3": 33384120320
},
"retest_memory_observation_count": 27,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-fp16-tp4-mtp0/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 22.81,
"retest_logged_model_load_seconds": 67.548319,
"retest_logged_available_KV_GiB": 6.53,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 6.53,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 106800,
"retest_logged_context_capacity_x": 3.23,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-fp16-tp4-mtp0",
"image": "sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04",
"container": "c8c0b898e9d81627eee172f20ca0783b4e5a809246b93619c4230e06528aae37",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 22.81,
"retest_logged_model_load_seconds": 67.548319,
"retest_logged_available_KV_GiB": 6.53,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 6.53,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 106800,
"retest_logged_context_capacity_x": 3.23,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-fp16-tp4-mtp0/service-final.log",
"source_log_sha256": "b6ce15ca515eb9d0dd6c076e8574f46fd917d7b27ba414136defce7cc7e4b42d",
"sealed_index_sha256": "352796834416d1920746822000adb67699cb30990ae230cd3e920799f86c1230",
"source_lines": {
"model_load": {
"line": 1161,
"text": "(Worker pid=197) (Worker_TP0 pid=197) INFO 10-26 20:32:47 [gpu_model_runner.py:4584] Model loading took 22.81 GiB memory and 67.548319 seconds"
},
"automatic_KV_budget": {
"line": 1240,
"text": "(Worker pid=197) (Worker_TP0 pid=197) INFO 10-26 20:34:54 [gpu_worker.py:453] Available KV cache memory: 6.53 GiB"
},
"grouped_KV_tokens": {
"line": 1245,
"text": "(EngineCore_DP0 pid=179) INFO 10-26 20:34:54 [kv_cache_utils.py:1314] GPU KV cache size: 106,800 tokens"
},
"context_capacity": {
"line": 1246,
"text": "(EngineCore_DP0 pid=179) INFO 10-26 20:34:54 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 3.23x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-fp16-tp4-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 31.14431381225586
} | Evidence and limitsselected FP16 reference {
"strict_tokens": 11170,
"source": "/home/joe/PhoneSync/Documents/gfx906/baseline/strict-control.json#$",
"source_revision": "57f22fd70c859e4df9036fcc075a798fb032c31d",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"latency_sources": "[{\"path\": \"/home/joe/PhoneSync/Documents/gfx906/results/gfx906-down-serving-qwen38-706be24-20261021-r0/ladder-1/dense27b_tp4_fullbar_p2pon_20261021T172635Z/c1_2000_proxy/summary.json\", \"sha256\": \"71fe5aebeb67eed51da8a43721550c6dfde41fcad6636aaf0c95eb6ec6606635\"}, {\"path\": \"/home/joe/PhoneSync/Documents/gfx906/results/gfx906-down-serving-qwen38-706be24-20261021-r0/ladder-1/dense27b_tp4_fullbar_p2pon_20261021T172635Z/c1_10000_proxy/summary.json\", \"sha256\": \"a3e5bc940b57d15910b03176bd82187c6535e8d3a14b6f77f3ed0c6558406f16\"}]",
"recorded_configuration_details": {
"cell": "FP16_TP4_MTP_off",
"status": "completed",
"TP": 4,
"native_MTP_tokens": 0,
"host": "10.0.0.50",
"strict_pass": true,
"strict_tokens": 11170,
"strict_diagnostic_TPS": 50.39692268013857,
"official_2K_TPS": 53.55497506197457,
"official_2K_tokens": 2000,
"official_2K_prefill_seconds": 11.319955765997292,
"official_2K_total_seconds": 48.67580153999734,
"diagnostic_10K_TPS": 50.62447766842328,
"diagnostic_10K_tokens": 10000,
"diagnostic_10K_prefill_seconds": 11.950416606996441,
"diagnostic_10K_total_seconds": 209.49722924898379,
"SSM_dtype": "",
"source": "57f22fd70c859e4df9036fcc075a798fb032c31d",
"image": "sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04",
"primary": "tuning/fixed3-down-early-repair/serving-result-57f22fd-20261021.json",
"strict_prefill_seconds": 11.748157959023956,
"strict_total_seconds": 233.402228802006,
"per_rank_memory_JSON": [
{
"rank": 0,
"parameter_logical_bytes": 13449011456,
"lossless_packed_logical_bytes": 10072367608,
"receipt": "results/gfx906-down-serving-qwen38-706be24-20261021-r0/loaded-model-rank0-pid250.json"
},
{
"rank": 1,
"parameter_logical_bytes": 13449011456,
"lossless_packed_logical_bytes": 10071559696,
"receipt": "results/gfx906-down-serving-qwen38-706be24-20261021-r0/loaded-model-rank1-pid251.json"
},
{
"rank": 2,
"parameter_logical_bytes": 13449011456,
"lossless_packed_logical_bytes": 10071485104,
"receipt": "results/gfx906-down-serving-qwen38-706be24-20261021-r0/loaded-model-rank2-pid252.json"
},
{
"rank": 3,
"parameter_logical_bytes": 13449011456,
"lossless_packed_logical_bytes": 10072117712,
"receipt": "results/gfx906-down-serving-qwen38-706be24-20261021-r0/loaded-model-rank3-pid253.json"
}
],
"KV_JSON": {
"logged_available_cache_memory_GiB": 6.53,
"available_memory_worker_rank": 0,
"logged_GPU_KV_cache_tokens": 106800,
"logged_max_concurrency_for_131072": 3.23,
"configured_kv_cache_dtype": "auto",
"actual_tensor_dtype": null,
"actual_SSM_tensor_dtype": null,
"actual_block_count": null,
"actual_block_size": null,
"evidence": "results/gfx906-down-serving-qwen38-706be24-20261021-r0/service-ready.log",
"lines": {
"configuration": 48,
"available_memory": 1220,
"token_capacity": 1225,
"maximum_concurrency": 1226
},
"limit": "Engine allocation reports/configuration, not direct cache-tensor observations, full-context inference, or a simple token-count/concurrency arithmetic equivalence. Rounded GiB is not exact bytes."
},
"VRAM_JSON": {
"after_supplemental_request_3_utc": "2026-10-21T17:26:34Z",
"after_supplemental_by_physical_card_bytes": {
"1": 33421770752,
"2": 33441353728,
"3": 33437638656,
"4": 33384116224
},
"after_ladder1_by_physical_card_bytes": {
"1": 33421770752,
"2": 33441353728,
"3": 33437638656,
"4": 33384116224
},
"samples_match": true,
"evidence": [
"results/gfx906-down-serving-qwen38-706be24-20261021-r0/hardware-after-ladder1.txt",
"results/gfx906-down-serving-qwen38-706be24-20261021-r0/supplemental-warmups/request-3/memory-after.txt"
],
"limit": "Two retained samples, not peak/continuous sampling or model-only allocation; hardware snapshot reports busy0 for all four assigned cards."
},
"attention_KV_dtype": "",
"context": 131072,
"speculative_accounting_JSON": null
},
"detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/fp16-matrix.csv",
"detail_source_sha256": "8643d90cf1f211a7f77be9f6212cd7706e6dbae3ae84b9bc80b5f42fa199ffe0",
"detail_source_pointer": "FP16_TP4_MTP_off",
"sampled_VRAM_after_ladder_bytes_by_rank": {
"0": 33421770752,
"1": 33441353728,
"2": 33437638656,
"3": 33384116224
},
"memory_accounting_limit": "All retained integer packed-buffer logical extents, separate from FP16 parameter extents; aliases/fallback representations are not deduplicated physical allocation. Physical VRAM is one post-ladder observation.",
"memory_source": "/home/joe/PhoneSync/Documents/gfx906/results/gfx906-down-serving-qwen38-706be24-20261021-r0/hardware-after-ladder1.txt",
"memory_source_sha256": "e8342ec2a51c7599ffafec4e5a7ab076b186cc3a93d40391d969a666d9055205",
"retest_strict_tokens": 6668,
"retest_MTP_accounting_by_tier": {},
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33421369344,
"1": 33440952320,
"2": 33438003200,
"3": 33384120320
},
"retest_memory_observation_count": 27,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-fp16-tp4-mtp0/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 6.53,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 6.53,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 106800,
"retest_memory_log_report": {
"profile": "mi50-fp16-tp4-mtp0",
"image": "sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04",
"container": "c8c0b898e9d81627eee172f20ca0783b4e5a809246b93619c4230e06528aae37",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 22.81,
"retest_logged_model_load_seconds": 67.548319,
"retest_logged_available_KV_GiB": 6.53,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 6.53,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 106800,
"retest_logged_context_capacity_x": 3.23,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-fp16-tp4-mtp0/service-final.log",
"source_log_sha256": "b6ce15ca515eb9d0dd6c076e8574f46fd917d7b27ba414136defce7cc7e4b42d",
"sealed_index_sha256": "352796834416d1920746822000adb67699cb30990ae230cd3e920799f86c1230",
"source_lines": {
"model_load": {
"line": 1161,
"text": "(Worker pid=197) (Worker_TP0 pid=197) INFO 10-26 20:32:47 [gpu_model_runner.py:4584] Model loading took 22.81 GiB memory and 67.548319 seconds"
},
"automatic_KV_budget": {
"line": 1240,
"text": "(Worker pid=197) (Worker_TP0 pid=197) INFO 10-26 20:34:54 [gpu_worker.py:453] Available KV cache memory: 6.53 GiB"
},
"grouped_KV_tokens": {
"line": 1245,
"text": "(EngineCore_DP0 pid=179) INFO 10-26 20:34:54 [kv_cache_utils.py:1314] GPU KV cache size: 106,800 tokens"
},
"context_capacity": {
"line": 1246,
"text": "(EngineCore_DP0 pid=179) INFO 10-26 20:34:54 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 3.23x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-fp16-tp4-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 31.14431381225586,
"sampled_peak_VRAM_GiB": 31.14468765258789
} |
| MI50Current Qwen3.8 | FP16Qwen/Qwen3.8-27B | TP2MTP off | PASS9596 tokens | 19.925 | 19.343 | 16.503 / 116.89110K: 16.035 / 533.021Strict: 16.509 / 511.505 | 25.05 / 31.42FP16 parameter extents / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:1c6b141d4c18f1390f718971c6e234633c15b9dfcfaf71d7038772741e70bf88Runtime image config: sha256:59e1ec913fbc1e54e9dd8c8a4181182779c02037fca6dfead3e5066f880e5dbb Exact compatibility changeReplaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately. | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp2-mtp0 GPU_RENDER_NODES=renderD128,renderD129 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-fp16-tp2-mtp0-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-7,16-23 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --user root \ --workdir /tmp \ -e HIP_VISIBLE_DEVICES=0,1 \ -e HF_HUB_OFFLINE=1 \ -e TRANSFORMERS_OFFLINE=1 \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \ -e MI50_FP16_KV_CACHE_BYTES=4495769600 \ -e MI50_FP16_TP=2 \ -e MI50_FP16_MTP=0 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ -e MAX_JOBS=1 \ --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp0/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp0,dst=/runwork \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp0/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp0/cache/.triton,dst=/root/.triton \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4Historical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp2-mtp0 GPU_RENDER_NODES=renderD128,renderD129 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 19.973 · eligible10K 19.391 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 7872,
"content_chars": 17192,
"content_has_think_close": false,
"elapsed_seconds": 416.38388748280704,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T14:00:57Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.008383386768400669,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 17192,
"qwen_answer_text_sha256": "5b3ea80d3153926a3eea07f1ea606c32e149bd79991eb2d0c41e64904fdea9c4",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 17192,
"qwen_visible_text_sha256": "5b3ea80d3153926a3eea07f1ea606c32e149bd79991eb2d0c41e64904fdea9c4",
"raw_has_think_close": false,
"reasoning_chars": 23183,
"text_chars": 40376,
"text_sha256": "067067e7523f151e2debbed8668a0976aca60ab82776edd0548deafa8a633379",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 402.27794865053147,
"vllm_decode_tps": 19.568559565363092,
"vllm_generation_tokens_delta": 7872.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.096986005082726,
"vllm_prefill_tps": 33.55327158794496,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 18.9056306851572
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 114.23982309456915,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T14:02:52Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.006374645978212357,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 10371,
"text_chars": 10371,
"text_sha256": "e0dc5c200d6b0ff17362adf0ce94343d11e60a7c1bfdacabcdb8395eb52bbf65",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 100.13581881020218,
"vllm_decode_tps": 19.972873081417628,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.095310986973345,
"vllm_prefill_tps": 33.55725889532617,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 17.507029911490452
},
"10K": {
"completion_tokens": 10000,
"content_chars": 28172,
"content_has_think_close": false,
"elapsed_seconds": 529.8064922206104,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T14:11:42Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.005857612006366253,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 28172,
"qwen_answer_text_sha256": "b0448c91cf4839e7736c792b9929579a50b908989111bbbf5d241369beeb9240",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 28172,
"qwen_visible_text_sha256": "b0448c91cf4839e7736c792b9929579a50b908989111bbbf5d241369beeb9240",
"raw_has_think_close": false,
"reasoning_chars": 23183,
"text_chars": 51356,
"text_sha256": "1ddcb1e14e4c15ffe80602abbc888941612d36c5c7329d1732bda38660286044",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 515.7007571328431,
"vllm_decode_tps": 19.391090398232684,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.096721733920276,
"vllm_prefill_tps": 33.553900610937255,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 18.874815893792444
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS8,748 natural tokens2K 19.87310K 19.302 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T21:48:44.733823837Z",
"FinishedAt": "2026-10-26T22:36:13.679245812Z"
},
"devices_after": [
{
"card": 1,
"render": 129,
"pci": "0000:06:00.0",
"actual_card": "card0",
"busy_percent": 0,
"vram_bytes": 18457292800,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:0a:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 18462343168,
"dpm": "auto"
}
],
"finished": "2026-10-26T22:36:13.825330+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 19.873362185229723,
"retest_10K_TPS": 19.30213175950557,
"retest_strict_tokens": 8748,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 8748,
"elapsed_seconds": 467.49833614297677,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 450.85449509404134,
"vllm_decode_tps": 19.403155774625915,
"vllm_generation_tokens_delta": 8748.0,
"vllm_prefill_seconds_delta": 16.6319247179199,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 116.15711760893464,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 100.63722390600014,
"vllm_decode_tps": 19.873362185229723,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 15.506479750969447,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 532.3052741159918,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 518.077491367003,
"vllm_decode_tps": 19.30213175950557,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 14.213958831969649,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 15.506479750969447,
"retest_2K_total_seconds": 116.15711760893464,
"retest_10K_prefill_seconds": 14.213958831969649,
"retest_10K_total_seconds": 532.3052741159918,
"retest_strict_prefill_seconds": 16.6319247179199,
"retest_strict_total_seconds": 467.49833614297677,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": "Complete record: chart-details/row-01-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33733324800,
"1": 33733308416
},
"retest_memory_observation_count": 45,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-fp16-tp2-mtp0/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 25.37,
"retest_logged_model_load_seconds": 16.160363,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 4495769600,
"retest_KV_budget_GiB": 4.18701171875,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 33712,
"retest_logged_context_capacity_x": 1.02,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-fp16-tp2-mtp0",
"image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
"container": "99381d2f6797e72404e8e186fec8c4bd3744a515945517d05b5b8548a689c542",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 25.37,
"retest_logged_model_load_seconds": 16.160363,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 4495769600,
"retest_KV_budget_GiB": 4.18701171875,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 33712,
"retest_logged_context_capacity_x": 1.02,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-fp16-tp2-mtp0/service-final.log",
"source_log_sha256": "9baac6ec7141f36335e506c7777e974cae19185b140c8112860fc3fbb3c05edb",
"sealed_index_sha256": "e4f8ebde61b26ca3d5a8e83d60d7ec38e19c442b556cfe06683fa5c49a7ecff3",
"source_lines": {
"model_load": {
"line": 122,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 21:50:52 [gpu_model_runner.py:4584] Model loading took 25.37 GiB memory and 16.160363 seconds"
},
"explicit_KV_budget": {
"line": 7,
"text": "(APIServer pid=1) INFO 10-26 21:49:16 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM']}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.fp16_matrix_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 4495769600, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'optimization_level': '3'}"
},
"explicit_budget_worker_reports": [
{
"line": 134,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 21:51:57 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.19 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
},
{
"line": 135,
"text": "(Worker pid=197) (Worker_TP1 pid=197) INFO 10-26 21:51:57 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.19 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
}
],
"grouped_KV_tokens": {
"line": 136,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 21:51:57 [kv_cache_utils.py:1314] GPU KV cache size: 33,712 tokens"
},
"context_capacity": {
"line": 137,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 21:51:57 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.02x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-fp16-tp2-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 31.416606903076172
}Complete retest loaded admission | Evidence and limitsCompleted, independently reviewed FP16 TP2 MTP-off matrix cell; one sample, no promotion or TP4 goal credit. {
"strict_tokens": 9596,
"source": "/home/joe/PhoneSync/Documents/gfx906/tuning/qwen38-fp16-matrix/fp16-tp2-off-result-20261023.json#$",
"source_revision": "90733170b8b007fca3401b68c04e5bc67e14a3e0",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"attention_KV_dtype": "FP16",
"memory_accounting_limit": "Logical parameter/cache extents are not unique physical allocations. Hybrid runtime KV token/concurrency reports are retained separately from block geometry; no full-context request was executed.",
"sampled_VRAM_before_ladder_bytes_by_rank": "{\"0\": 33733308416, \"1\": 33733312512}",
"sampled_VRAM_after_ladder_bytes_by_rank": "{\"0\": 33733308416, \"1\": 33733312512}",
"KV_capacity": "{\"SSM\": {\"bytes\": 275251200, \"dtype\": \"torch.float32\", \"shape\": [175, 24, 128, 128]}, \"attention\": {\"bytes\": 280985600, \"dtype\": \"torch.float16\", \"shape\": [175, 2, 784, 2, 256]}, \"block_size\": 784, \"blocks\": 175, \"conv\": {\"bytes\": 5376000, \"dtype\": \"torch.float16\", \"shape\": [175, 3, 5120]}, \"logged_KV_tokens\": 33712, \"logged_max_concurrency_at_131072\": 1.02}",
"explicit_KV_cache_budget_bytes": 4495769600,
"post_KV_allocator_allocated_bytes_by_rank": "{\"0\": 31604896256, \"1\": 31604896256}",
"post_KV_allocator_reserved_bytes_by_rank": "{\"0\": 31878807552, \"1\": 31878807552}",
"source_evidence_commit": "26c009d",
"comparison_limit": "Explicit original dense FP16/collective fallback at TP2; TP4 packed/posted guards not widened and native DSOs unchanged. | Explicit 4495769600-byte KV budget, context131072 unchanged; actual capacity admission is configuration evidence, not full-context execution. | Actual target/attention KV/conv FP16 and SSM FP32. Non-MTP packed convolution and recurrent flags remain1. | Original greedy concurrency1, non-eager O3/maxseq2/maxbatched4 requests and scorer; eight warmups and three tracked startup requests excluded. | One official sample, no repeatability/causal gain or TP4 qualification. | Logical parameter/buffer bytes are not unique physical storage. Cache view bytes may alias; no summed physical cache claim. | Loaded route metadata/generated source/maps are not a per-kernel execution trace. | Outer metrics do not establish internal scorer scrape alignment/full settlement trajectory. | KV tokens/concurrency are logged hybrid-cache reports; do not equate175x784 with usable tokens or infer actual131072 execution. | Final kernel journal filter covers assigned four-card PCI scope, broader than TP2 before/after scope. | Read-only retained cache preserves regular files; socket endpoints inventoried, not byte-hashed. | No native MTP head in this cell; all prior failed/historical attempts preserved.",
"recorded_configuration_details": {
"cell": "FP16_TP2_MTPoff",
"status": "completed",
"TP": 2,
"native_MTP_tokens": 0,
"host": "10.0.0.50",
"strict_pass": true,
"strict_tokens": 9596,
"strict_diagnostic_TPS": 19.386496043730823,
"official_2K_TPS": 19.924717816026906,
"official_2K_tokens": 2000,
"official_2K_prefill_seconds": 16.502791117993183,
"official_2K_total_seconds": 116.89143691700883,
"diagnostic_10K_TPS": 19.34330520874968,
"diagnostic_10K_tokens": 10000,
"diagnostic_10K_prefill_seconds": 16.035188685986213,
"diagnostic_10K_total_seconds": 533.0213894949993,
"SSM_dtype": "torch.float32",
"source": "90733170b8b007fca3401b68c04e5bc67e14a3e0",
"image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
"primary": "tuning/qwen38-fp16-matrix/fp16-tp2-off-result-20261023.json",
"strict_prefill_seconds": 16.508799550996628,
"strict_total_seconds": 511.50529021699913,
"per_rank_memory_JSON": [
{
"rank": 0,
"pid": 196,
"parameter_count": 611,
"parameter_logical_bytes": 26896673280,
"parameter_dtypes": [
"torch.float16"
],
"packed_counts": {
"_gfx906_fixed3_gate_primary": 0,
"_gfx906_fixed3_primary": 0,
"_gfx906_gdn_fixed3_primary": 0,
"_gfx906_output_fixed3_primary": 0,
"_gfx906_qkv_fixed3_primary": 0
},
"packed_logical_bytes": 0,
"after_load_allocated_bytes": 27105545728,
"after_load_reserved_bytes": 27231518720,
"after_cache_allocated_bytes": 31604896256,
"after_cache_reserved_bytes": 31878807552,
"installed_manifest_sha256": "44ea477cf0ae8d0a6da890f35540f66d477c93913eb7d728b45ae58aacb57df0",
"attention_layers": 16,
"GDN_layers": 48
},
{
"rank": 1,
"pid": 197,
"parameter_count": 611,
"parameter_logical_bytes": 26896673280,
"parameter_dtypes": [
"torch.float16"
],
"packed_counts": {
"_gfx906_fixed3_gate_primary": 0,
"_gfx906_fixed3_primary": 0,
"_gfx906_gdn_fixed3_primary": 0,
"_gfx906_output_fixed3_primary": 0,
"_gfx906_qkv_fixed3_primary": 0
},
"packed_logical_bytes": 0,
"after_load_allocated_bytes": 27105545728,
"after_load_reserved_bytes": 27231518720,
"after_cache_allocated_bytes": 31604896256,
"after_cache_reserved_bytes": 31878807552,
"installed_manifest_sha256": "44ea477cf0ae8d0a6da890f35540f66d477c93913eb7d728b45ae58aacb57df0",
"attention_layers": 16,
"GDN_layers": 48
}
],
"KV_JSON": {
"attention": {
"shape": [
175,
2,
784,
2,
256
],
"dtype": "torch.float16",
"bytes": 280985600
},
"conv": {
"shape": [
175,
3,
5120
],
"dtype": "torch.float16",
"bytes": 5376000
},
"SSM": {
"shape": [
175,
24,
128,
128
],
"dtype": "torch.float32",
"bytes": 275251200
},
"blocks": 175,
"block_size": 784,
"logged_KV_tokens": 33712,
"logged_max_concurrency_at_131072": 1.02
},
"VRAM_JSON": {
"before": "renderD129 33733308416\nrenderD130 33733312512",
"after": "renderD129 33733308416\nrenderD130 33733312512"
},
"attention_KV_dtype": "torch.float16",
"context": 131072,
"speculative_accounting_JSON": null
},
"detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/fp16-matrix.csv",
"detail_source_sha256": "8643d90cf1f211a7f77be9f6212cd7706e6dbae3ae84b9bc80b5f42fa199ffe0",
"detail_source_pointer": "FP16_TP2_MTPoff",
"retest_strict_tokens": 8748,
"retest_MTP_accounting_by_tier": {},
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33733324800,
"1": 33733308416
},
"retest_memory_observation_count": 45,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-fp16-tp2-mtp0/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 4495769600,
"retest_KV_budget_GiB": 4.18701171875,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 33712,
"retest_memory_log_report": {
"profile": "mi50-fp16-tp2-mtp0",
"image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
"container": "99381d2f6797e72404e8e186fec8c4bd3744a515945517d05b5b8548a689c542",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 25.37,
"retest_logged_model_load_seconds": 16.160363,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 4495769600,
"retest_KV_budget_GiB": 4.18701171875,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 33712,
"retest_logged_context_capacity_x": 1.02,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-fp16-tp2-mtp0/service-final.log",
"source_log_sha256": "9baac6ec7141f36335e506c7777e974cae19185b140c8112860fc3fbb3c05edb",
"sealed_index_sha256": "e4f8ebde61b26ca3d5a8e83d60d7ec38e19c442b556cfe06683fa5c49a7ecff3",
"source_lines": {
"model_load": {
"line": 122,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 21:50:52 [gpu_model_runner.py:4584] Model loading took 25.37 GiB memory and 16.160363 seconds"
},
"explicit_KV_budget": {
"line": 7,
"text": "(APIServer pid=1) INFO 10-26 21:49:16 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM']}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.fp16_matrix_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 4495769600, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'optimization_level': '3'}"
},
"explicit_budget_worker_reports": [
{
"line": 134,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 21:51:57 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.19 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
},
{
"line": 135,
"text": "(Worker pid=197) (Worker_TP1 pid=197) INFO 10-26 21:51:57 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.19 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
}
],
"grouped_KV_tokens": {
"line": 136,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 21:51:57 [kv_cache_utils.py:1314] GPU KV cache size: 33,712 tokens"
},
"context_capacity": {
"line": 137,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 21:51:57 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.02x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-fp16-tp2-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 31.416606903076172,
"sampled_peak_VRAM_GiB": 31.416595458984375
} |
| MI50Current Qwen3.8 | FP16Qwen/Qwen3.8-27B | TP4MTP22K accepted 1122 / 1756 (63.90%) | PASS7114 tokens | 27.819 | 23.428 | 18.001 / 89.90710K: 18.015 / 444.865Strict: 26.326 / 311.798 | 12.53 / 29.47retained FP16 parameter extents; projection buffers separate / VRAM GiBSeparate packed-buffer extents: 9.381 GiB/rank; includes alternate views, not additive physical storage. | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:1c6b141d4c18f1390f718971c6e234633c15b9dfcfaf71d7038772741e70bf88Runtime image config: sha256:59e1ec913fbc1e54e9dd8c8a4181182779c02037fca6dfead3e5066f880e5dbb Exact compatibility changeReplaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately. | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp4-mtp2 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-fp16-tp4-mtp2-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-7,16-23 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \ --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \ --user root \ --workdir /tmp \ -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \ -e MI50_FP16_TP=4 \ -e MI50_FP16_MTP=1 \ -e HIP_VISIBLE_DEVICES=0,1,2,3 \ -e HF_HUB_OFFLINE=1 \ -e TRANSFORMERS_OFFLINE=1 \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ -e MAX_JOBS=1 \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp2/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp2/cache/.triton,dst=/root/.triton \ --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp2/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp4-mtp2,dst=/runwork \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4Historical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp4-mtp2 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 30.146 · eligible10K 23.866 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 8240,
"content_chars": 16196,
"content_has_think_close": false,
"elapsed_seconds": 338.21908202953637,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T14:51:41Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.008258074522018433,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 16196,
"qwen_answer_text_sha256": "850242d24439ba048f921216fb301df5a9ed94509f467bf49c1b4e69c45882e5",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 16196,
"qwen_visible_text_sha256": "850242d24439ba048f921216fb301df5a9ed94509f467bf49c1b4e69c45882e5",
"raw_has_think_close": false,
"reasoning_chars": 26490,
"text_chars": 42687,
"text_sha256": "f0646379f7a2e598a9d348a84e9e4e9fc44af518966167d370e66b73b2c3259f",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 325.45470919366926,
"vllm_decode_tps": 25.318423016262457,
"vllm_generation_tokens_delta": 8240.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 12.75549241527915,
"vllm_prefill_tps": 37.08206509012679,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 24.362906878448708
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 79.18325763661414,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T14:53:00Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.00859201792627573,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 8930,
"text_chars": 8930,
"text_sha256": "c3ada5acb4981cef88965e9d67305b457dc31cc108eea3e48c5b766e2dbfa012",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 66.3432620735839,
"vllm_decode_tps": 30.146241494452322,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 12.832328528165817,
"vllm_prefill_tps": 36.86002886863496,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 25.257864600347094
},
"10K": {
"completion_tokens": 10000,
"content_chars": 25611,
"content_has_think_close": false,
"elapsed_seconds": 431.6993595920503,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T15:00:12Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008361870422959328,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 25611,
"qwen_answer_text_sha256": "f08a2e9e6af3876e798eddcc7b7904cdb6d70e093f95d7510c8e71b6d3ccf434",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 25611,
"qwen_visible_text_sha256": "f08a2e9e6af3876e798eddcc7b7904cdb6d70e093f95d7510c8e71b6d3ccf434",
"raw_has_think_close": false,
"reasoning_chars": 26490,
"text_chars": 52102,
"text_sha256": "d7dd0e23e90221d9579b7b90513868c19cd7a8354623944daf195f0a241a3936",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 418.99922180734575,
"vllm_decode_tps": 23.86639277482468,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 12.691327778622508,
"vllm_prefill_tps": 37.26954407376739,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 23.16426878522557
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestMTP counter replayPASS11,252 natural tokens2K 27.10310K 22.724 · diagnostic2K MTP accepted 1109 / 1782 (62.23%) Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T20:59:39.988715707Z",
"FinishedAt": "2026-10-26T21:46:24.608013465Z"
},
"devices_after": [
{
"card": 1,
"render": 129,
"pci": "0000:06:00.0",
"actual_card": "card0",
"busy_percent": 0,
"vram_bytes": 25278005248,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:0a:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 25393348608,
"dpm": "auto"
},
{
"card": 3,
"render": 131,
"pci": "0000:0d:00.0",
"actual_card": "card3",
"busy_percent": 0,
"vram_bytes": 25848430592,
"dpm": "auto"
},
{
"card": 4,
"render": 132,
"pci": "0000:10:00.0",
"actual_card": "card4",
"busy_percent": 0,
"vram_bytes": 25334628352,
"dpm": "auto"
}
],
"finished": "2026-10-26T21:46:24.749264+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 27.10269994925174,
"retest_10K_TPS": 22.72398827389817,
"retest_strict_tokens": 11252,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 11252,
"elapsed_seconds": 532.8966229540529,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 507.4504561870126,
"vllm_decode_tps": 22.17359323025864,
"vllm_generation_tokens_delta": 11252.0,
"vllm_prefill_seconds_delta": 25.432538520894013,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 98.17414218490012,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 73.7933860369958,
"vllm_decode_tps": 27.10269994925174,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 24.37041282595601,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 464.32734813098796,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 440.06359620799776,
"vllm_decode_tps": 22.72398827389817,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 24.250391291920096,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 24.37041282595601,
"retest_2K_total_seconds": 98.17414218490012,
"retest_10K_prefill_seconds": 24.250391291920096,
"retest_10K_total_seconds": 464.32734813098796,
"retest_strict_prefill_seconds": 25.432538520894013,
"retest_strict_total_seconds": 532.8966229540529,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6926.0,
"vllm:spec_decode_num_drafts_total": 4326.0,
"vllm:request_generation_tokens_sum": 11252.0,
"vllm:spec_decode_num_accepted_tokens_total": 6926.0,
"vllm:generation_tokens_total": 11252.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 8652.0
},
"accepted_per_position": {
"0": 3729.0,
"1": 3197.0
},
"accepted_fraction": 0.8005085529357374,
"committed_emitted_tokens": 11252.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "29c2ed4ec314d2eb6a82fa83a792c5b39f1efa0c03863de816efe5de6b3540dc",
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "43f683bf9787a3fabc5c5d56c85b9e7fa4b9d476097a175297a7bd013ce52a0d"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 1782.0,
"vllm:spec_decode_num_drafts_total": 891.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1109.0,
"vllm:spec_decode_num_accepted_tokens_total": 1109.0,
"vllm:generation_tokens_total": 2000.0
},
"accepted_per_position": {
"0": 667.0,
"1": 442.0
},
"accepted_fraction": 0.622334455667789,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "44c060e7c7f6ad1f8d10cfb4974c2f609f97c8488318396b5e2d1d1776646fe8",
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "e0fe496bed1252fd10a56f00ec24c4b278fd954cce3899754f7921b693c9239c"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:generation_tokens_total": 10000.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_accepted_tokens_total": 6092.0,
"vllm:spec_decode_num_drafts_total": 3909.0,
"vllm:spec_decode_num_draft_tokens_total": 7818.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6092.0
},
"accepted_per_position": {
"0": 3312.0,
"1": 2780.0
},
"accepted_fraction": 0.7792274238935789,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "c9b911e4b1a22b9368c93204f4938392aafed794a173dcf60d30f084df540283",
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "052278aac272623437a14aa1a168e01093f65f46c73b392ddecd652f3ce4e37c"
}
}
},
"retest_MTP_2K_accepted": 1109.0,
"retest_MTP_2K_proposed": 1782.0,
"retest_MTP_2K_draft_batches": 891.0,
"retest_MTP_2K_acceptance_percent": 62.2334455667789,
"retest_loaded_admission": "Complete record: chart-details/row-02-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 31621283840,
"1": 31639273472,
"2": 31637164032,
"3": 31580545024
},
"retest_memory_observation_count": 44,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-fp16-tp4-mtp2/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 24.18,
"retest_logged_model_load_seconds": 82.437392,
"retest_logged_available_KV_GiB": 4.79,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 4.79,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 73600,
"retest_logged_context_capacity_x": 2.13,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-fp16-tp4-mtp2",
"image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
"container": "a6900dc2002295b8c6786cb3cbe72fc0905467b837eb0c385bd9c52d17f73453",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 24.18,
"retest_logged_model_load_seconds": 82.437392,
"retest_logged_available_KV_GiB": 4.79,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 4.79,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 73600,
"retest_logged_context_capacity_x": 2.13,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-fp16-tp4-mtp2/service-final.log",
"source_log_sha256": "01da5b75e4ef49cb1d7d6bc6ad0f0bedd32b3e985961684407826dc906561b9a",
"sealed_index_sha256": "287b09e42de876cf580b22e48e3779b0b496040a93d8af8611dfe440da40deec",
"source_lines": {
"model_load": {
"line": 1173,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 21:03:21 [gpu_model_runner.py:4584] Model loading took 24.18 GiB memory and 82.437392 seconds"
},
"automatic_KV_budget": {
"line": 1317,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 21:09:19 [gpu_worker.py:453] Available KV cache memory: 4.79 GiB"
},
"grouped_KV_tokens": {
"line": 1321,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 21:09:19 [kv_cache_utils.py:1314] GPU KV cache size: 73,600 tokens"
},
"context_capacity": {
"line": 1322,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 21:09:19 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 2.13x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-fp16-tp4-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 29.46636962890625
}Complete retest loaded admission | Evidence and limitsCompleted, independently reviewed FP16 TP4 native MTP2 matrix cell; separate from non-speculative FP16 qualification. No promotion. {
"strict_tokens": 7114,
"source": "/home/joe/PhoneSync/Documents/gfx906/tuning/qwen38-fp16-matrix/fp16-tp4-mtp-result-20261023.json#$",
"source_revision": "90733170b8b007fca3401b68c04e5bc67e14a3e0",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"attention_KV_dtype": "FP16",
"memory_accounting_limit": "Parameter lists counted once by name and route buffer fields once by module/field. Parameter and packed-buffer logical extents are reported separately, not summed as physical storage. Shared draft embedding/head excluded from unshared head bytes; views/allocator aliases are not de-duplicated physical storage.",
"sampled_VRAM_before_ladder_bytes_by_rank": "{\"0\": 31620239360, \"1\": 31640121344, \"2\": 31637413888, \"3\": 31580463104}",
"sampled_VRAM_after_ladder_bytes_by_rank": "{\"0\": 31620239360, \"1\": 31640121344, \"2\": 31637413888, \"3\": 31580463104}",
"KV_capacity": "{\"actual_blocks_per_physical_pool\": 368, \"full_attention_block_tokens\": 800, \"limit\": \"Different hybrid group accounting; do not substitute naive blocks\\u00d7tokens for logged capacity or claim actual full-context inference.\", \"logged_GPU_KV_cache_tokens\": 73600, \"logged_available_KV_GiB\": 4.79, \"logged_maximum_concurrency_for_131072\": 2.13, \"physical_pools\": 17}",
"post_KV_allocator_allocated_bytes_by_rank": "{\"0\": 29316868096, \"1\": 29315055104, \"2\": 29315478016, \"3\": 29319622656}",
"post_KV_allocator_reserved_bytes_by_rank": "{\"0\": 29806821376, \"1\": 29808918528, \"2\": 29806821376, \"3\": 29800529920}",
"MTP_accounting_by_tier": {
"tiers": {
"strict": {
"generation_tokens_total": 7114.0,
"spec_decode_num_accepted_tokens_per_pos_total_0": 2349.0,
"spec_decode_num_accepted_tokens_per_pos_total_1": 1986.0,
"spec_decode_num_accepted_tokens_total": 4335.0,
"spec_decode_num_draft_tokens_total": 5560.0,
"spec_decode_num_drafts_total": 2780.0,
"adjusted_unaccepted_proposed_minus_accepted": 1225.0,
"iteration_histogram_count_delta": 2781.0,
"iteration_histogram_sum_delta": 7587.0,
"full_report": "c1_128_strict.json"
},
"fixed_2000": {
"generation_tokens_total": 2000.0,
"spec_decode_num_accepted_tokens_per_pos_total_0": 646.0,
"spec_decode_num_accepted_tokens_per_pos_total_1": 476.0,
"spec_decode_num_accepted_tokens_total": 1122.0,
"spec_decode_num_draft_tokens_total": 1756.0,
"spec_decode_num_drafts_total": 878.0,
"adjusted_unaccepted_proposed_minus_accepted": 634.0,
"iteration_histogram_count_delta": 879.0,
"iteration_histogram_sum_delta": 2473.0,
"full_report": "c1_2000.json"
},
"diagnostic_10000": {
"generation_tokens_total": 10000.0,
"spec_decode_num_accepted_tokens_per_pos_total_0": 3312.0,
"spec_decode_num_accepted_tokens_per_pos_total_1": 2886.0,
"spec_decode_num_accepted_tokens_total": 6198.0,
"spec_decode_num_draft_tokens_total": 7602.0,
"spec_decode_num_drafts_total": 3801.0,
"adjusted_unaccepted_proposed_minus_accepted": 1404.0,
"iteration_histogram_count_delta": 3802.0,
"iteration_histogram_sum_delta": 10473.0,
"full_report": "c1_10000.json"
}
},
"limits": [
"Returned completion tokens determine fixed-tier length; drafts and accepted proposals are separate counters.",
"Proposed minus accepted is adjusted unaccepted, not scheduler rollback rejection.",
"Iteration histogram counts and sums include prefill; they do not give pure decode-step emissions or exact 1/2/3 token chronology.",
"Warmups are excluded from the three reported tiers; all eleven reports remain in the linked primary."
],
"all_samples_and_histograms_primary": "tuning/qwen38-fp16-matrix/fp16-tp4-mtp-result-20261023.json"
},
"MTP_accounting_limit": "TPS uses committed returned tokens only. Proposed minus accepted is adjusted unaccepted, not a scheduler rollback count; iteration counts include prefill.",
"comparison_limit": "Native two-token MTP and same-checkpoint unquantized draft head; no quantization. | Target TP4 fixed3/interleaved/posted routes retained; exact draft prefixes excluded from target packing. | MTP cache mode none; FP16 activation/attention KV and explicitly float32 SSM. | Packed QKVZ convolution disabled to preserve history5 compatibility; generic speculative rollback unchanged; packed non-spec recurrent decode remains enabled. | MTP compile cache disabled; original context/O3/non-eager/maxseq2/maxbatched4 retained. | One-time loaded metadata worker; no native arithmetic or binary change. | Native MTP cell is separate from the original non-speculative FP16 qualification. | Strict PASS is sole model correctness result; forced-length tier parser invalidity does not replace strict adjudication. | Accepted/proposed/draft counters differ from returned outputs; P-A is not scheduler rollback rejection. | Outer snapshots cannot recover per-step committed-token distribution or internal scrape trajectory. | No promotion, target credit, causal speed claim or repetition is inferred. | All earlier failed quantized/MTP and source/runtime attempts remain historical; none overwritten. | Actual368 blocks,800-token full-attention blocks,17 physical pools; logical tensor capacity is not a successful131072-token request or unique allocation proof. | Parameter lists counted once by name and route buffer fields once by module/field. Parameter and packed-buffer logical extents are reported separately, not summed as physical storage. Shared draft embedding/head excluded from unshared head bytes; views/allocator aliases are not de-duplicated physical storage. | Owner observed bash -lc no-op explicit exit0 returns1, versus bash -c returns0. Preserve outer SSH1 separately from remote startup/ladder/closure0; not a runtime failure or all-SSH0 claim.",
"recorded_configuration_details": {
"cell": "FP16_TP4_MTP2",
"status": "completed",
"TP": 4,
"native_MTP_tokens": 2,
"host": "10.0.0.50",
"strict_pass": true,
"strict_tokens": 7114,
"strict_diagnostic_TPS": 24.92135282587446,
"official_2K_TPS": 27.818932990951446,
"official_2K_tokens": 2000,
"official_2K_prefill_seconds": 18.001278546988033,
"official_2K_total_seconds": 89.90659736702219,
"diagnostic_10K_TPS": 23.42828193269942,
"diagnostic_10K_tokens": 10000,
"diagnostic_10K_prefill_seconds": 18.015202388050966,
"diagnostic_10K_total_seconds": 444.86478443199303,
"SSM_dtype": "float32",
"source": "90733170b8b007fca3401b68c04e5bc67e14a3e0",
"image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
"primary": "tuning/qwen38-fp16-matrix/fp16-tp4-mtp-result-20261023.json",
"strict_prefill_seconds": 26.3263557300088,
"strict_total_seconds": 311.7976599709946,
"per_rank_memory_JSON": [
{
"rank": 0,
"target_parameter_logical_bytes": 13449011456,
"packed_buffer_logical_bytes": 10072367608,
"after_load_allocated_bytes": 24180564992,
"after_load_reserved_bytes": 25945964544,
"after_cache_allocated_bytes": 29316868096,
"after_cache_reserved_bytes": 29806821376
},
{
"rank": 1,
"target_parameter_logical_bytes": 13449011456,
"packed_buffer_logical_bytes": 10071559696,
"after_load_allocated_bytes": 24178752000,
"after_load_reserved_bytes": 25948061696,
"after_cache_allocated_bytes": 29315055104,
"after_cache_reserved_bytes": 29808918528
},
{
"rank": 2,
"target_parameter_logical_bytes": 13449011456,
"packed_buffer_logical_bytes": 10071485104,
"after_load_allocated_bytes": 24179174912,
"after_load_reserved_bytes": 25945964544,
"after_cache_allocated_bytes": 29315478016,
"after_cache_reserved_bytes": 29806821376
},
{
"rank": 3,
"target_parameter_logical_bytes": 13449011456,
"packed_buffer_logical_bytes": 10072117712,
"after_load_allocated_bytes": 24183319552,
"after_load_reserved_bytes": 25939673088,
"after_cache_allocated_bytes": 29319622656,
"after_cache_reserved_bytes": 29800529920
}
],
"KV_JSON": {
"actual_blocks_per_physical_pool": 368,
"full_attention_block_tokens": 800,
"physical_pools": 17,
"logged_GPU_KV_cache_tokens": 73600,
"logged_maximum_concurrency_for_131072": 2.13,
"logged_available_KV_GiB": 4.79,
"limit": "Different hybrid group accounting; do not substitute naive blocks\u00d7tokens for logged capacity or claim actual full-context inference."
},
"VRAM_JSON": {
"before": "renderD129 31620239360\nrenderD130 31640121344\nrenderD131 31637413888\nrenderD132 31580463104\n",
"after": "renderD129 31620239360\nrenderD130 31640121344\nrenderD131 31637413888\nrenderD132 31580463104\n"
},
"attention_KV_dtype": "FP16",
"context": 131072,
"speculative_accounting_JSON": {
"tiers": {
"strict": {
"generation_tokens_total": 7114.0,
"spec_decode_num_accepted_tokens_per_pos_total_0": 2349.0,
"spec_decode_num_accepted_tokens_per_pos_total_1": 1986.0,
"spec_decode_num_accepted_tokens_total": 4335.0,
"spec_decode_num_draft_tokens_total": 5560.0,
"spec_decode_num_drafts_total": 2780.0,
"adjusted_unaccepted_proposed_minus_accepted": 1225.0,
"iteration_histogram_count_delta": 2781.0,
"iteration_histogram_sum_delta": 7587.0,
"full_report": "c1_128_strict.json"
},
"fixed_2000": {
"generation_tokens_total": 2000.0,
"spec_decode_num_accepted_tokens_per_pos_total_0": 646.0,
"spec_decode_num_accepted_tokens_per_pos_total_1": 476.0,
"spec_decode_num_accepted_tokens_total": 1122.0,
"spec_decode_num_draft_tokens_total": 1756.0,
"spec_decode_num_drafts_total": 878.0,
"adjusted_unaccepted_proposed_minus_accepted": 634.0,
"iteration_histogram_count_delta": 879.0,
"iteration_histogram_sum_delta": 2473.0,
"full_report": "c1_2000.json"
},
"diagnostic_10000": {
"generation_tokens_total": 10000.0,
"spec_decode_num_accepted_tokens_per_pos_total_0": 3312.0,
"spec_decode_num_accepted_tokens_per_pos_total_1": 2886.0,
"spec_decode_num_accepted_tokens_total": 6198.0,
"spec_decode_num_draft_tokens_total": 7602.0,
"spec_decode_num_drafts_total": 3801.0,
"adjusted_unaccepted_proposed_minus_accepted": 1404.0,
"iteration_histogram_count_delta": 3802.0,
"iteration_histogram_sum_delta": 10473.0,
"full_report": "c1_10000.json"
}
},
"limits": [
"Returned completion tokens determine fixed-tier length; drafts and accepted proposals are separate counters.",
"Proposed minus accepted is adjusted unaccepted, not scheduler rollback rejection.",
"Iteration histogram counts and sums include prefill; they do not give pure decode-step emissions or exact 1/2/3 token chronology.",
"Warmups are excluded from the three reported tiers; all eleven reports remain in the linked primary."
],
"all_samples_and_histograms_primary": "tuning/qwen38-fp16-matrix/fp16-tp4-mtp-result-20261023.json"
}
},
"MTP_2K_draft_batches": 878.0,
"MTP_2K_proposed": 1756.0,
"MTP_2K_accepted": 1122.0,
"MTP_2K_acceptance_percent": 63.89521640091116,
"MTP_2K_committed": 2000.0,
"detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/fp16-matrix.csv",
"detail_source_sha256": "8643d90cf1f211a7f77be9f6212cd7706e6dbae3ae84b9bc80b5f42fa199ffe0",
"detail_source_pointer": "FP16_TP4_MTP2",
"retest_strict_tokens": 11252,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6926.0,
"vllm:spec_decode_num_drafts_total": 4326.0,
"vllm:request_generation_tokens_sum": 11252.0,
"vllm:spec_decode_num_accepted_tokens_total": 6926.0,
"vllm:generation_tokens_total": 11252.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 8652.0
},
"accepted_per_position": {
"0": 3729.0,
"1": 3197.0
},
"accepted_fraction": 0.8005085529357374,
"committed_emitted_tokens": 11252.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "29c2ed4ec314d2eb6a82fa83a792c5b39f1efa0c03863de816efe5de6b3540dc",
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "43f683bf9787a3fabc5c5d56c85b9e7fa4b9d476097a175297a7bd013ce52a0d"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 1782.0,
"vllm:spec_decode_num_drafts_total": 891.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1109.0,
"vllm:spec_decode_num_accepted_tokens_total": 1109.0,
"vllm:generation_tokens_total": 2000.0
},
"accepted_per_position": {
"0": 667.0,
"1": 442.0
},
"accepted_fraction": 0.622334455667789,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "44c060e7c7f6ad1f8d10cfb4974c2f609f97c8488318396b5e2d1d1776646fe8",
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "e0fe496bed1252fd10a56f00ec24c4b278fd954cce3899754f7921b693c9239c"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:generation_tokens_total": 10000.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_accepted_tokens_total": 6092.0,
"vllm:spec_decode_num_drafts_total": 3909.0,
"vllm:spec_decode_num_draft_tokens_total": 7818.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6092.0
},
"accepted_per_position": {
"0": 3312.0,
"1": 2780.0
},
"accepted_fraction": 0.7792274238935789,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "c9b911e4b1a22b9368c93204f4938392aafed794a173dcf60d30f084df540283",
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "052278aac272623437a14aa1a168e01093f65f46c73b392ddecd652f3ce4e37c"
}
}
},
"retest_MTP_2K_accepted": 1109.0,
"retest_MTP_2K_proposed": 1782.0,
"retest_MTP_2K_draft_batches": 891.0,
"retest_MTP_2K_acceptance_percent": 62.2334455667789,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 31621283840,
"1": 31639273472,
"2": 31637164032,
"3": 31580545024
},
"retest_memory_observation_count": 44,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-fp16-tp4-mtp2/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 4.79,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 4.79,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 73600,
"retest_memory_log_report": {
"profile": "mi50-fp16-tp4-mtp2",
"image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
"container": "a6900dc2002295b8c6786cb3cbe72fc0905467b837eb0c385bd9c52d17f73453",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 24.18,
"retest_logged_model_load_seconds": 82.437392,
"retest_logged_available_KV_GiB": 4.79,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 4.79,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 73600,
"retest_logged_context_capacity_x": 2.13,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-fp16-tp4-mtp2/service-final.log",
"source_log_sha256": "01da5b75e4ef49cb1d7d6bc6ad0f0bedd32b3e985961684407826dc906561b9a",
"sealed_index_sha256": "287b09e42de876cf580b22e48e3779b0b496040a93d8af8611dfe440da40deec",
"source_lines": {
"model_load": {
"line": 1173,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 21:03:21 [gpu_model_runner.py:4584] Model loading took 24.18 GiB memory and 82.437392 seconds"
},
"automatic_KV_budget": {
"line": 1317,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 21:09:19 [gpu_worker.py:453] Available KV cache memory: 4.79 GiB"
},
"grouped_KV_tokens": {
"line": 1321,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 21:09:19 [kv_cache_utils.py:1314] GPU KV cache size: 73,600 tokens"
},
"context_capacity": {
"line": 1322,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 21:09:19 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 2.13x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-fp16-tp4-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 29.46636962890625,
"sampled_peak_VRAM_GiB": 29.467159271240234
} |
| MI50Current Qwen3.8 | FP16Qwen/Qwen3.8-27B | TP2MTP22K accepted 1119 / 1764 (63.44%) | PASS8452 tokens | 18.156 | 17.195 | 24.747 / 134.91610K: 24.715 / 606.308Strict: 18.314 / 495.761 | 25.05 / 31.50FP16 parameter extents / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:1c6b141d4c18f1390f718971c6e234633c15b9dfcfaf71d7038772741e70bf88Runtime image config: sha256:59e1ec913fbc1e54e9dd8c8a4181182779c02037fca6dfead3e5066f880e5dbb Exact compatibility changeReplaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately. | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp2-mtp2 GPU_RENDER_NODES=renderD128,renderD129 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-fp16-tp2-mtp2-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-7,16-23 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --user root \ --workdir /tmp \ -e MI50_FP16_TP=2 \ -e MI50_FP16_MTP=1 \ -e HIP_VISIBLE_DEVICES=0,1 \ -e HF_HUB_OFFLINE=1 \ -e TRANSFORMERS_OFFLINE=1 \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \ -e MI50_FP16_KV_CACHE_BYTES=4929945600 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ -e MAX_JOBS=1 \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp2/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp2/cache/.triton,dst=/root/.triton \ --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp2/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp2-mtp2,dst=/runwork \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4Historical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp2-mtp2 GPU_RENDER_NODES=renderD128,renderD129 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-fp16-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 18.303 · eligible10K 16.868 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 7733,
"content_chars": 15664,
"content_has_think_close": false,
"elapsed_seconds": 454.4751425785944,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T15:47:21Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.008119790814816952,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 15664,
"qwen_answer_text_sha256": "8164b1a96681e827dd0917ca18f9a1b538da8a1e71026cdf89d328166d5e6319",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 15664,
"qwen_visible_text_sha256": "8164b1a96681e827dd0917ca18f9a1b538da8a1e71026cdf89d328166d5e6319",
"raw_has_think_close": false,
"reasoning_chars": 24035,
"text_chars": 39700,
"text_sha256": "a3b515cfead4289fa8a99c3d092d9bfe6915e7ad59e6d7142f40264fb59b53c7",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 439.371760125272,
"vllm_decode_tps": 17.600129780291745,
"vllm_generation_tokens_delta": 7733.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.093696534633636,
"vllm_prefill_tps": 31.337585124668795,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 17.01523202374638
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 124.39862818829715,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T15:49:26Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008269759826362133,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9971,
"text_chars": 9971,
"text_sha256": "9eed22714e313734d169dd51b23d37c39b31db6c5bd9f411f04a191ebb554027",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 109.26931860670447,
"vllm_decode_tps": 18.303399577320008,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.09930254612118,
"vllm_prefill_tps": 31.325950225529304,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 16.077347709756744
},
"10K": {
"completion_tokens": 10000,
"content_chars": 27909,
"content_has_think_close": false,
"elapsed_seconds": 607.9516332549974,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T15:59:34Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.007648466154932976,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 27909,
"qwen_answer_text_sha256": "a81f6effa1157fb8ecc4b9944da9aeb6aad02c9a5e3f1768aa7e23f383af6794",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 27909,
"qwen_visible_text_sha256": "a81f6effa1157fb8ecc4b9944da9aeb6aad02c9a5e3f1768aa7e23f383af6794",
"raw_has_think_close": false,
"reasoning_chars": 24035,
"text_chars": 51945,
"text_sha256": "d4454027af91bd30f70c2174af129107b4b93f98393fb941f3c2f53db3a26159",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 592.8418835354969,
"vllm_decode_tps": 16.86790403600295,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.100885062478483,
"vllm_prefill_tps": 31.322667382938633,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 16.44867692263544
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestMTP counter replayPASS9,193 natural tokens2K 17.98710K 16.598 · diagnostic2K MTP accepted 1111 / 1780 (62.42%) Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T22:38:26.508242705Z",
"FinishedAt": "2026-10-26T23:32:28.555393816Z"
},
"devices_after": [
{
"card": 1,
"render": 129,
"pci": "0000:06:00.0",
"actual_card": "card0",
"busy_percent": 0,
"vram_bytes": 28219174912,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:0a:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 27819536384,
"dpm": "auto"
}
],
"finished": "2026-10-26T23:32:28.696241+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 17.986992732445394,
"retest_10K_TPS": 16.597632050314704,
"retest_strict_tokens": 9193,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 9193,
"elapsed_seconds": 562.7220106379827,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 543.6877332479926,
"vllm_decode_tps": 16.908602931099775,
"vllm_generation_tokens_delta": 9193.0,
"vllm_prefill_seconds_delta": 19.022434039972723,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 130.33776032400783,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 111.19146094901953,
"vllm_decode_tps": 17.986992732445394,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 19.13075027789455,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 625.5376734989695,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 602.4955830859253,
"vllm_decode_tps": 16.597632050314704,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 23.02978788409382,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 19.13075027789455,
"retest_2K_total_seconds": 130.33776032400783,
"retest_10K_prefill_seconds": 23.02978788409382,
"retest_10K_total_seconds": 625.5376734989695,
"retest_strict_prefill_seconds": 19.022434039972723,
"retest_strict_total_seconds": 562.7220106379827,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 5591.0,
"vllm:spec_decode_num_draft_tokens_total": 7204.0,
"vllm:generation_tokens_total": 9193.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 5591.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:request_generation_tokens_sum": 9193.0,
"vllm:spec_decode_num_drafts_total": 3602.0
},
"accepted_per_position": {
"0": 3054.0,
"1": 2537.0
},
"accepted_fraction": 0.7760966129927818,
"committed_emitted_tokens": 9193.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "27fa225cc8c4a6d7f8827719f2bd337459e89a5177a1bcf2cc19615df8298915",
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "88bc7a5f4e148bcaad2371bb058820ccffa01b02023c18216c6e58318503e5d5"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1111.0,
"vllm:spec_decode_num_accepted_tokens_total": 1111.0,
"vllm:spec_decode_num_drafts_total": 890.0,
"vllm:spec_decode_num_draft_tokens_total": 1780.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 2000.0
},
"accepted_per_position": {
"0": 663.0,
"1": 448.0
},
"accepted_fraction": 0.6241573033707866,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "88bc7a5f4e148bcaad2371bb058820ccffa01b02023c18216c6e58318503e5d5",
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "0a2b24d4501925f978e6ed227d810b69d2b565bbb4b2ac956c9270723cebf9ce"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:generation_tokens_total": 10000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6100.0,
"vllm:spec_decode_num_drafts_total": 3900.0,
"vllm:spec_decode_num_accepted_tokens_total": 6100.0,
"vllm:spec_decode_num_draft_tokens_total": 7800.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:request_generation_tokens_count": 1.0
},
"accepted_per_position": {
"0": 3325.0,
"1": 2775.0
},
"accepted_fraction": 0.782051282051282,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "9709d01df025fec6ae6aeaee1653bcd74d5bd21bfe14c67de92a1daa08723393",
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "106866b7537531838ae81399415e724a034b478609f3d59bf2ed18afd6dbc442"
}
}
},
"retest_MTP_2K_accepted": 1111.0,
"retest_MTP_2K_proposed": 1780.0,
"retest_MTP_2K_draft_batches": 890.0,
"retest_MTP_2K_acceptance_percent": 62.41573033707866,
"retest_loaded_admission": "Complete record: chart-details/row-03-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 34144878592,
"1": 34144944128
},
"retest_memory_observation_count": 51,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-fp16-tp2-mtp2/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 28.13,
"retest_logged_model_load_seconds": 17.482613,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 4929945600,
"retest_KV_budget_GiB": 4.59136962890625,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 35200,
"retest_logged_context_capacity_x": 1.02,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-fp16-tp2-mtp2",
"image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
"container": "01770f22e65cc4feb5e7ba102d94ef9688bb226605fc07e4cce66124eb50ca8a",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 28.13,
"retest_logged_model_load_seconds": 17.482613,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 4929945600,
"retest_KV_budget_GiB": 4.59136962890625,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 35200,
"retest_logged_context_capacity_x": 1.02,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-fp16-tp2-mtp2/service-final.log",
"source_log_sha256": "c9b1261d2fad38310f2950a7f12730e059bee01e33ed9344f95799be1edcf54f",
"sealed_index_sha256": "c66a8e5ffb498b9534473eb9e911f1479f90a4415fd1b1f2266c5f3f84e3e2f8",
"source_lines": {
"model_load": {
"line": 148,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 22:40:54 [gpu_model_runner.py:4584] Model loading took 28.13 GiB memory and 17.482613 seconds"
},
"explicit_KV_budget": {
"line": 7,
"text": "(APIServer pid=1) INFO 10-26 22:38:57 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM']}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.fp16_matrix_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 4929945600, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 2}, 'optimization_level': '3'}"
},
"explicit_budget_worker_reports": [
{
"line": 160,
"text": "(Worker pid=217) (Worker_TP1 pid=217) INFO 10-26 22:41:46 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.59 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
},
{
"line": 161,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 22:41:46 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.59 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
}
],
"grouped_KV_tokens": {
"line": 163,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 22:41:46 [kv_cache_utils.py:1314] GPU KV cache size: 35,200 tokens"
},
"context_capacity": {
"line": 164,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 22:41:46 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.02x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-fp16-tp2-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 31.799957275390625
}Complete retest loaded admission | Evidence and limitsCompleted, independently reviewed FP16 TP2 native MTP2 cell; one strict-valid sample, no promotion or non-speculative FP16 qualification. {
"strict_tokens": 8452,
"source": "/home/joe/PhoneSync/Documents/gfx906/tuning/qwen38-fp16-matrix/fp16-tp2-mtp-result-20261023.json#$",
"source_revision": "90733170b8b007fca3401b68c04e5bc67e14a3e0",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"attention_KV_dtype": "torch.float16",
"memory_accounting_limit": "Logical target weights and unshared native-head parameters are reported separately. Shared embedding/head overlaps are excluded from unshared head bytes. Cache views may alias; logical bytes, allocator counters and sampled physical VRAM must not be summed as disjoint storage. Allocation is not a full-context inference result.",
"sampled_VRAM_before_ladder_bytes_by_rank": "{\"0\": 33825607680, \"1\": 33825673216}",
"sampled_VRAM_after_ladder_bytes_by_rank": "{\"0\": 33825607680, \"1\": 33825673216}",
"KV_capacity": "{\"SSM\": {\"bytes\": 278396928, \"dtype\": \"torch.float32\", \"shape\": [177, 24, 128, 128]}, \"attention\": {\"bytes\": 289996800, \"dtype\": \"torch.float16\", \"shape\": [177, 2, 800, 2, 256]}, \"block_size\": 800, \"blocks\": 177, \"conv\": {\"bytes\": 9062400, \"dtype\": \"torch.float16\", \"shape\": [177, 5, 5120]}, \"logged_KV_tokens\": 35200, \"logged_max_concurrency_at_131072\": 1.02}",
"explicit_KV_cache_budget_bytes": 4929945600,
"post_KV_allocator_allocated_bytes_by_rank": "{\"0\": 32466124800, \"1\": 32466124800}",
"post_KV_allocator_reserved_bytes_by_rank": "{\"0\": 32828817408, \"1\": 32828817408}",
"MTP_accounting_by_tier": {
"tiers": {
"strict": {
"generation_tokens_total": 8452.0,
"spec_decode_num_accepted_tokens_per_pos_total_0": 2804.0,
"spec_decode_num_accepted_tokens_per_pos_total_1": 2429.0,
"spec_decode_num_accepted_tokens_total": 5233.0,
"spec_decode_num_draft_tokens_total": 6440.0,
"spec_decode_num_drafts_total": 3220.0,
"adjusted_unaccepted_proposed_minus_accepted": 1207.0,
"iteration_histogram_count_delta": 3221.0,
"iteration_histogram_sum_delta": 8925.0,
"full_report": "/home/joe/PhoneSync/Documents/gfx906/results/mi50-fp16-tp2-mtp2-9073317-r0-mirror/offline_analysis/c1_128_strict-mtp-counters.json"
},
"fixed_2000": {
"generation_tokens_total": 2000.0,
"spec_decode_num_accepted_tokens_per_pos_total_0": 653.0,
"spec_decode_num_accepted_tokens_per_pos_total_1": 466.0,
"spec_decode_num_accepted_tokens_total": 1119.0,
"spec_decode_num_draft_tokens_total": 1764.0,
"spec_decode_num_drafts_total": 882.0,
"adjusted_unaccepted_proposed_minus_accepted": 645.0,
"iteration_histogram_count_delta": 883.0,
"iteration_histogram_sum_delta": 2473.0,
"full_report": "/home/joe/PhoneSync/Documents/gfx906/results/mi50-fp16-tp2-mtp2-9073317-r0-mirror/offline_analysis/c1_2000-mtp-counters.json"
},
"diagnostic_10000": {
"generation_tokens_total": 10000.0,
"spec_decode_num_accepted_tokens_per_pos_total_0": 3318.0,
"spec_decode_num_accepted_tokens_per_pos_total_1": 2924.0,
"spec_decode_num_accepted_tokens_total": 6242.0,
"spec_decode_num_draft_tokens_total": 7514.0,
"spec_decode_num_drafts_total": 3757.0,
"adjusted_unaccepted_proposed_minus_accepted": 1272.0,
"iteration_histogram_count_delta": 3758.0,
"iteration_histogram_sum_delta": 10473.0,
"full_report": "/home/joe/PhoneSync/Documents/gfx906/results/mi50-fp16-tp2-mtp2-9073317-r0-mirror/offline_analysis/c1_10000-mtp-counters.json"
}
},
"limits": [
"Returned completion tokens determine fixed-tier length; drafts and accepted proposals are separate counters.",
"Proposed minus accepted is adjusted unaccepted, not scheduler rollback rejection.",
"Iteration histogram counts and sums include prefill; they do not give pure decode-step emissions or exact 1/2/3 token chronology.",
"Warmups are excluded from the three reported tiers; all eleven reports remain in the linked primary."
],
"all_samples_and_histograms_primary": "tuning/qwen38-fp16-matrix/fp16-tp2-mtp-result-20261023.json"
},
"MTP_accounting_limit": "Only committed emitted tokens enter TPS. Proposed minus accepted is adjusted unaccepted, not scheduler rollback. Iteration histograms include prefill and do not measure pure decode-step emissions.",
"source_evidence_commit": "69c6721",
"comparison_limit": "Explicit original dense FP16/collective fallback at TP2; TP4 packed/posted guards not widened and native DSOs unchanged. | Explicit 4929945600-byte KV budget, context131072 unchanged; actual capacity admission is configuration evidence, not full-context execution. | Actual target/attention KV/conv FP16 and SSM FP32. MTP-only generic convolution fallback (packed convolution0), recurrent1; cache mode none and compile-cache disable, original graph policy. | Original greedy concurrency1, non-eager O3/maxseq2/maxbatched4 requests and scorer; eight warmups and three tracked startup requests excluded. | One official sample, no repeatability/causal gain or TP4 qualification. | Logical parameter/buffer bytes are not unique physical storage. Cache view bytes may alias; no summed physical cache claim. | Loaded route metadata/generated source/maps are not a per-kernel execution trace. | Outer metrics do not establish internal scorer scrape alignment/full settlement trajectory. | KV tokens/concurrency are logged hybrid-cache reports; do not equate177x800 with usable tokens or infer actual131072 execution. | Final kernel journal filter covers assigned four-card PCI scope, broader than TP2 before/after scope. | Read-only retained cache preserves regular files; socket endpoints inventoried, not byte-hashed. | Same-checkpoint native FP16 MTP head, shared embedding/head; all prior failed/historical attempts preserved. | Host exclusivity is unproven; runtime/cache/compiler differences prevent attributing differences solely to MTP. | Proposed minus accepted is adjusted unaccepted, not scheduler rollback rejection; aggregate iteration histograms are not per-step decode emission distributions. Full labeled before/after and histogram reports remain pinned outside hashed source. | One official sample, no repeatability, causal, promotion or TP4 qualification claim. | Original dense TP2 projection/generic collective fallback and generic convolution MTP route are composition differences. | Histogram mean includes prefill/engine output iterations and text clipping boundaries; not pure decode-step emission. | P-A is adjusted unaccepted proposals, not scheduler rollback rejections. | No per-token kernel trace or actual131072-token request. | Outer scrapes do not expose scorer internal scrapes/full settlement trajectory. | Before/after kernels cover selected PCI06/0a; final covers four assigned MI50 cards; selected intersection unchanged. | Cache same-filesystem rename/read-only; socket endpoint state not byte-hashed.",
"recorded_configuration_details": {
"cell": "FP16_TP2_MTP2",
"status": "completed",
"TP": 2,
"native_MTP_tokens": 2,
"host": "10.0.0.50",
"strict_pass": true,
"strict_tokens": 8452,
"strict_diagnostic_TPS": 17.703028402850663,
"official_2K_TPS": 18.15566891881604,
"official_2K_tokens": 2000,
"official_2K_prefill_seconds": 24.74742527602939,
"official_2K_total_seconds": 134.91621195804328,
"diagnostic_10K_TPS": 17.194536494951368,
"diagnostic_10K_tokens": 10000,
"diagnostic_10K_prefill_seconds": 24.714972343004774,
"diagnostic_10K_total_seconds": 606.308061289019,
"SSM_dtype": "torch.float32",
"source": "90733170b8b007fca3401b68c04e5bc67e14a3e0",
"image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
"primary": "tuning/qwen38-fp16-matrix/fp16-tp2-mtp-result-20261023.json",
"strict_prefill_seconds": 18.314323675003834,
"strict_total_seconds": 495.76133046799805,
"per_rank_memory_JSON": [
{
"rank": 0,
"pid": 216,
"parameter_count": 611,
"parameter_logical_bytes": 26896673280,
"parameter_dtypes": [
"torch.float16"
],
"packed_counts": {
"_gfx906_fixed3_gate_primary": 0,
"_gfx906_fixed3_primary": 0,
"_gfx906_gdn_fixed3_primary": 0,
"_gfx906_output_fixed3_primary": 0,
"_gfx906_qkv_fixed3_primary": 0
},
"packed_logical_bytes": 0,
"after_load_allocated_bytes": 27531406336,
"after_load_reserved_bytes": 30203183104,
"after_cache_allocated_bytes": 32466124800,
"after_cache_reserved_bytes": 32828817408,
"installed_manifest_sha256": "44ea477cf0ae8d0a6da890f35540f66d477c93913eb7d728b45ae58aacb57df0",
"attention_layers": 16,
"GDN_layers": 48
},
{
"rank": 1,
"pid": 217,
"parameter_count": 611,
"parameter_logical_bytes": 26896673280,
"parameter_dtypes": [
"torch.float16"
],
"packed_counts": {
"_gfx906_fixed3_gate_primary": 0,
"_gfx906_fixed3_primary": 0,
"_gfx906_gdn_fixed3_primary": 0,
"_gfx906_output_fixed3_primary": 0,
"_gfx906_qkv_fixed3_primary": 0
},
"packed_logical_bytes": 0,
"after_load_allocated_bytes": 27531406336,
"after_load_reserved_bytes": 30203183104,
"after_cache_allocated_bytes": 32466124800,
"after_cache_reserved_bytes": 32828817408,
"installed_manifest_sha256": "44ea477cf0ae8d0a6da890f35540f66d477c93913eb7d728b45ae58aacb57df0",
"attention_layers": 16,
"GDN_layers": 48
}
],
"KV_JSON": {
"attention": {
"shape": [
177,
2,
800,
2,
256
],
"dtype": "torch.float16",
"bytes": 289996800
},
"conv": {
"shape": [
177,
5,
5120
],
"dtype": "torch.float16",
"bytes": 9062400
},
"SSM": {
"shape": [
177,
24,
128,
128
],
"dtype": "torch.float32",
"bytes": 278396928
},
"blocks": 177,
"block_size": 800,
"logged_KV_tokens": 35200,
"logged_max_concurrency_at_131072": 1.02
},
"VRAM_JSON": {
"before": "renderD129 33825607680\nrenderD130 33825673216",
"after": "renderD129 33825607680\nrenderD130 33825673216"
},
"attention_KV_dtype": "torch.float16",
"context": 131072,
"speculative_accounting_JSON": {
"tiers": {
"strict": {
"generation_tokens_total": 8452.0,
"spec_decode_num_accepted_tokens_per_pos_total_0": 2804.0,
"spec_decode_num_accepted_tokens_per_pos_total_1": 2429.0,
"spec_decode_num_accepted_tokens_total": 5233.0,
"spec_decode_num_draft_tokens_total": 6440.0,
"spec_decode_num_drafts_total": 3220.0,
"adjusted_unaccepted_proposed_minus_accepted": 1207.0,
"iteration_histogram_count_delta": 3221.0,
"iteration_histogram_sum_delta": 8925.0,
"full_report": "/home/joe/PhoneSync/Documents/gfx906/results/mi50-fp16-tp2-mtp2-9073317-r0-mirror/offline_analysis/c1_128_strict-mtp-counters.json"
},
"fixed_2000": {
"generation_tokens_total": 2000.0,
"spec_decode_num_accepted_tokens_per_pos_total_0": 653.0,
"spec_decode_num_accepted_tokens_per_pos_total_1": 466.0,
"spec_decode_num_accepted_tokens_total": 1119.0,
"spec_decode_num_draft_tokens_total": 1764.0,
"spec_decode_num_drafts_total": 882.0,
"adjusted_unaccepted_proposed_minus_accepted": 645.0,
"iteration_histogram_count_delta": 883.0,
"iteration_histogram_sum_delta": 2473.0,
"full_report": "/home/joe/PhoneSync/Documents/gfx906/results/mi50-fp16-tp2-mtp2-9073317-r0-mirror/offline_analysis/c1_2000-mtp-counters.json"
},
"diagnostic_10000": {
"generation_tokens_total": 10000.0,
"spec_decode_num_accepted_tokens_per_pos_total_0": 3318.0,
"spec_decode_num_accepted_tokens_per_pos_total_1": 2924.0,
"spec_decode_num_accepted_tokens_total": 6242.0,
"spec_decode_num_draft_tokens_total": 7514.0,
"spec_decode_num_drafts_total": 3757.0,
"adjusted_unaccepted_proposed_minus_accepted": 1272.0,
"iteration_histogram_count_delta": 3758.0,
"iteration_histogram_sum_delta": 10473.0,
"full_report": "/home/joe/PhoneSync/Documents/gfx906/results/mi50-fp16-tp2-mtp2-9073317-r0-mirror/offline_analysis/c1_10000-mtp-counters.json"
}
},
"limits": [
"Returned completion tokens determine fixed-tier length; drafts and accepted proposals are separate counters.",
"Proposed minus accepted is adjusted unaccepted, not scheduler rollback rejection.",
"Iteration histogram counts and sums include prefill; they do not give pure decode-step emissions or exact 1/2/3 token chronology.",
"Warmups are excluded from the three reported tiers; all eleven reports remain in the linked primary."
],
"all_samples_and_histograms_primary": "tuning/qwen38-fp16-matrix/fp16-tp2-mtp-result-20261023.json"
}
},
"MTP_2K_draft_batches": 882.0,
"MTP_2K_proposed": 1764.0,
"MTP_2K_accepted": 1119.0,
"MTP_2K_acceptance_percent": 63.435374149659864,
"MTP_2K_committed": 2000.0,
"detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/fp16-matrix.csv",
"detail_source_sha256": "8643d90cf1f211a7f77be9f6212cd7706e6dbae3ae84b9bc80b5f42fa199ffe0",
"detail_source_pointer": "FP16_TP2_MTP2",
"retest_strict_tokens": 9193,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 5591.0,
"vllm:spec_decode_num_draft_tokens_total": 7204.0,
"vllm:generation_tokens_total": 9193.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 5591.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:request_generation_tokens_sum": 9193.0,
"vllm:spec_decode_num_drafts_total": 3602.0
},
"accepted_per_position": {
"0": 3054.0,
"1": 2537.0
},
"accepted_fraction": 0.7760966129927818,
"committed_emitted_tokens": 9193.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "27fa225cc8c4a6d7f8827719f2bd337459e89a5177a1bcf2cc19615df8298915",
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "88bc7a5f4e148bcaad2371bb058820ccffa01b02023c18216c6e58318503e5d5"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1111.0,
"vllm:spec_decode_num_accepted_tokens_total": 1111.0,
"vllm:spec_decode_num_drafts_total": 890.0,
"vllm:spec_decode_num_draft_tokens_total": 1780.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 2000.0
},
"accepted_per_position": {
"0": 663.0,
"1": 448.0
},
"accepted_fraction": 0.6241573033707866,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "88bc7a5f4e148bcaad2371bb058820ccffa01b02023c18216c6e58318503e5d5",
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "0a2b24d4501925f978e6ed227d810b69d2b565bbb4b2ac956c9270723cebf9ce"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:generation_tokens_total": 10000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6100.0,
"vllm:spec_decode_num_drafts_total": 3900.0,
"vllm:spec_decode_num_accepted_tokens_total": 6100.0,
"vllm:spec_decode_num_draft_tokens_total": 7800.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:request_generation_tokens_count": 1.0
},
"accepted_per_position": {
"0": 3325.0,
"1": 2775.0
},
"accepted_fraction": 0.782051282051282,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "9709d01df025fec6ae6aeaee1653bcd74d5bd21bfe14c67de92a1daa08723393",
"/home/ai/qwen38-release-retests-20261026/mi50-fp16-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "106866b7537531838ae81399415e724a034b478609f3d59bf2ed18afd6dbc442"
}
}
},
"retest_MTP_2K_accepted": 1111.0,
"retest_MTP_2K_proposed": 1780.0,
"retest_MTP_2K_draft_batches": 890.0,
"retest_MTP_2K_acceptance_percent": 62.41573033707866,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 34144878592,
"1": 34144944128
},
"retest_memory_observation_count": 51,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-fp16-tp2-mtp2/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 4929945600,
"retest_KV_budget_GiB": 4.59136962890625,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 35200,
"retest_memory_log_report": {
"profile": "mi50-fp16-tp2-mtp2",
"image": "sha256:b6fa047585772ad215c9dd1cf6ff0a57dc37039743aab0637ace0753c20ff2e4",
"container": "01770f22e65cc4feb5e7ba102d94ef9688bb226605fc07e4cce66124eb50ca8a",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 28.13,
"retest_logged_model_load_seconds": 17.482613,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 4929945600,
"retest_KV_budget_GiB": 4.59136962890625,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 35200,
"retest_logged_context_capacity_x": 1.02,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-fp16-tp2-mtp2/service-final.log",
"source_log_sha256": "c9b1261d2fad38310f2950a7f12730e059bee01e33ed9344f95799be1edcf54f",
"sealed_index_sha256": "c66a8e5ffb498b9534473eb9e911f1479f90a4415fd1b1f2266c5f3f84e3e2f8",
"source_lines": {
"model_load": {
"line": 148,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 22:40:54 [gpu_model_runner.py:4584] Model loading took 28.13 GiB memory and 17.482613 seconds"
},
"explicit_KV_budget": {
"line": 7,
"text": "(APIServer pid=1) INFO 10-26 22:38:57 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM']}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.fp16_matrix_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 4929945600, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 2}, 'optimization_level': '3'}"
},
"explicit_budget_worker_reports": [
{
"line": 160,
"text": "(Worker pid=217) (Worker_TP1 pid=217) INFO 10-26 22:41:46 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.59 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
},
{
"line": 161,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 22:41:46 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 4.59 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
}
],
"grouped_KV_tokens": {
"line": 163,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 22:41:46 [kv_cache_utils.py:1314] GPU KV cache size: 35,200 tokens"
},
"context_capacity": {
"line": 164,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 22:41:46 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.02x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-fp16-tp2-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 31.799957275390625,
"sampled_peak_VRAM_GiB": 31.502613067626953
} |
| MI50Current Qwen3.8 | 8-bit RTNQwen/Qwen3.8-27B | TP4MTP off | PASS14098 tokens | 40.861 | 39.515 | 15.177 / 64.13710K: 16.029 / 269.109Strict: 13.244 / 376.013 | 5.80 / 25.35packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:a506e684a1f3142d071857d89f8fee8f17f0a30e92970c76d163c03380004961Runtime image config: sha256:fe0d81ba97b6b8a432db928d19d03b511f15de9e40e5d25be81797d9c3e1694b Exact compatibility changeReplaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately. | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp4-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-w8-tp4-mtp0-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-7,16-23 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \ --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \ --user root \ --workdir /tmp \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \ -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \ -e MI50_RTN_TP=4 \ -e HIP_VISIBLE_DEVICES=0,1,2,3 \ -e HF_HUB_OFFLINE=1 \ -e MI50_RTN_BITS=8 \ -e TRANSFORMERS_OFFLINE=1 \ -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp0/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp0/cache/.triton,dst=/root/.triton \ --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp0/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp0,dst=/runwork \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24aHistorical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp4-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 41.516 · eligible10K 40.124 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 9254,
"content_chars": 17704,
"content_has_think_close": false,
"elapsed_seconds": 239.1448707189411,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T16:30:45Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.008517238311469555,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 17704,
"qwen_answer_text_sha256": "2136309f6e466a777e2cc34c8bf82f4d72be9c4cefff367e37709495d31b0a1f",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 17704,
"qwen_visible_text_sha256": "2136309f6e466a777e2cc34c8bf82f4d72be9c4cefff367e37709495d31b0a1f",
"raw_has_think_close": false,
"reasoning_chars": 27545,
"text_chars": 45250,
"text_sha256": "fb06c0d2793348020838c839569222efaa9132372c78d6fcdcc657af1da0e551",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 229.69770350586623,
"vllm_decode_tps": 40.28773409031346,
"vllm_generation_tokens_delta": 9254.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 9.437211033888161,
"vllm_prefill_tps": 50.12073994122843,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 38.696209423934974
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 57.735280628316104,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T16:31:43Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.00803779810667038,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 6532,
"text_chars": 6532,
"text_sha256": "ef46298b33bf1677f80fde93f2648cae9345517992f93fc8d2b1d90d68cf392e",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 48.173785145394504,
"vllm_decode_tps": 41.516355710138825,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 9.553459473885596,
"vllm_prefill_tps": 49.5108605728581,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 34.64086392643436
},
"10K": {
"completion_tokens": 10000,
"content_chars": 21707,
"content_has_think_close": false,
"elapsed_seconds": 258.7847386011854,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T16:36:02Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008156619034707546,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 21707,
"qwen_answer_text_sha256": "99fb6e9f1c94acc87a5110d7453e700d80ed6ffed621a85fee60a797d181be6e",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 21707,
"qwen_visible_text_sha256": "99fb6e9f1c94acc87a5110d7453e700d80ed6ffed621a85fee60a797d181be6e",
"raw_has_think_close": false,
"reasoning_chars": 27545,
"text_chars": 49253,
"text_sha256": "3e84f6a3408e1b88b0a7d7ab7aaf7e6f673e8abb2a6c53a4ae0dd2fd99bcf36d",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 249.2266788072884,
"vllm_decode_tps": 40.1241153148471,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 9.549467314966023,
"vllm_prefill_tps": 49.53155860942207,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 38.64215507472817
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS16,128 natural tokens2K 40.83310K 39.500 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T15:30:09.216444552Z",
"FinishedAt": "2026-10-26T16:04:27.834638844Z"
},
"devices_after": [
{
"card": 1,
"render": 129,
"pci": "0000:06:00.0",
"actual_card": "card0",
"busy_percent": 0,
"vram_bytes": 7933067264,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:0a:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 4747493376,
"dpm": "auto"
},
{
"card": 3,
"render": 131,
"pci": "0000:0d:00.0",
"actual_card": "card3",
"busy_percent": 0,
"vram_bytes": 9868738560,
"dpm": "auto"
},
{
"card": 4,
"render": 132,
"pci": "0000:10:00.0",
"actual_card": "card4",
"busy_percent": 0,
"vram_bytes": 9818406912,
"dpm": "auto"
}
],
"finished": "2026-10-26T16:04:27.971854+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 40.83310567872094,
"retest_10K_TPS": 39.49955384999737,
"retest_strict_tokens": 16128,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 16128,
"elapsed_seconds": 439.7262021850329,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 418.7522614010377,
"vllm_decode_tps": 38.514418873918075,
"vllm_generation_tokens_delta": 16128.0,
"vllm_prefill_seconds_delta": 20.95829319197219,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 70.49048449797556,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 48.97986491001211,
"vllm_decode_tps": 40.83310567872094,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 21.49934186495375,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 275.08431653201114,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 253.16741647198796,
"vllm_decode_tps": 39.49955384999737,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 21.903418170055375,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 21.49934186495375,
"retest_2K_total_seconds": 70.49048449797556,
"retest_10K_prefill_seconds": 21.903418170055375,
"retest_10K_total_seconds": 275.08431653201114,
"retest_strict_prefill_seconds": 20.95829319197219,
"retest_strict_total_seconds": 439.7262021850329,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": "Complete record: chart-details/row-04-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_logged_model_load_delta_GiB": 13.25,
"retest_logged_model_load_seconds": 64.44897,
"retest_logged_available_KV_GiB": 15.92,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 15.92,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 260288,
"retest_logged_context_capacity_x": 7.77,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-w8-tp4-mtp0",
"image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
"container": "5a3463ca7a0d4134be09562c677c092b3a8f8952eb1d9d6f0f6a24795e966fc0",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 13.25,
"retest_logged_model_load_seconds": 64.44897,
"retest_logged_available_KV_GiB": 15.92,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 15.92,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 260288,
"retest_logged_context_capacity_x": 7.77,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w8-tp4-mtp0/service-final.log",
"source_log_sha256": "00785c973ab4b2f5244959832bc6928c3b8483fd02cedd719bad4475c078b8b4",
"sealed_index_sha256": "6bf9c86f222b1baf9d1e5e4e47cf58baa38ee70a9f0b2006431655307d80db75",
"source_lines": {
"model_load": {
"line": 144,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 15:33:02 [gpu_model_runner.py:4584] Model loading took 13.25 GiB memory and 64.448970 seconds"
},
"automatic_KV_budget": {
"line": 166,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 15:34:08 [gpu_worker.py:453] Available KV cache memory: 15.92 GiB"
},
"grouped_KV_tokens": {
"line": 171,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 15:34:08 [kv_cache_utils.py:1314] GPU KV cache size: 260,288 tokens"
},
"context_capacity": {
"line": 172,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 15:34:08 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 7.77x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w8-tp4-mtp0/memory-log-report.json"
}Complete retest loaded admission | Evidence and limitscompleted matrix cell {
"strict_tokens": 14098,
"source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w8-tp4-mtpoff",
"source_revision": "d72edcb54835d75267f4e852d0290847824c584e",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"sampled_VRAM_after_ladder_bytes_by_rank": "[27206279168,27223011328,27222994944,27172745216]",
"comparison_limit": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
"recorded_configuration_details": {
"cell": "w8-tp4-mtpoff",
"bits": 8,
"group_size": 128,
"TP": 4,
"native_MTP": false,
"num_speculative_tokens": 0,
"strict_valid": true,
"official_2k_eligible": true,
"strict_returned_tokens": 14098,
"strict_decode_TPS": 38.863979519826344,
"strict_prefill_seconds": 13.244176653999602,
"strict_total_seconds": 376.0125174979912,
"fixed_2000_returned_tokens": 2000,
"fixed_2000_decode_TPS": 40.860679847930975,
"fixed_2000_prefill_seconds": 15.176513558981242,
"fixed_2000_total_seconds": 64.13704536799924,
"fixed_10000_returned_tokens": 10000,
"fixed_10000_decode_TPS": 39.51520953183532,
"fixed_10000_prefill_seconds": 16.028607391985133,
"fixed_10000_total_seconds": 269.10898952701245,
"packed_weight_bytes_per_rank": 6230318080,
"packed_weight_bytes_total": 24921272320,
"packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
"render_nodes_by_rank": [
"renderD129",
"renderD130",
"renderD131",
"renderD132"
],
"sampled_VRAM_before_ladder_bytes_by_rank": [
27206131712,
27222863872,
27222847488,
27172597760
],
"sampled_VRAM_after_ladder_bytes_by_rank": [
27206279168,
27223011328,
27222994944,
27172745216
],
"allocated_after_load_bytes_by_rank": [
7698443776,
7698443776,
7698443776,
7698443776
],
"allocated_after_cache_bytes_by_rank": [
24779252224,
24779252224,
24779252224,
24779252224
],
"reserved_after_cache_bytes_by_rank": [
25272778752,
25272778752,
25272778752,
25272778752
],
"sampled_attention_KV_first_layer_views_by_rank": [
{
"rank": 0,
"views": [
{
"shape": [
1329,
2,
784,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 1066942464
}
]
},
{
"rank": 1,
"views": [
{
"shape": [
1329,
2,
784,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 1066942464
}
]
},
{
"rank": 2,
"views": [
{
"shape": [
1329,
2,
784,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 1066942464
}
]
},
{
"rank": 3,
"views": [
{
"shape": [
1329,
2,
784,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 1066942464
}
]
}
],
"KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
"KV_budget": "automatic; actual allocation receipts retained",
"context": 131072,
"greedy": true,
"concurrency": 1,
"activation_dtype": "FP16",
"attention_KV_dtype": "FP16",
"SSM_dtype": "float32",
"native_MTP_head_dtype": "not used",
"native_MTP_unshared_parameter_bytes_by_rank": [],
"model_repository": "Qwen/Qwen3.8-27B",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"serving_source": "d72edcb54835d75267f4e852d0290847824c584e",
"image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
"native_MTP_model_source_sha256": "not used",
"MTP_request_accounting_by_tier": {},
"MTP_accounting_scope": "not used",
"runtime_difference": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
"MTP_scheduler_rollback_rejection_count": "not used",
"host_exclusivity": "unproven",
"serving_CPUs": "0-7,16-23",
"cache_placement": "fresh private admitted rootFS",
"compile_cache_disabled": true,
"official_2k_samples": 1,
"repeatability_established": false,
"FP16_goal_qualifying_ladders": 0,
"evidence": "../../tuning/qwen38-rtn-mi50/w8-tp4-result-20261022.json"
},
"detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
"detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
"detail_source_pointer": "w8-tp4-mtpoff",
"retest_strict_tokens": 16128,
"retest_MTP_accounting_by_tier": {},
"retest_logged_available_KV_GiB": 15.92,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 15.92,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 260288,
"retest_memory_log_report": {
"profile": "mi50-w8-tp4-mtp0",
"image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
"container": "5a3463ca7a0d4134be09562c677c092b3a8f8952eb1d9d6f0f6a24795e966fc0",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 13.25,
"retest_logged_model_load_seconds": 64.44897,
"retest_logged_available_KV_GiB": 15.92,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 15.92,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 260288,
"retest_logged_context_capacity_x": 7.77,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w8-tp4-mtp0/service-final.log",
"source_log_sha256": "00785c973ab4b2f5244959832bc6928c3b8483fd02cedd719bad4475c078b8b4",
"sealed_index_sha256": "6bf9c86f222b1baf9d1e5e4e47cf58baa38ee70a9f0b2006431655307d80db75",
"source_lines": {
"model_load": {
"line": 144,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 15:33:02 [gpu_model_runner.py:4584] Model loading took 13.25 GiB memory and 64.448970 seconds"
},
"automatic_KV_budget": {
"line": 166,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 15:34:08 [gpu_worker.py:453] Available KV cache memory: 15.92 GiB"
},
"grouped_KV_tokens": {
"line": 171,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 15:34:08 [kv_cache_utils.py:1314] GPU KV cache size: 260,288 tokens"
},
"context_capacity": {
"line": 172,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 15:34:08 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 7.77x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w8-tp4-mtp0/memory-log-report.json",
"sampled_peak_VRAM_GiB": 25.353404998779297
} |
| MI50Current Qwen3.8 | 8-bit RTNQwen/Qwen3.8-27B | TP2MTP off | PASS7711 tokens | 24.022 | 23.177 | 17.135 / 100.40610K: 17.255 / 448.730Strict: 17.178 / 345.797 | 11.60 / 22.82packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:a506e684a1f3142d071857d89f8fee8f17f0a30e92970c76d163c03380004961Runtime image config: sha256:fe0d81ba97b6b8a432db928d19d03b511f15de9e40e5d25be81797d9c3e1694b Exact compatibility changeReplaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately. | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp2-mtp0 GPU_RENDER_NODES=renderD128,renderD129 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-w8-tp2-mtp0-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-7,16-23 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --user root \ --workdir /tmp \ -e HF_HUB_OFFLINE=1 \ -e TRANSFORMERS_OFFLINE=1 \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \ -e MI50_RTN_BITS=8 \ -e MI50_RTN_TP=2 \ -e HIP_VISIBLE_DEVICES=0,1 \ -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \ -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp0/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp0,dst=/runwork \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp0/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp0/cache/.triton,dst=/root/.triton \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24aHistorical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp2-mtp0 GPU_RENDER_NODES=renderD128,renderD129 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 23.958 · eligible10K 23.139 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 7319,
"content_chars": 16222,
"content_has_think_close": false,
"elapsed_seconds": 329.2658855440095,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T17:16:41Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.008773885667324066,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 16222,
"qwen_answer_text_sha256": "a7be1ad298a7592e1101ba37a866b77cbefeeb356eeaa898e32541ef83392157",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 16222,
"qwen_visible_text_sha256": "a7be1ad298a7592e1101ba37a866b77cbefeeb356eeaa898e32541ef83392157",
"raw_has_think_close": false,
"reasoning_chars": 22466,
"text_chars": 38689,
"text_sha256": "bc50a4ac34f0594c65f428bf83f691971fdfc2cbf8b1dfd3bf586d539eb1b6c9",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 312.1824915353209,
"vllm_decode_tps": 23.444620369339052,
"vllm_generation_tokens_delta": 7319.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 17.07360925897956,
"vllm_prefill_tps": 27.703574143307403,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 22.228236575154536
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 100.557428477332,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T17:18:22Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.00600853469222784,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 10572,
"text_chars": 10572,
"text_sha256": "15ddcc3add7dbf167ef060d6e3340b39995f49d194e0d06e516e1ea573caee41",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 83.47772400546819,
"vllm_decode_tps": 23.958487414786134,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 17.07128318399191,
"vllm_prefill_tps": 27.70734893810102,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 19.889132312595354
},
"10K": {
"completion_tokens": 10000,
"content_chars": 31057,
"content_has_think_close": false,
"elapsed_seconds": 449.2523580007255,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T17:25:51Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.006314177066087723,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 31057,
"qwen_answer_text_sha256": "218046609c9b388c7a955b81e313225e4f1c3821840a1bdf99771f1e43421044",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 31057,
"qwen_visible_text_sha256": "218046609c9b388c7a955b81e313225e4f1c3821840a1bdf99771f1e43421044",
"raw_has_think_close": false,
"reasoning_chars": 22466,
"text_chars": 53524,
"text_sha256": "711f175195df588992b8ece3772dcee43db2f47e50fd5c6d8a23ff24fab7c1e3",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 432.1707717254758,
"vllm_decode_tps": 23.139001187133072,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 17.072354597970843,
"vllm_prefill_tps": 27.705610101152597,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 22.25920425771889
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS13,641 natural tokens2K 24.06710K 23.240 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T16:57:55.889934026Z",
"FinishedAt": "2026-10-26T17:44:23.886246589Z"
},
"devices_after": [
{
"card": 1,
"render": 129,
"pci": "0000:06:00.0",
"actual_card": "card0",
"busy_percent": 0,
"vram_bytes": 10047279104,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:0a:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 8687751168,
"dpm": "auto"
}
],
"finished": "2026-10-26T17:44:24.026755+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 24.06705207684622,
"retest_10K_TPS": 23.240077979944296,
"retest_strict_tokens": 13641,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 13641,
"elapsed_seconds": 614.2203862669412,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 596.9754239380127,
"vllm_decode_tps": 22.85018688041741,
"vllm_generation_tokens_delta": 13641.0,
"vllm_prefill_seconds_delta": 17.231190228951164,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 100.30374919692986,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 83.10116227006074,
"vllm_decode_tps": 24.06705207684622,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 17.19083948188927,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 447.68319084704854,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 430.2911551600555,
"vllm_decode_tps": 23.240077979944296,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 17.377528968965635,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 17.19083948188927,
"retest_2K_total_seconds": 100.30374919692986,
"retest_10K_prefill_seconds": 17.377528968965635,
"retest_10K_total_seconds": 447.68319084704854,
"retest_strict_prefill_seconds": 17.231190228951164,
"retest_strict_total_seconds": 614.2203862669412,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": "Complete record: chart-details/row-05-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 29245403136,
"1": 29245374464
},
"retest_memory_observation_count": 47,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w8-tp2-mtp0/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 25.72,
"retest_logged_model_load_seconds": 117.649267,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 48608,
"retest_logged_context_capacity_x": 1.46,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-w8-tp2-mtp0",
"image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
"container": "08a36b4fd94cf9307b1d837bee7665ffdd40b6e7a7cca6c1a12120691a552194",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 25.72,
"retest_logged_model_load_seconds": 117.649267,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 48608,
"retest_logged_context_capacity_x": 1.46,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w8-tp2-mtp0/service-final.log",
"source_log_sha256": "6383592438234ae67e293c662569aaced83d7ca748bd67ddeed7c4e75341295f",
"sealed_index_sha256": "4cc128355232a7a0e150440bc784203cf03d7b7562ba35660c0b5577679dc4cc",
"source_lines": {
"model_load": {
"line": 118,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:01:41 [gpu_model_runner.py:4584] Model loading took 25.72 GiB memory and 117.649267 seconds"
},
"explicit_KV_budget": {
"line": 7,
"text": "(APIServer pid=1) INFO 10-26 16:58:27 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 8, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'optimization_level': '3'}"
},
"explicit_budget_worker_reports": [
{
"line": 125,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:02:26 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
},
{
"line": 126,
"text": "(Worker pid=197) (Worker_TP1 pid=197) INFO 10-26 17:02:26 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
}
],
"grouped_KV_tokens": {
"line": 127,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 17:02:26 [kv_cache_utils.py:1314] GPU KV cache size: 48,608 tokens"
},
"context_capacity": {
"line": 128,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 17:02:26 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.46x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w8-tp2-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 27.23690414428711
}Complete retest loaded admission | Evidence and limitscompleted matrix cell {
"strict_tokens": 7711,
"source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w8-tp2-mtpoff",
"source_revision": "d72edcb54835d75267f4e852d0290847824c584e",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"sampled_VRAM_after_ladder_bytes_by_rank": "[24503320576,24482340864]",
"comparison_limit": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
"recorded_configuration_details": {
"cell": "w8-tp2-mtpoff",
"bits": 8,
"group_size": 128,
"TP": 2,
"native_MTP": false,
"num_speculative_tokens": 0,
"strict_valid": true,
"official_2k_eligible": true,
"strict_returned_tokens": 7711,
"strict_decode_TPS": 23.46576383615205,
"strict_prefill_seconds": 17.17835429200204,
"strict_total_seconds": 345.7971052789944,
"fixed_2000_returned_tokens": 2000,
"fixed_2000_decode_TPS": 24.02199034482997,
"fixed_2000_prefill_seconds": 17.13484327500919,
"fixed_2000_total_seconds": 100.40560724699753,
"fixed_10000_returned_tokens": 10000,
"fixed_10000_decode_TPS": 23.17706206378481,
"fixed_10000_prefill_seconds": 17.25501243400504,
"fixed_10000_total_seconds": 448.7302281940065,
"packed_weight_bytes_per_rank": 12460636160,
"packed_weight_bytes_total": 24921272320,
"packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
"render_nodes_by_rank": [
"renderD129",
"renderD130"
],
"sampled_VRAM_before_ladder_bytes_by_rank": [
24503320576,
24482336768
],
"sampled_VRAM_after_ladder_bytes_by_rank": [
24503320576,
24482340864
],
"allocated_after_load_bytes_by_rank": [
15213055488,
15213055488
],
"allocated_after_cache_bytes_by_rank": [
21639049728,
21639049728
],
"reserved_after_cache_bytes_by_rank": [
22716350464,
22716350464
],
"sampled_attention_KV_first_layer_views_by_rank": [
{
"rank": 0,
"views": [
{
"shape": [
250,
2,
784,
2,
256
],
"dtype": "torch.float16",
"logical_bytes": 401408000
}
]
},
{
"rank": 1,
"views": [
{
"shape": [
250,
2,
784,
2,
256
],
"dtype": "torch.float16",
"logical_bytes": 401408000
}
]
}
],
"KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
"KV_budget": "6442450944 bytes explicit",
"context": 131072,
"greedy": true,
"concurrency": 1,
"activation_dtype": "FP16",
"attention_KV_dtype": "FP16",
"SSM_dtype": "float32",
"native_MTP_head_dtype": "not used",
"native_MTP_unshared_parameter_bytes_by_rank": [],
"model_repository": "Qwen/Qwen3.8-27B",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"serving_source": "d72edcb54835d75267f4e852d0290847824c584e",
"image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
"native_MTP_model_source_sha256": "not used",
"MTP_request_accounting_by_tier": {},
"MTP_accounting_scope": "not used",
"runtime_difference": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
"MTP_scheduler_rollback_rejection_count": "not used",
"host_exclusivity": "unproven",
"serving_CPUs": "0-7,16-23",
"cache_placement": "fresh private admitted rootFS",
"compile_cache_disabled": true,
"official_2k_samples": 1,
"repeatability_established": false,
"FP16_goal_qualifying_ladders": 0,
"evidence": "../../tuning/qwen38-rtn-mi50/w8-tp2-result-20261022.json"
},
"detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
"detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
"detail_source_pointer": "w8-tp2-mtpoff",
"retest_strict_tokens": 13641,
"retest_MTP_accounting_by_tier": {},
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 29245403136,
"1": 29245374464
},
"retest_memory_observation_count": 47,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w8-tp2-mtp0/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 48608,
"retest_memory_log_report": {
"profile": "mi50-w8-tp2-mtp0",
"image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
"container": "08a36b4fd94cf9307b1d837bee7665ffdd40b6e7a7cca6c1a12120691a552194",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 25.72,
"retest_logged_model_load_seconds": 117.649267,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 48608,
"retest_logged_context_capacity_x": 1.46,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w8-tp2-mtp0/service-final.log",
"source_log_sha256": "6383592438234ae67e293c662569aaced83d7ca748bd67ddeed7c4e75341295f",
"sealed_index_sha256": "4cc128355232a7a0e150440bc784203cf03d7b7562ba35660c0b5577679dc4cc",
"source_lines": {
"model_load": {
"line": 118,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:01:41 [gpu_model_runner.py:4584] Model loading took 25.72 GiB memory and 117.649267 seconds"
},
"explicit_KV_budget": {
"line": 7,
"text": "(APIServer pid=1) INFO 10-26 16:58:27 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 8, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'optimization_level': '3'}"
},
"explicit_budget_worker_reports": [
{
"line": 125,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:02:26 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
},
{
"line": 126,
"text": "(Worker pid=197) (Worker_TP1 pid=197) INFO 10-26 17:02:26 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
}
],
"grouped_KV_tokens": {
"line": 127,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 17:02:26 [kv_cache_utils.py:1314] GPU KV cache size: 48,608 tokens"
},
"context_capacity": {
"line": 128,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 17:02:26 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.46x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w8-tp2-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 27.23690414428711,
"sampled_peak_VRAM_GiB": 22.82049560546875
} |
| MI50Current Qwen3.8 | 8-bit RTNQwen/Qwen3.8-27B | TP4MTP22K accepted 1085 / 1832 (59.22%) | PASS6410 tokens | 30.280 | 25.717 | 14.502 / 80.56510K: 14.462 / 403.327Strict: 14.460 / 236.118 | 5.80 / 23.93packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:1ee2a65021b0a54e0f8784ea5e7487504cb48741de9af1bad226b5427c4fa96bRuntime image config: sha256:b1450af604a4f69af53a3ac83f783d9572fcfafb3705c44d5730d50ba99baaad Exact compatibility changeReplaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately. | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp4-mtp2 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-w8-tp4-mtp2-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-7,16-23 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \ --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \ --user root \ --workdir /tmp \ -e MI50_RTN_TP=4 \ -e HIP_VISIBLE_DEVICES=0,1,2,3 \ -e HF_HUB_OFFLINE=1 \ -e MI50_RTN_BITS=8 \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \ -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \ -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \ -e TRANSFORMERS_OFFLINE=1 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ -e MAX_JOBS=1 \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp2/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp2/cache/.triton,dst=/root/.triton \ --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp2/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp4-mtp2,dst=/runwork \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7Historical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp4-mtp2 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 34.170 · eligible10K 27.357 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 6977,
"content_chars": 15337,
"content_has_think_close": false,
"elapsed_seconds": 241.16507857106626,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T17:59:50Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.007626298815011978,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 15337,
"qwen_answer_text_sha256": "3d4173726447a9c1f69d408bd8ce8c8fbcf609a93736ca85a0879ec41ea6dd58",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 15337,
"qwen_visible_text_sha256": "3d4173726447a9c1f69d408bd8ce8c8fbcf609a93736ca85a0879ec41ea6dd58",
"raw_has_think_close": false,
"reasoning_chars": 20610,
"text_chars": 35948,
"text_sha256": "ef0d1ca01a68cb4926196ff3a217df69b6d80218a4a3509a6f6eb16ff6a0368d",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 230.64176619797945,
"vllm_decode_tps": 30.250375354874137,
"vllm_generation_tokens_delta": 6977.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 10.514229037798941,
"vllm_prefill_tps": 44.986655540748835,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 28.930390922846755
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 68.95196704845876,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T18:01:00Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.005916460417211056,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 10087,
"text_chars": 10087,
"text_sha256": "0e8ac24e709139ba097dd23070a7a19e68a3eb2378b79f656bd3e03ffa3763ae",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 58.531507852487266,
"vllm_decode_tps": 34.16963056958067,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 10.413011768832803,
"vllm_prefill_tps": 45.42393790581672,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 29.005698975845316
},
"10K": {
"completion_tokens": 10000,
"content_chars": 31161,
"content_has_think_close": false,
"elapsed_seconds": 376.00577663164586,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T18:07:16Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.007972470484673977,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 31161,
"qwen_answer_text_sha256": "7f3809ce4035f946acab56de8b0a0f62f14f1542f6c57c680a966286083b7820",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 31161,
"qwen_visible_text_sha256": "7f3809ce4035f946acab56de8b0a0f62f14f1542f6c57c680a966286083b7820",
"raw_has_think_close": false,
"reasoning_chars": 20610,
"text_chars": 51772,
"text_sha256": "5efdd6da02c5c6693aab66ce303140d9d7a0ffae7e22abdc30844352272582c9",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 365.5408342368901,
"vllm_decode_tps": 27.35672478527929,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 10.456269898451865,
"vllm_prefill_tps": 45.23601672428439,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 26.595336086542368
}
}Earlier exported-image retest · separate evidenceRetest completed; strict failed; fixed speeds diagnostic onlyVerification · Raw retestMTP counter replayFAIL5,065 natural tokens2K 33.452*10K 26.146* · diagnostic2K MTP accepted 1178 / 1644 (71.65%) Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T18:58:23.196020003Z",
"FinishedAt": "2026-10-26T19:33:39.95166382Z"
},
"devices_after": [
{
"card": 1,
"render": 129,
"pci": "0000:06:00.0",
"actual_card": "card0",
"busy_percent": 0,
"vram_bytes": 19930054656,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:0a:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 19457867776,
"dpm": "auto"
},
{
"card": 3,
"render": 131,
"pci": "0000:0d:00.0",
"actual_card": "card3",
"busy_percent": 0,
"vram_bytes": 19892056064,
"dpm": "auto"
},
{
"card": 4,
"render": 132,
"pci": "0000:10:00.0",
"actual_card": "card4",
"busy_percent": 0,
"vram_bytes": 19969708032,
"dpm": "auto"
}
],
"finished": "2026-10-26T19:33:40.094119+00:00"
},
"retest_strict": "FAIL",
"retest_official_2K_eligible": false,
"retest_2K_TPS": 33.45204890335249,
"retest_10K_TPS": 26.145610878939074,
"retest_strict_tokens": 5065,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 5065,
"elapsed_seconds": 198.25056633690838,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 175.8506427659886,
"vllm_decode_tps": 28.802851785649686,
"vllm_generation_tokens_delta": 5065.0,
"vllm_prefill_seconds_delta": 22.387725064065307,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 82.28246686095372,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 59.78707031602971,
"vllm_decode_tps": 33.45204890335249,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 22.482500477926806,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 404.386289892951,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 382.4733736879425,
"vllm_decode_tps": 26.145610878939074,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 21.900252138031647,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 22.482500477926806,
"retest_2K_total_seconds": 82.28246686095372,
"retest_10K_prefill_seconds": 21.900252138031647,
"retest_10K_total_seconds": 404.386289892951,
"retest_strict_prefill_seconds": 22.387725064065307,
"retest_strict_total_seconds": 198.25056633690838,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 4156.0,
"vllm:request_generation_tokens_sum": 5065.0,
"vllm:spec_decode_num_accepted_tokens_total": 2986.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 2986.0,
"vllm:spec_decode_num_drafts_total": 2078.0,
"vllm:generation_tokens_total": 5065.0,
"vllm:request_generation_tokens_count": 1.0
},
"accepted_per_position": {
"0": 1725.0,
"1": 1261.0
},
"accepted_fraction": 0.7184793070259865,
"committed_emitted_tokens": 5065.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "52626453501dda855f1e312dff99fb8be8ab0168b8ceb2aed5f39894f51dcec0",
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "d08dc97c94d35a272c3296a02796beec082e2f0f769b29ee9dbc79fd8d61a501"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_drafts_total": 822.0,
"vllm:spec_decode_num_accepted_tokens_total": 1178.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1178.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_draft_tokens_total": 1644.0
},
"accepted_per_position": {
"0": 669.0,
"1": 509.0
},
"accepted_fraction": 0.7165450121654501,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "afedac1311f18a5f412481eecac893a8c4e54b59275f102c389b4eb471af9cd8",
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "91830c16667855b20d0b7897c58007bd337986836036f4de1a6476a12faf324b"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 6275.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6275.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:spec_decode_num_draft_tokens_total": 7452.0,
"vllm:spec_decode_num_drafts_total": 3726.0
},
"accepted_per_position": {
"0": 3373.0,
"1": 2902.0
},
"accepted_fraction": 0.8420558239398819,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "7f92ce7ac776a0f277eabd37bb200446633e17d526b88b9190bc2ad03f51111a",
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "5016c951dee5e71c622875bcd6b1bd81c125066f0d8a119b07898f086e4ad657"
}
}
},
"retest_MTP_2K_accepted": 1178.0,
"retest_MTP_2K_proposed": 1644.0,
"retest_MTP_2K_draft_batches": 822.0,
"retest_MTP_2K_acceptance_percent": 71.65450121654501,
"retest_loaded_admission": "Complete record: chart-details/row-06-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 1, \"fixed\": 1, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
"retest_diagnostic_counter_audit": "retest-evidence/mi50-w8-tp4-mtp2/local-diagnostic-fixed-audit.json",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 25681870848,
"1": 25698664448,
"2": 25698631680,
"3": 25648300032
},
"retest_memory_observation_count": 34,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w8-tp4-mtp2/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 14.43,
"retest_logged_model_load_seconds": 63.480969,
"retest_logged_available_KV_GiB": 14.35,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 14.35,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 220800,
"retest_logged_context_capacity_x": 6.39,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-w8-tp4-mtp2",
"image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
"container": "2901da8586316a8b0a053199945a78afd8c877762f777773550bd1a8463cbba4",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 14.43,
"retest_logged_model_load_seconds": 63.480969,
"retest_logged_available_KV_GiB": 14.35,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 14.35,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 220800,
"retest_logged_context_capacity_x": 6.39,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w8-tp4-mtp2/service-final.log",
"source_log_sha256": "37ad86fe2abc818ffd5877f746e3891fdfc6c948a6eec70e813d4b171a7d0e3d",
"sealed_index_sha256": "cad06a029821c6b96a4260e7a9e1f3938969ecef85de855a0ab156d226b112e4",
"source_lines": {
"model_load": {
"line": 178,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:01:37 [gpu_model_runner.py:4584] Model loading took 14.43 GiB memory and 63.480969 seconds"
},
"automatic_KV_budget": {
"line": 265,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:06:40 [gpu_worker.py:453] Available KV cache memory: 14.35 GiB"
},
"grouped_KV_tokens": {
"line": 269,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 19:06:40 [kv_cache_utils.py:1314] GPU KV cache size: 220,800 tokens"
},
"context_capacity": {
"line": 270,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 19:06:40 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.39x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w8-tp4-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 23.933746337890625
}Complete retest loaded admission | Evidence and limitscompleted matrix cell {
"strict_tokens": 6410,
"source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w8-tp4-mtp2",
"source_revision": "016143b00c7a08e25ff85eb83dc052be148215ab",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"sampled_VRAM_after_ladder_bytes_by_rank": "[25681883136,25698689024,25698684928,25648316416]",
"comparison_limit": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
"recorded_configuration_details": {
"cell": "w8-tp4-mtp2",
"bits": 8,
"group_size": 128,
"TP": 4,
"native_MTP": true,
"num_speculative_tokens": 2,
"strict_valid": true,
"official_2k_eligible": true,
"strict_returned_tokens": 6410,
"strict_decode_TPS": 28.920106957529065,
"strict_prefill_seconds": 14.459649450000143,
"strict_total_seconds": 236.11842984199757,
"fixed_2000_returned_tokens": 2000,
"fixed_2000_decode_TPS": 30.279763374132372,
"fixed_2000_prefill_seconds": 14.502473675995134,
"fixed_2000_total_seconds": 80.56469545399887,
"fixed_10000_returned_tokens": 10000,
"fixed_10000_decode_TPS": 25.71673654115124,
"fixed_10000_prefill_seconds": 14.462411346990848,
"fixed_10000_total_seconds": 403.3265637109871,
"packed_weight_bytes_per_rank": 6230318080,
"packed_weight_bytes_total": 24921272320,
"packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
"render_nodes_by_rank": [
"renderD129",
"renderD130",
"renderD131",
"renderD132"
],
"sampled_VRAM_before_ladder_bytes_by_rank": [
25681883136,
25698689024,
25698684928,
25648316416
],
"sampled_VRAM_after_ladder_bytes_by_rank": [
25681883136,
25698689024,
25698684928,
25648316416
],
"allocated_after_load_bytes_by_rank": [
7913016320,
7913016320,
7913016320,
7913016320
],
"allocated_after_cache_bytes_by_rank": [
23317684224,
23317684224,
23317684224,
23317684224
],
"reserved_after_cache_bytes_by_rank": [
23886561280,
23886561280,
23886561280,
23886561280
],
"sampled_attention_KV_first_layer_views_by_rank": [
{
"rank": 0,
"views": [
{
"shape": [
1105,
2,
800,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 905216000
}
]
},
{
"rank": 1,
"views": [
{
"shape": [
1105,
2,
800,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 905216000
}
]
},
{
"rank": 2,
"views": [
{
"shape": [
1105,
2,
800,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 905216000
}
]
},
{
"rank": 3,
"views": [
{
"shape": [
1105,
2,
800,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 905216000
}
]
}
],
"KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
"KV_budget": "automatic; actual allocation receipts retained",
"context": 131072,
"greedy": true,
"concurrency": 1,
"activation_dtype": "FP16",
"attention_KV_dtype": "FP16",
"SSM_dtype": "float32",
"native_MTP_head_dtype": "FP16",
"native_MTP_unshared_parameter_bytes_by_rank": [
212388864,
212388864,
212388864,
212388864
],
"model_repository": "Qwen/Qwen3.8-27B",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"serving_source": "016143b00c7a08e25ff85eb83dc052be148215ab",
"image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
"native_MTP_model_source_sha256": "46f57b9c511c9199aa83421fa7fb167b08d5d80c14377948690cdf67d75453c3",
"MTP_request_accounting_by_tier": {
"c1_128_strict": {
"drafts": 2517.0,
"proposed": 5034.0,
"accepted": 3894.0,
"adjusted_unaccepted": 1140.0,
"generation": 6410.0,
"prompt": 473.0,
"histogram_count": 2518.0,
"histogram_sum": 6883.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.733518665607625,
"generation_per_all_observed_iterations": 2.545671167593328,
"sum_equals_prompt_plus_generation": true
}
},
"c1_2000": {
"drafts": 916.0,
"proposed": 1832.0,
"accepted": 1085.0,
"adjusted_unaccepted": 747.0,
"generation": 2000.0,
"prompt": 473.0,
"histogram_count": 917.0,
"histogram_sum": 2473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.6968375136314067,
"generation_per_all_observed_iterations": 2.1810250817884405,
"sum_equals_prompt_plus_generation": true
}
},
"c1_10000": {
"drafts": 3808.0,
"proposed": 7616.0,
"accepted": 6191.0,
"adjusted_unaccepted": 1425.0,
"generation": 10000.0,
"prompt": 473.0,
"histogram_count": 3809.0,
"histogram_sum": 10473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.749540561827251,
"generation_per_all_observed_iterations": 2.625360987135731,
"sum_equals_prompt_plus_generation": true
}
}
},
"MTP_accounting_scope": "P-A is adjusted unaccepted, not scheduler rollback rejection; iteration means include prefill, not exact decode-only emissions.",
"runtime_difference": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
"MTP_scheduler_rollback_rejection_count": "unobservable",
"host_exclusivity": "unproven",
"serving_CPUs": "0-7,16-23",
"cache_placement": "fresh private admitted rootFS",
"compile_cache_disabled": true,
"official_2k_samples": 1,
"repeatability_established": false,
"FP16_goal_qualifying_ladders": 0,
"evidence": "../../tuning/qwen38-rtn-mi50/w8-tp4-mtp-result-20261022.json"
},
"MTP_accounting_by_tier": {
"c1_128_strict": {
"drafts": 2517.0,
"proposed": 5034.0,
"accepted": 3894.0,
"adjusted_unaccepted": 1140.0,
"generation": 6410.0,
"prompt": 473.0,
"histogram_count": 2518.0,
"histogram_sum": 6883.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.733518665607625,
"generation_per_all_observed_iterations": 2.545671167593328,
"sum_equals_prompt_plus_generation": true
}
},
"c1_2000": {
"drafts": 916.0,
"proposed": 1832.0,
"accepted": 1085.0,
"adjusted_unaccepted": 747.0,
"generation": 2000.0,
"prompt": 473.0,
"histogram_count": 917.0,
"histogram_sum": 2473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.6968375136314067,
"generation_per_all_observed_iterations": 2.1810250817884405,
"sum_equals_prompt_plus_generation": true
}
},
"c1_10000": {
"drafts": 3808.0,
"proposed": 7616.0,
"accepted": 6191.0,
"adjusted_unaccepted": 1425.0,
"generation": 10000.0,
"prompt": 473.0,
"histogram_count": 3809.0,
"histogram_sum": 10473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.749540561827251,
"generation_per_all_observed_iterations": 2.625360987135731,
"sum_equals_prompt_plus_generation": true
}
}
},
"MTP_2K_draft_batches": 916.0,
"MTP_2K_proposed": 1832.0,
"MTP_2K_accepted": 1085.0,
"MTP_2K_acceptance_percent": 59.224890829694324,
"MTP_2K_committed": 2000.0,
"detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
"detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
"detail_source_pointer": "w8-tp4-mtp2",
"retest_strict_tokens": 5065,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 4156.0,
"vllm:request_generation_tokens_sum": 5065.0,
"vllm:spec_decode_num_accepted_tokens_total": 2986.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 2986.0,
"vllm:spec_decode_num_drafts_total": 2078.0,
"vllm:generation_tokens_total": 5065.0,
"vllm:request_generation_tokens_count": 1.0
},
"accepted_per_position": {
"0": 1725.0,
"1": 1261.0
},
"accepted_fraction": 0.7184793070259865,
"committed_emitted_tokens": 5065.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "52626453501dda855f1e312dff99fb8be8ab0168b8ceb2aed5f39894f51dcec0",
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "d08dc97c94d35a272c3296a02796beec082e2f0f769b29ee9dbc79fd8d61a501"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_drafts_total": 822.0,
"vllm:spec_decode_num_accepted_tokens_total": 1178.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1178.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_draft_tokens_total": 1644.0
},
"accepted_per_position": {
"0": 669.0,
"1": 509.0
},
"accepted_fraction": 0.7165450121654501,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "afedac1311f18a5f412481eecac893a8c4e54b59275f102c389b4eb471af9cd8",
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "91830c16667855b20d0b7897c58007bd337986836036f4de1a6476a12faf324b"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 6275.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6275.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:spec_decode_num_draft_tokens_total": 7452.0,
"vllm:spec_decode_num_drafts_total": 3726.0
},
"accepted_per_position": {
"0": 3373.0,
"1": 2902.0
},
"accepted_fraction": 0.8420558239398819,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "7f92ce7ac776a0f277eabd37bb200446633e17d526b88b9190bc2ad03f51111a",
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "5016c951dee5e71c622875bcd6b1bd81c125066f0d8a119b07898f086e4ad657"
}
}
},
"retest_MTP_2K_accepted": 1178.0,
"retest_MTP_2K_proposed": 1644.0,
"retest_MTP_2K_draft_batches": 822.0,
"retest_MTP_2K_acceptance_percent": 71.65450121654501,
"retest_diagnostic_counter_audit": "retest-evidence/mi50-w8-tp4-mtp2/local-diagnostic-fixed-audit.json",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 25681870848,
"1": 25698664448,
"2": 25698631680,
"3": 25648300032
},
"retest_memory_observation_count": 34,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w8-tp4-mtp2/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 14.35,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 14.35,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 220800,
"retest_memory_log_report": {
"profile": "mi50-w8-tp4-mtp2",
"image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
"container": "2901da8586316a8b0a053199945a78afd8c877762f777773550bd1a8463cbba4",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 14.43,
"retest_logged_model_load_seconds": 63.480969,
"retest_logged_available_KV_GiB": 14.35,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 14.35,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 220800,
"retest_logged_context_capacity_x": 6.39,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w8-tp4-mtp2/service-final.log",
"source_log_sha256": "37ad86fe2abc818ffd5877f746e3891fdfc6c948a6eec70e813d4b171a7d0e3d",
"sealed_index_sha256": "cad06a029821c6b96a4260e7a9e1f3938969ecef85de855a0ab156d226b112e4",
"source_lines": {
"model_load": {
"line": 178,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:01:37 [gpu_model_runner.py:4584] Model loading took 14.43 GiB memory and 63.480969 seconds"
},
"automatic_KV_budget": {
"line": 265,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:06:40 [gpu_worker.py:453] Available KV cache memory: 14.35 GiB"
},
"grouped_KV_tokens": {
"line": 269,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 19:06:40 [kv_cache_utils.py:1314] GPU KV cache size: 220,800 tokens"
},
"context_capacity": {
"line": 270,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 19:06:40 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.39x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w8-tp4-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 23.933746337890625,
"sampled_peak_VRAM_GiB": 23.93376922607422
} |
| MI50Current Qwen3.8 | 8-bit RTNQwen/Qwen3.8-27B | TP2MTP22K accepted 1138 / 1724 (66.01%) | PASS8595 tokens | 19.479 | 17.083 | 18.051 / 120.73810K: 18.045 / 603.450Strict: 18.056 / 501.280 | 11.60 / 23.27packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:1ee2a65021b0a54e0f8784ea5e7487504cb48741de9af1bad226b5427c4fa96bRuntime image config: sha256:b1450af604a4f69af53a3ac83f783d9572fcfafb3705c44d5730d50ba99baaad Exact compatibility changeReplaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately. | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp2-mtp2 GPU_RENDER_NODES=renderD128,renderD129 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-w8-tp2-mtp2-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-7,16-23 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --user root \ --workdir /tmp \ -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \ -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \ -e MI50_RTN_BITS=8 \ -e HIP_VISIBLE_DEVICES=0,1 \ -e TRANSFORMERS_OFFLINE=1 \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \ -e MI50_RTN_TP=2 \ -e HF_HUB_OFFLINE=1 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ -e MAX_JOBS=1 \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp2/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp2/cache/.triton,dst=/root/.triton \ --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp2/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp2,dst=/runwork \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp2-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7Historical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp2-mtp2 GPU_RENDER_NODES=renderD128,renderD129 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w8-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 18.427 · eligible10K 16.555 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 17304,
"content_chars": 15852,
"content_has_think_close": false,
"elapsed_seconds": 1191.0941870175302,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T19:08:11Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.00875687412917614,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 15852,
"qwen_answer_text_sha256": "36bfb02ddbea6ad955edaccfcbdc78935636fadd3c34ac94c890d877d6ade3d7",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 15852,
"qwen_visible_text_sha256": "36bfb02ddbea6ad955edaccfcbdc78935636fadd3c34ac94c890d877d6ade3d7",
"raw_has_think_close": false,
"reasoning_chars": 74240,
"text_chars": 90093,
"text_sha256": "af766f503825569b7901850243fb5fb03e584e6691d634ed3a89c9873e3b730a",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 1173.0919499928132,
"vllm_decode_tps": 14.75076186492117,
"vllm_generation_tokens_delta": 17304.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 17.993215464055538,
"vllm_prefill_tps": 26.287686097290212,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 14.527818361139667
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 126.62693756446242,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T19:10:18Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008112105540931225,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9243,
"text_chars": 9243,
"text_sha256": "c23f3a143c4c1bb3d52df4c7713c11ee2fa0b4c7ef0afc41c3f94f658443aa44",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 108.535968715325,
"vllm_decode_tps": 18.427070985524868,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 17.998336000368,
"vllm_prefill_tps": 26.280207236398347,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 15.79442761917742
},
"10K": {
"completion_tokens": 10000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 622.0402731243521,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T19:20:40Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.007797887548804283,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 51149,
"text_chars": 51149,
"text_sha256": "7df23c557b4477aa92c6e33883414281ad35c7c170650fa101d9593bd5e39d64",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 604.0367387775332,
"vllm_decode_tps": 16.555284402465794,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 17.99445286858827,
"vllm_prefill_tps": 26.28587840120913,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 16.076129524174554
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestMTP counter replayPASS6,643 natural tokens2K 18.08110K 17.682 · diagnostic2K MTP accepted 1077 / 1844 (58.41%) Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T19:35:52.777581181Z",
"FinishedAt": "2026-10-26T20:27:29.68673334Z"
},
"devices_after": [
{
"card": 1,
"render": 129,
"pci": "0000:06:00.0",
"actual_card": "card0",
"busy_percent": 0,
"vram_bytes": 17268699136,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:0a:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 17447673856,
"dpm": "auto"
}
],
"finished": "2026-10-26T20:27:29.842167+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 18.08137387676853,
"retest_10K_TPS": 17.681917045281846,
"retest_strict_tokens": 6643,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 6643,
"elapsed_seconds": 381.4662659961032,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 361.9941003039712,
"vllm_decode_tps": 18.351127806839354,
"vllm_generation_tokens_delta": 6643.0,
"vllm_prefill_seconds_delta": 19.46000845602248,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 129.1342354209628,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 110.61106382904109,
"vllm_decode_tps": 18.08137387676853,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 18.51244649104774,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 586.0752006319817,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 565.5495371000143,
"vllm_decode_tps": 17.681917045281846,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 20.5129211419262,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 18.51244649104774,
"retest_2K_total_seconds": 129.1342354209628,
"retest_10K_prefill_seconds": 20.5129211419262,
"retest_10K_total_seconds": 586.0752006319817,
"retest_strict_prefill_seconds": 19.46000845602248,
"retest_strict_total_seconds": 381.4662659961032,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_drafts_total": 2651.0,
"vllm:spec_decode_num_draft_tokens_total": 5302.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_total": 3991.0,
"vllm:request_generation_tokens_sum": 6643.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 3991.0,
"vllm:generation_tokens_total": 6643.0
},
"accepted_per_position": {
"0": 2190.0,
"1": 1801.0
},
"accepted_fraction": 0.7527348170501698,
"committed_emitted_tokens": 6643.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "ee5967d10ba0e760ed2c0a067f815ef122fb81d0ea3f9fc41067bbc21c23493b",
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "13f8368194830434abc764415caecd01230d8c66b06bfa2260089fd04e785f83"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 1844.0,
"vllm:spec_decode_num_accepted_tokens_total": 1077.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1077.0,
"vllm:spec_decode_num_drafts_total": 922.0,
"vllm:generation_tokens_total": 2000.0
},
"accepted_per_position": {
"0": 653.0,
"1": 424.0
},
"accepted_fraction": 0.584056399132321,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "0c4587f09022dd78756b9c5bf8a50385d3399caa00b05b618df181b715b0190f",
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "0f55e3c3423fdc6d85ab588c3654e51882febc921946a4ec66cac49e21e74a78"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 7582.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6208.0,
"vllm:spec_decode_num_accepted_tokens_total": 6208.0,
"vllm:spec_decode_num_drafts_total": 3791.0,
"vllm:generation_tokens_total": 10000.0
},
"accepted_per_position": {
"0": 3310.0,
"1": 2898.0
},
"accepted_fraction": 0.8187813241888684,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "e806c3d489768238aaa0c29d2db857e00cbefa0f0aba50d5e411a7344842ee89",
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "67a912927c33d611ba2f51361c6d0b04ffbbae8ff08a9f92a137738da181cb21"
}
}
},
"retest_MTP_2K_accepted": 1077.0,
"retest_MTP_2K_proposed": 1844.0,
"retest_MTP_2K_draft_batches": 922.0,
"retest_MTP_2K_acceptance_percent": 58.4056399132321,
"retest_loaded_admission": "Complete record: chart-details/row-07-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30519541760,
"1": 30519545856
},
"retest_memory_observation_count": 49,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w8-tp2-mtp2/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 28.09,
"retest_logged_model_load_seconds": 123.709991,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 45600,
"retest_logged_context_capacity_x": 1.34,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-w8-tp2-mtp2",
"image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
"container": "4b05f105aed3db4ec411d4497fe439e146cb346b8d83fd5b1527647707fdcef3",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 28.09,
"retest_logged_model_load_seconds": 123.709991,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 45600,
"retest_logged_context_capacity_x": 1.34,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w8-tp2-mtp2/service-final.log",
"source_log_sha256": "553939c869da8d99f5604c5504ab31cb9157600462eed3a586e6c78531ee843f",
"sealed_index_sha256": "0bae66b49db8c96e0a89da91a15f660adb325e42c22238bd46faecbe24194dbe",
"source_lines": {
"model_load": {
"line": 144,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:40:05 [gpu_model_runner.py:4584] Model loading took 28.09 GiB memory and 123.709991 seconds"
},
"explicit_KV_budget": {
"line": 7,
"text": "(APIServer pid=1) INFO 10-26 19:36:24 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 8, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 2}, 'optimization_level': '3'}"
},
"explicit_budget_worker_reports": [
{
"line": 156,
"text": "(Worker pid=217) (Worker_TP1 pid=217) INFO 10-26 19:41:01 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
},
{
"line": 157,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:41:01 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
}
],
"grouped_KV_tokens": {
"line": 159,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 19:41:01 [kv_cache_utils.py:1314] GPU KV cache size: 45,600 tokens"
},
"context_capacity": {
"line": 160,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 19:41:01 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.34x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w8-tp2-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.423542022705078
}Complete retest loaded admission | Evidence and limitscompleted matrix cell {
"strict_tokens": 8595,
"source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w8-tp2-mtp2",
"source_revision": "016143b00c7a08e25ff85eb83dc052be148215ab",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"sampled_VRAM_after_ladder_bytes_by_rank": "[24989736960,24989675520]",
"comparison_limit": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
"recorded_configuration_details": {
"cell": "w8-tp2-mtp2",
"bits": 8,
"group_size": 128,
"TP": 2,
"native_MTP": true,
"num_speculative_tokens": 2,
"strict_valid": true,
"official_2k_eligible": true,
"strict_returned_tokens": 8595,
"strict_decode_TPS": 17.78735896682357,
"strict_prefill_seconds": 18.055707121035084,
"strict_total_seconds": 501.27962843398564,
"fixed_2000_returned_tokens": 2000,
"fixed_2000_decode_TPS": 19.47859055870054,
"fixed_2000_prefill_seconds": 18.05051256698789,
"fixed_2000_total_seconds": 120.73758545302553,
"fixed_10000_returned_tokens": 10000,
"fixed_10000_decode_TPS": 17.08257754036289,
"fixed_10000_prefill_seconds": 18.045061738986988,
"fixed_10000_total_seconds": 603.449919919949,
"packed_weight_bytes_per_rank": 12460636160,
"packed_weight_bytes_total": 24921272320,
"packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
"render_nodes_by_rank": [
"renderD129",
"renderD130"
],
"sampled_VRAM_before_ladder_bytes_by_rank": [
24989736960,
24989675520
],
"sampled_VRAM_after_ladder_bytes_by_rank": [
24989736960,
24989675520
],
"allocated_after_load_bytes_by_rank": [
15637867520,
15637867520
],
"allocated_after_cache_bytes_by_rank": [
22076637184,
22076637184
],
"reserved_after_cache_bytes_by_rank": [
23408410624,
23408410624
],
"sampled_attention_KV_first_layer_views_by_rank": [
{
"rank": 0,
"views": [
{
"shape": [
231,
2,
800,
2,
256
],
"dtype": "torch.float16",
"logical_bytes": 378470400
}
]
},
{
"rank": 1,
"views": [
{
"shape": [
231,
2,
800,
2,
256
],
"dtype": "torch.float16",
"logical_bytes": 378470400
}
]
}
],
"KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
"KV_budget": "6442450944 bytes explicit",
"context": 131072,
"greedy": true,
"concurrency": 1,
"activation_dtype": "FP16",
"attention_KV_dtype": "FP16",
"SSM_dtype": "float32",
"native_MTP_head_dtype": "FP16",
"native_MTP_unshared_parameter_bytes_by_rank": [
424725504,
424725504
],
"model_repository": "Qwen/Qwen3.8-27B",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"serving_source": "016143b00c7a08e25ff85eb83dc052be148215ab",
"image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
"native_MTP_model_source_sha256": "46f57b9c511c9199aa83421fa7fb167b08d5d80c14377948690cdf67d75453c3",
"MTP_request_accounting_by_tier": {
"c1_128_strict": {
"drafts": 3364.0,
"proposed": 6728.0,
"accepted": 5230.0,
"adjusted_unaccepted": 1498.0,
"generation": 8595.0,
"prompt": 473.0,
"histogram_count": 3365.0,
"histogram_sum": 9068.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.69479940564636,
"generation_per_all_observed_iterations": 2.554234769687964,
"sum_equals_prompt_plus_generation": true
}
},
"c1_2000": {
"drafts": 862.0,
"proposed": 1724.0,
"accepted": 1138.0,
"adjusted_unaccepted": 586.0,
"generation": 2000.0,
"prompt": 473.0,
"histogram_count": 863.0,
"histogram_sum": 2473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.86558516801854,
"generation_per_all_observed_iterations": 2.317497103128621,
"sum_equals_prompt_plus_generation": true
}
},
"c1_10000": {
"drafts": 3914.0,
"proposed": 7828.0,
"accepted": 6087.0,
"adjusted_unaccepted": 1741.0,
"generation": 10000.0,
"prompt": 473.0,
"histogram_count": 3915.0,
"histogram_sum": 10473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.675095785440613,
"generation_per_all_observed_iterations": 2.554278416347382,
"sum_equals_prompt_plus_generation": true
}
}
},
"MTP_accounting_scope": "P-A is adjusted unaccepted, not scheduler rollback rejection; iteration means include prefill, not exact decode-only emissions.",
"runtime_difference": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
"MTP_scheduler_rollback_rejection_count": "unobservable",
"host_exclusivity": "unproven",
"serving_CPUs": "0-7,16-23",
"cache_placement": "fresh private admitted rootFS",
"compile_cache_disabled": true,
"official_2k_samples": 1,
"repeatability_established": false,
"FP16_goal_qualifying_ladders": 0,
"evidence": "../../tuning/qwen38-rtn-mi50/w8-tp2-mtp-result-20261023.json"
},
"MTP_accounting_by_tier": {
"c1_128_strict": {
"drafts": 3364.0,
"proposed": 6728.0,
"accepted": 5230.0,
"adjusted_unaccepted": 1498.0,
"generation": 8595.0,
"prompt": 473.0,
"histogram_count": 3365.0,
"histogram_sum": 9068.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.69479940564636,
"generation_per_all_observed_iterations": 2.554234769687964,
"sum_equals_prompt_plus_generation": true
}
},
"c1_2000": {
"drafts": 862.0,
"proposed": 1724.0,
"accepted": 1138.0,
"adjusted_unaccepted": 586.0,
"generation": 2000.0,
"prompt": 473.0,
"histogram_count": 863.0,
"histogram_sum": 2473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.86558516801854,
"generation_per_all_observed_iterations": 2.317497103128621,
"sum_equals_prompt_plus_generation": true
}
},
"c1_10000": {
"drafts": 3914.0,
"proposed": 7828.0,
"accepted": 6087.0,
"adjusted_unaccepted": 1741.0,
"generation": 10000.0,
"prompt": 473.0,
"histogram_count": 3915.0,
"histogram_sum": 10473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.675095785440613,
"generation_per_all_observed_iterations": 2.554278416347382,
"sum_equals_prompt_plus_generation": true
}
}
},
"MTP_2K_draft_batches": 862.0,
"MTP_2K_proposed": 1724.0,
"MTP_2K_accepted": 1138.0,
"MTP_2K_acceptance_percent": 66.0092807424594,
"MTP_2K_committed": 2000.0,
"detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
"detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
"detail_source_pointer": "w8-tp2-mtp2",
"retest_strict_tokens": 6643,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_drafts_total": 2651.0,
"vllm:spec_decode_num_draft_tokens_total": 5302.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_total": 3991.0,
"vllm:request_generation_tokens_sum": 6643.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 3991.0,
"vllm:generation_tokens_total": 6643.0
},
"accepted_per_position": {
"0": 2190.0,
"1": 1801.0
},
"accepted_fraction": 0.7527348170501698,
"committed_emitted_tokens": 6643.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "ee5967d10ba0e760ed2c0a067f815ef122fb81d0ea3f9fc41067bbc21c23493b",
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "13f8368194830434abc764415caecd01230d8c66b06bfa2260089fd04e785f83"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 1844.0,
"vllm:spec_decode_num_accepted_tokens_total": 1077.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1077.0,
"vllm:spec_decode_num_drafts_total": 922.0,
"vllm:generation_tokens_total": 2000.0
},
"accepted_per_position": {
"0": 653.0,
"1": 424.0
},
"accepted_fraction": 0.584056399132321,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "0c4587f09022dd78756b9c5bf8a50385d3399caa00b05b618df181b715b0190f",
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "0f55e3c3423fdc6d85ab588c3654e51882febc921946a4ec66cac49e21e74a78"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 7582.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6208.0,
"vllm:spec_decode_num_accepted_tokens_total": 6208.0,
"vllm:spec_decode_num_drafts_total": 3791.0,
"vllm:generation_tokens_total": 10000.0
},
"accepted_per_position": {
"0": 3310.0,
"1": 2898.0
},
"accepted_fraction": 0.8187813241888684,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "e806c3d489768238aaa0c29d2db857e00cbefa0f0aba50d5e411a7344842ee89",
"/home/ai/qwen38-release-retests-20261026/mi50-w8-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "67a912927c33d611ba2f51361c6d0b04ffbbae8ff08a9f92a137738da181cb21"
}
}
},
"retest_MTP_2K_accepted": 1077.0,
"retest_MTP_2K_proposed": 1844.0,
"retest_MTP_2K_draft_batches": 922.0,
"retest_MTP_2K_acceptance_percent": 58.4056399132321,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30519541760,
"1": 30519545856
},
"retest_memory_observation_count": 49,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w8-tp2-mtp2/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 45600,
"retest_memory_log_report": {
"profile": "mi50-w8-tp2-mtp2",
"image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
"container": "4b05f105aed3db4ec411d4497fe439e146cb346b8d83fd5b1527647707fdcef3",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 28.09,
"retest_logged_model_load_seconds": 123.709991,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 45600,
"retest_logged_context_capacity_x": 1.34,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w8-tp2-mtp2/service-final.log",
"source_log_sha256": "553939c869da8d99f5604c5504ab31cb9157600462eed3a586e6c78531ee843f",
"sealed_index_sha256": "0bae66b49db8c96e0a89da91a15f660adb325e42c22238bd46faecbe24194dbe",
"source_lines": {
"model_load": {
"line": 144,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:40:05 [gpu_model_runner.py:4584] Model loading took 28.09 GiB memory and 123.709991 seconds"
},
"explicit_KV_budget": {
"line": 7,
"text": "(APIServer pid=1) INFO 10-26 19:36:24 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 8, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 2}, 'optimization_level': '3'}"
},
"explicit_budget_worker_reports": [
{
"line": 156,
"text": "(Worker pid=217) (Worker_TP1 pid=217) INFO 10-26 19:41:01 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
},
{
"line": 157,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 19:41:01 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
}
],
"grouped_KV_tokens": {
"line": 159,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 19:41:01 [kv_cache_utils.py:1314] GPU KV cache size: 45,600 tokens"
},
"context_capacity": {
"line": 160,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 19:41:01 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.34x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w8-tp2-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.423542022705078,
"sampled_peak_VRAM_GiB": 23.27350616455078
} |
| MI50Current Qwen3.8 | 4-bit RTNQwen/Qwen3.8-27B | TP4MTP off | FAIL3407 tokens | 66.085* | 62.679 | 11.545 / 41.82210K: 11.992 / 171.549Strict: 11.261 / 63.139 | 2.95 / 23.25packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:4faf22e8db7711abff5cd83ee1a7bb8e65ae8041fc5552b4652083b093aba065Runtime image config: sha256:8376baaa8d3491c010ada3c623d856873b60675a433fd512e258d730cb7b6513 Exact compatibility changeReplaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately. | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp4-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-w4-tp4-mtp0-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-7,16-23 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \ --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \ --user root \ --workdir /tmp \ -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \ -e MI50_RTN_TP=4 \ -e HIP_VISIBLE_DEVICES=0,1,2,3 \ -e TRANSFORMERS_OFFLINE=1 \ -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \ -e MI50_RTN_BITS=4 \ -e HF_HUB_OFFLINE=1 \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp0/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp0,dst=/runwork \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp0/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp0/cache/.triton,dst=/root/.triton \ --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:72f25a82297a6d478d4f4fee305b8c7853999edeb0c86d852b8370849f810fe3Historical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp4-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 67.903 · eligible10K 64.254 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3973,
"content_chars": 7161,
"content_has_think_close": false,
"elapsed_seconds": 67.34906083345413,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T19:45:03Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.008497083559632301,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 7161,
"qwen_answer_text_sha256": "734dd5ae16a2c3fa33a02e8e60a6948b7777cfd78b2a6a161ac39b16f47824d3",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 7161,
"qwen_visible_text_sha256": "734dd5ae16a2c3fa33a02e8e60a6948b7777cfd78b2a6a161ac39b16f47824d3",
"raw_has_think_close": false,
"reasoning_chars": 13048,
"text_chars": 20210,
"text_sha256": "23747931019b8d463b4463290119d973395421ba6b542dbe1079699ef06d0062",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 59.143225917592645,
"vllm_decode_tps": 67.1759096390141,
"vllm_generation_tokens_delta": 3973.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 8.197571682743728,
"vllm_prefill_tps": 57.70001389505225,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 58.99117152984116
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 37.77216265629977,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T19:45:41Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.005713380873203278,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9486,
"text_chars": 9486,
"text_sha256": "62d22931eb970f7b66da3c3cb82ac07ae6bc058eb360faff682ef9ccb1d67817",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 29.453894706442952,
"vllm_decode_tps": 67.90273476337599,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 8.310188611969352,
"vllm_prefill_tps": 56.91808237886772,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 52.949046582230395
},
"10K": {
"completion_tokens": 10000,
"content_chars": 6738,
"content_has_think_close": false,
"elapsed_seconds": 164.05896650161594,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T19:48:26Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008259626105427742,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 6738,
"qwen_answer_text_sha256": "ceea9ed46978d39a3f977aea6b797cdfb9234d9d38cdbb94405b5f282eb6536d",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 6738,
"qwen_visible_text_sha256": "ceea9ed46978d39a3f977aea6b797cdfb9234d9d38cdbb94405b5f282eb6536d",
"raw_has_think_close": false,
"reasoning_chars": 33979,
"text_chars": 40718,
"text_sha256": "560fb1ae582b985f835b761352697d70b5f65cdb19bf1a40c75d0c44407bb112",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 155.63261205423623,
"vllm_decode_tps": 64.25388527511902,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 8.417382342740893,
"vllm_prefill_tps": 56.19324164452536,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 60.95369374341087
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS3,098 natural tokens2K 66.39710K 62.979 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T16:06:48.027490738Z",
"FinishedAt": "2026-10-26T16:29:20.154019642Z"
},
"devices_after": [
{
"card": 1,
"render": 129,
"pci": "0000:06:00.0",
"actual_card": "card0",
"busy_percent": 0,
"vram_bytes": 9762500608,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:0a:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 10040770560,
"dpm": "auto"
},
{
"card": 3,
"render": 131,
"pci": "0000:0d:00.0",
"actual_card": "card3",
"busy_percent": 0,
"vram_bytes": 10040770560,
"dpm": "auto"
},
{
"card": 4,
"render": 132,
"pci": "0000:10:00.0",
"actual_card": "card4",
"busy_percent": 0,
"vram_bytes": 9990438912,
"dpm": "auto"
}
],
"finished": "2026-10-26T16:29:20.298170+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 66.39721642663191,
"retest_10K_TPS": 62.979410578946336,
"retest_strict_tokens": 3098,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 3098,
"elapsed_seconds": 62.61535412305966,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 46.883774308953434,
"vllm_decode_tps": 66.07829778346945,
"vllm_generation_tokens_delta": 3098.0,
"vllm_prefill_seconds_delta": 15.720751523040235,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 47.03673107700888,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 30.12174466997385,
"vllm_decode_tps": 66.39721642663191,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 16.904422716936097,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 175.82567254197784,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 158.78205127792899,
"vllm_decode_tps": 62.979410578946336,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 17.030237389029935,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 16.904422716936097,
"retest_2K_total_seconds": 47.03673107700888,
"retest_10K_prefill_seconds": 17.030237389029935,
"retest_10K_total_seconds": 175.82567254197784,
"retest_strict_prefill_seconds": 15.720751523040235,
"retest_strict_total_seconds": 62.61535412305966,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": "Complete record: chart-details/row-08-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 24943259648,
"1": 24960045056,
"2": 24959987712,
"3": 24909742080
},
"retest_memory_observation_count": 3,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w4-tp4-mtp0/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 13.16,
"retest_logged_model_load_seconds": 79.138209,
"retest_logged_available_KV_GiB": 16.01,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 16.01,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 261856,
"retest_logged_context_capacity_x": 7.82,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-w4-tp4-mtp0",
"image": "sha256:72f25a82297a6d478d4f4fee305b8c7853999edeb0c86d852b8370849f810fe3",
"container": "562cda56103fb421e6adf66cd6726f7ba830c1f89fbcd0e042644e91a1d8e6e8",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 13.16,
"retest_logged_model_load_seconds": 79.138209,
"retest_logged_available_KV_GiB": 16.01,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 16.01,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 261856,
"retest_logged_context_capacity_x": 7.82,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w4-tp4-mtp0/service-final.log",
"source_log_sha256": "82234bdbf1435b75c9834c4de1afa8954c2357b23180cc78f3c19bc1ab470fab",
"sealed_index_sha256": "ce64d445ea0f358d6dcc1f3f3a09f310416abdf4782a992701442ea1c778834b",
"source_lines": {
"model_load": {
"line": 144,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:10:29 [gpu_model_runner.py:4584] Model loading took 13.16 GiB memory and 79.138209 seconds"
},
"automatic_KV_budget": {
"line": 167,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:11:47 [gpu_worker.py:453] Available KV cache memory: 16.01 GiB"
},
"grouped_KV_tokens": {
"line": 171,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:11:47 [kv_cache_utils.py:1314] GPU KV cache size: 261,856 tokens"
},
"context_capacity": {
"line": 172,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:11:47 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 7.82x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w4-tp4-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 23.245853424072266
}Complete retest loaded admission | Evidence and limitsstrict failed; speeds diagnostic/ineligible {
"strict_tokens": 3407,
"source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w4-tp4-mtpoff",
"source_revision": "e60a9ec31b6bed8a09f6d88972d861f4253127a1",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"sampled_VRAM_after_ladder_bytes_by_rank": "[24943263744,24960000000,24960081920,24909647872]",
"comparison_limit": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
"recorded_configuration_details": {
"cell": "w4-tp4-mtpoff",
"bits": 4,
"group_size": 128,
"TP": 4,
"native_MTP": false,
"num_speculative_tokens": 0,
"strict_valid": false,
"official_2k_eligible": false,
"strict_returned_tokens": 3407,
"strict_decode_TPS": 65.68917102453123,
"strict_prefill_seconds": 11.260670032002963,
"strict_total_seconds": 63.138696900976356,
"fixed_2000_returned_tokens": 2000,
"fixed_2000_decode_TPS": 66.08481149139995,
"fixed_2000_prefill_seconds": 11.544884058996104,
"fixed_2000_total_seconds": 41.821840891003376,
"fixed_10000_returned_tokens": 10000,
"fixed_10000_decode_TPS": 62.67869554052399,
"fixed_10000_prefill_seconds": 11.992151816986734,
"fixed_10000_total_seconds": 171.5490621229983,
"packed_weight_bytes_per_rank": 3162718720,
"packed_weight_bytes_total": 12650874880,
"packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
"render_nodes_by_rank": [
"renderD129",
"renderD130",
"renderD131",
"renderD132"
],
"sampled_VRAM_before_ladder_bytes_by_rank": [
24943263744,
24960000000,
24960081920,
24909647872
],
"sampled_VRAM_after_ladder_bytes_by_rank": [
24943263744,
24960000000,
24960081920,
24909647872
],
"allocated_after_load_bytes_by_rank": [
4590513664,
4590513664,
4590513664,
4590513664
],
"allocated_after_cache_bytes_by_rank": [
21839094272,
21839094272,
21839094272,
21839094272
],
"reserved_after_cache_bytes_by_rank": [
23108517888,
23108517888,
23108517888,
23108517888
],
"sampled_attention_KV_first_layer_views_by_rank": [
{
"rank": 0,
"views": [
{
"shape": [
1337,
2,
784,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 1073364992
}
]
},
{
"rank": 1,
"views": [
{
"shape": [
1337,
2,
784,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 1073364992
}
]
},
{
"rank": 2,
"views": [
{
"shape": [
1337,
2,
784,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 1073364992
}
]
},
{
"rank": 3,
"views": [
{
"shape": [
1337,
2,
784,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 1073364992
}
]
}
],
"KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
"KV_budget": "automatic; actual allocation receipts retained",
"context": 131072,
"greedy": true,
"concurrency": 1,
"activation_dtype": "FP16",
"attention_KV_dtype": "FP16",
"SSM_dtype": "float32",
"native_MTP_head_dtype": "not used",
"native_MTP_unshared_parameter_bytes_by_rank": [],
"model_repository": "Qwen/Qwen3.8-27B",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"serving_source": "e60a9ec31b6bed8a09f6d88972d861f4253127a1",
"image": "sha256:72f25a82297a6d478d4f4fee305b8c7853999edeb0c86d852b8370849f810fe3",
"native_MTP_model_source_sha256": "not used",
"MTP_request_accounting_by_tier": {},
"MTP_accounting_scope": "not used",
"runtime_difference": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
"MTP_scheduler_rollback_rejection_count": "not used",
"host_exclusivity": "unproven",
"serving_CPUs": "0-7,16-23",
"cache_placement": "fresh private admitted rootFS",
"compile_cache_disabled": true,
"official_2k_samples": 1,
"repeatability_established": false,
"FP16_goal_qualifying_ladders": 0,
"evidence": "../../tuning/qwen38-rtn-mi50/w4-tp4-result-20261022.json"
},
"detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
"detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
"detail_source_pointer": "w4-tp4-mtpoff",
"retest_strict_tokens": 3098,
"retest_MTP_accounting_by_tier": {},
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 24943259648,
"1": 24960045056,
"2": 24959987712,
"3": 24909742080
},
"retest_memory_observation_count": 3,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w4-tp4-mtp0/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 16.01,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 16.01,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 261856,
"retest_memory_log_report": {
"profile": "mi50-w4-tp4-mtp0",
"image": "sha256:72f25a82297a6d478d4f4fee305b8c7853999edeb0c86d852b8370849f810fe3",
"container": "562cda56103fb421e6adf66cd6726f7ba830c1f89fbcd0e042644e91a1d8e6e8",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 13.16,
"retest_logged_model_load_seconds": 79.138209,
"retest_logged_available_KV_GiB": 16.01,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 16.01,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 261856,
"retest_logged_context_capacity_x": 7.82,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w4-tp4-mtp0/service-final.log",
"source_log_sha256": "82234bdbf1435b75c9834c4de1afa8954c2357b23180cc78f3c19bc1ab470fab",
"sealed_index_sha256": "ce64d445ea0f358d6dcc1f3f3a09f310416abdf4782a992701442ea1c778834b",
"source_lines": {
"model_load": {
"line": 144,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:10:29 [gpu_model_runner.py:4584] Model loading took 13.16 GiB memory and 79.138209 seconds"
},
"automatic_KV_budget": {
"line": 167,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:11:47 [gpu_worker.py:453] Available KV cache memory: 16.01 GiB"
},
"grouped_KV_tokens": {
"line": 171,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:11:47 [kv_cache_utils.py:1314] GPU KV cache size: 261,856 tokens"
},
"context_capacity": {
"line": 172,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:11:47 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 7.82x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w4-tp4-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 23.245853424072266,
"sampled_peak_VRAM_GiB": 23.245887756347656
} |
| MI50Current Qwen3.8 | 4-bit RTNQwen/Qwen3.8-27B | TP2MTP off | PASS9449 tokens | 51.181 | 47.579 | 16.011 / 55.10010K: 15.533 / 225.722Strict: 14.604 / 211.881 | 5.89 / 17.89packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:a506e684a1f3142d071857d89f8fee8f17f0a30e92970c76d163c03380004961Runtime image config: sha256:fe0d81ba97b6b8a432db928d19d03b511f15de9e40e5d25be81797d9c3e1694b Exact compatibility changeReplaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately. | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp2-mtp0 GPU_RENDER_NODES=renderD128,renderD129 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-w4-tp2-mtp0-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-7,16-23 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --user root \ --workdir /tmp \ -e HIP_VISIBLE_DEVICES=0,1 \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \ -e MI50_RTN_BITS=4 \ -e MI50_RTN_TP=2 \ -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \ -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \ -e HF_HUB_OFFLINE=1 \ -e TRANSFORMERS_OFFLINE=1 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp0/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp0,dst=/runwork \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp0/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp0/cache/.triton,dst=/root/.triton \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24aHistorical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp2-mtp0 GPU_RENDER_NODES=renderD128,renderD129 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 51.245 · eligible10K 47.628 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3523,
"content_chars": 5608,
"content_has_think_close": false,
"elapsed_seconds": 77.33335476461798,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T20:14:37Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.00762627087533474,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 5608,
"qwen_answer_text_sha256": "8dd65b4456cd761c6f12ec9bc4fff7e98a8f640fc7a21baad81ed563a4a236ab",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 5608,
"qwen_visible_text_sha256": "8dd65b4456cd761c6f12ec9bc4fff7e98a8f640fc7a21baad81ed563a4a236ab",
"raw_has_think_close": false,
"reasoning_chars": 11957,
"text_chars": 17566,
"text_sha256": "a0a97116032c440598376109954cb6d4cc4fbfb0762c8537e312d97fdbfecb05",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 69.49865089077502,
"vllm_decode_tps": 50.69163148989456,
"vllm_generation_tokens_delta": 3523.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 7.825316600501537,
"vllm_prefill_tps": 60.44483873913608,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 45.556021857878385
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 47.061583733186126,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T20:15:24Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.006401632912456989,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9891,
"text_chars": 9891,
"text_sha256": "1f3ca45660e8651054584f47d321fb72ef4d84418ec484185e47892b95bedb23",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 39.02831846382469,
"vllm_decode_tps": 51.24484166167185,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 8.024602881632745,
"vllm_prefill_tps": 58.94372680829284,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 42.49750733717175
},
"10K": {
"completion_tokens": 10000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 217.98573838174343,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T20:19:02Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.005619527772068977,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 18708,
"text_chars": 18708,
"text_sha256": "631eb28bb7d6d51e464222a4b4838025b684054be2c3a363f83eaa830161d1e0",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 209.96061483211815,
"vllm_decode_tps": 47.6279801714044,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 8.016675692982972,
"vllm_prefill_tps": 59.00201256912747,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 45.874560759051526
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS3,227 natural tokens2K 51.08310K 47.512 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T16:31:32.061488539Z",
"FinishedAt": "2026-10-26T16:55:44.011700778Z"
},
"devices_after": [
{
"card": 1,
"render": 129,
"pci": "0000:06:00.0",
"actual_card": "card0",
"busy_percent": 0,
"vram_bytes": 5398183936,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:0a:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 6673534976,
"dpm": "auto"
}
],
"finished": "2026-10-26T16:55:44.148546+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 51.08344546346769,
"retest_10K_TPS": 47.51204361074414,
"retest_strict_tokens": 3227,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 3227,
"elapsed_seconds": 76.7849499079166,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 63.73261856404133,
"vllm_decode_tps": 50.63341304197895,
"vllm_generation_tokens_delta": 3227.0,
"vllm_prefill_seconds_delta": 13.041100402013399,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 54.03698424703907,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 39.151626947917975,
"vllm_decode_tps": 51.08344546346769,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 14.875148234074004,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 224.74859275098424,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 210.47295043605845,
"vllm_decode_tps": 47.51204361074414,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 14.263138746959157,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 14.875148234074004,
"retest_2K_total_seconds": 54.03698424703907,
"retest_10K_prefill_seconds": 14.263138746959157,
"retest_10K_total_seconds": 224.74859275098424,
"retest_strict_prefill_seconds": 13.041100402013399,
"retest_strict_total_seconds": 76.7849499079166,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": "Complete record: chart-details/row-09-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 27801952256,
"1": 27801944064
},
"retest_memory_observation_count": 24,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w4-tp2-mtp0/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 25.57,
"retest_logged_model_load_seconds": 102.064312,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 48608,
"retest_logged_context_capacity_x": 1.46,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-w4-tp2-mtp0",
"image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
"container": "03dcc2c0c19b25cf8daefae1396e31ccfe2cc837186d1972d193e2f122174499",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 25.57,
"retest_logged_model_load_seconds": 102.064312,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 48608,
"retest_logged_context_capacity_x": 1.46,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w4-tp2-mtp0/service-final.log",
"source_log_sha256": "a408ac50b7c0f5e6e9df9755c37e8b000d217b6ce9b28a0465a7d8b9bf3436a3",
"sealed_index_sha256": "9eb60e96cc59b2775938ceb3c9ca2747e620a5c010e06ff616f4da65f7050403",
"source_lines": {
"model_load": {
"line": 118,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:35:01 [gpu_model_runner.py:4584] Model loading took 25.57 GiB memory and 102.064312 seconds"
},
"explicit_KV_budget": {
"line": 7,
"text": "(APIServer pid=1) INFO 10-26 16:32:03 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 4, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'optimization_level': '3'}"
},
"explicit_budget_worker_reports": [
{
"line": 125,
"text": "(Worker pid=197) (Worker_TP1 pid=197) INFO 10-26 16:35:45 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
},
{
"line": 126,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:35:45 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
}
],
"grouped_KV_tokens": {
"line": 127,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:35:45 [kv_cache_utils.py:1314] GPU KV cache size: 48,608 tokens"
},
"context_capacity": {
"line": 128,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:35:45 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.46x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w4-tp2-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 25.89258575439453
}Complete retest loaded admission | Evidence and limitscompleted matrix cell {
"strict_tokens": 9449,
"source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w4-tp2-mtpoff",
"source_revision": "d72edcb54835d75267f4e852d0290847824c584e",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"sampled_VRAM_after_ladder_bytes_by_rank": "[19207479296,19207450624]",
"comparison_limit": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
"recorded_configuration_details": {
"cell": "w4-tp2-mtpoff",
"bits": 4,
"group_size": 128,
"TP": 2,
"native_MTP": false,
"num_speculative_tokens": 0,
"strict_valid": true,
"official_2k_eligible": true,
"strict_returned_tokens": 9449,
"strict_decode_TPS": 47.89995832486492,
"strict_prefill_seconds": 14.604201089998242,
"strict_total_seconds": 211.88102579300175,
"fixed_2000_returned_tokens": 2000,
"fixed_2000_decode_TPS": 51.18094703903334,
"fixed_2000_prefill_seconds": 16.0112418270146,
"fixed_2000_total_seconds": 55.10041077597998,
"fixed_10000_returned_tokens": 10000,
"fixed_10000_decode_TPS": 47.57945031867218,
"fixed_10000_prefill_seconds": 15.533215646981262,
"fixed_10000_total_seconds": 225.72210267998162,
"packed_weight_bytes_per_rank": 6325437440,
"packed_weight_bytes_total": 12650874880,
"packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
"render_nodes_by_rank": [
"renderD129",
"renderD130"
],
"sampled_VRAM_before_ladder_bytes_by_rank": [
19207479296,
19207450624
],
"sampled_VRAM_after_ladder_bytes_by_rank": [
19207479296,
19207450624
],
"allocated_after_load_bytes_by_rank": [
9030266368,
9030266368
],
"allocated_after_cache_bytes_by_rank": [
15490159104,
15490159104
],
"reserved_after_cache_bytes_by_rank": [
16882073600,
16882073600
],
"sampled_attention_KV_first_layer_views_by_rank": [
{
"rank": 0,
"views": [
{
"shape": [
250,
2,
784,
2,
256
],
"dtype": "torch.float16",
"logical_bytes": 401408000
}
]
},
{
"rank": 1,
"views": [
{
"shape": [
250,
2,
784,
2,
256
],
"dtype": "torch.float16",
"logical_bytes": 401408000
}
]
}
],
"KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
"KV_budget": "6442450944 bytes explicit",
"context": 131072,
"greedy": true,
"concurrency": 1,
"activation_dtype": "FP16",
"attention_KV_dtype": "FP16",
"SSM_dtype": "float32",
"native_MTP_head_dtype": "not used",
"native_MTP_unshared_parameter_bytes_by_rank": [],
"model_repository": "Qwen/Qwen3.8-27B",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"serving_source": "d72edcb54835d75267f4e852d0290847824c584e",
"image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
"native_MTP_model_source_sha256": "not used",
"MTP_request_accounting_by_tier": {},
"MTP_accounting_scope": "not used",
"runtime_difference": "No MTP; pinned local RTN runtime. TP2 uses explicit6GiB cache.",
"MTP_scheduler_rollback_rejection_count": "not used",
"host_exclusivity": "unproven",
"serving_CPUs": "0-7,16-23",
"cache_placement": "fresh private admitted rootFS",
"compile_cache_disabled": true,
"official_2k_samples": 1,
"repeatability_established": false,
"FP16_goal_qualifying_ladders": 0,
"evidence": "../../tuning/qwen38-rtn-mi50/w4-tp2-result-20261022.json"
},
"detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
"detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
"detail_source_pointer": "w4-tp2-mtpoff",
"retest_strict_tokens": 3227,
"retest_MTP_accounting_by_tier": {},
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 27801952256,
"1": 27801944064
},
"retest_memory_observation_count": 24,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w4-tp2-mtp0/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 48608,
"retest_memory_log_report": {
"profile": "mi50-w4-tp2-mtp0",
"image": "sha256:1cdbeac5f8956a3068a57f28acd7f9dd2a4aa4de590cb90d8db25606f19db24a",
"container": "03dcc2c0c19b25cf8daefae1396e31ccfe2cc837186d1972d193e2f122174499",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 25.57,
"retest_logged_model_load_seconds": 102.064312,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 48608,
"retest_logged_context_capacity_x": 1.46,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w4-tp2-mtp0/service-final.log",
"source_log_sha256": "a408ac50b7c0f5e6e9df9755c37e8b000d217b6ce9b28a0465a7d8b9bf3436a3",
"sealed_index_sha256": "9eb60e96cc59b2775938ceb3c9ca2747e620a5c010e06ff616f4da65f7050403",
"source_lines": {
"model_load": {
"line": 118,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:35:01 [gpu_model_runner.py:4584] Model loading took 25.57 GiB memory and 102.064312 seconds"
},
"explicit_KV_budget": {
"line": 7,
"text": "(APIServer pid=1) INFO 10-26 16:32:03 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 4, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'optimization_level': '3'}"
},
"explicit_budget_worker_reports": [
{
"line": 125,
"text": "(Worker pid=197) (Worker_TP1 pid=197) INFO 10-26 16:35:45 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
},
{
"line": 126,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:35:45 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
}
],
"grouped_KV_tokens": {
"line": 127,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:35:45 [kv_cache_utils.py:1314] GPU KV cache size: 48,608 tokens"
},
"context_capacity": {
"line": 128,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:35:45 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.46x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w4-tp2-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 25.89258575439453,
"sampled_peak_VRAM_GiB": 17.88835906982422
} |
| MI50Current Qwen3.8 | 4-bit RTNQwen/Qwen3.8-27B | TP4MTP22K accepted 1243 / 1512 (82.21%) | PASS5882 tokens | 63.602 | 39.246 | 22.624 / 54.08110K: 23.475 / 278.288Strict: 23.362 / 152.955 | 2.95 / 22.01packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:1ee2a65021b0a54e0f8784ea5e7487504cb48741de9af1bad226b5427c4fa96bRuntime image config: sha256:b1450af604a4f69af53a3ac83f783d9572fcfafb3705c44d5730d50ba99baaad Exact compatibility changeReplaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately. | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp4-mtp2 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-w4-tp4-mtp2-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-7,16-23 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \ --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \ --user root \ --workdir /tmp \ -e HF_HUB_OFFLINE=1 \ -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \ -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \ -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \ -e MI50_RTN_BITS=4 \ -e MI50_RTN_TP=4 \ -e HIP_VISIBLE_DEVICES=0,1,2,3 \ -e TRANSFORMERS_OFFLINE=1 \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ -e MAX_JOBS=1 \ --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp2/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp2,dst=/runwork \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp2/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp4-mtp2/cache/.triton,dst=/root/.triton \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7Historical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp4-mtp2 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 58.906 · eligible10K 41.409 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 4001,
"content_chars": 6233,
"content_has_think_close": false,
"elapsed_seconds": 83.75218992121518,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T20:45:47Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.008191794157028198,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 6233,
"qwen_answer_text_sha256": "554b14189ce88a7e754ef12db9cccfcc3806d2a4e6aec967f964cbfc932b3064",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 6233,
"qwen_visible_text_sha256": "554b14189ce88a7e754ef12db9cccfcc3806d2a4e6aec967f964cbfc932b3064",
"raw_has_think_close": false,
"reasoning_chars": 13030,
"text_chars": 19264,
"text_sha256": "596f08acb3c9150c6168651db02f471064a7f17a4cc99fa1680722d02507b8a8",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 72.83459853101522,
"vllm_decode_tps": 54.93268420085066,
"vllm_generation_tokens_delta": 4001.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 10.909531355835497,
"vllm_prefill_tps": 43.356582842304476,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 47.77188517415126
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 44.91906757093966,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T20:46:32Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.005639274604618549,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 8944,
"text_chars": 8944,
"text_sha256": "d8ddbdc35e64d91111afb2dcbcc23ad1d82db5e1f23d4c6783b5c3ae59526314",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 33.952197367325425,
"vllm_decode_tps": 58.90634937003341,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 10.95891628973186,
"vllm_prefill_tps": 43.161202028998545,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 44.5245217265796
},
"10K": {
"completion_tokens": 10000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 252.5263803936541,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T20:50:45Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.005935841239988804,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 15159,
"text_chars": 15159,
"text_sha256": "7d0dad85f37cf14d6d03b0b1681550a739e02f5b7a62ace29c039b0137dac6ee",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 241.49360792525113,
"vllm_decode_tps": 41.40896351631499,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 11.024237572215497,
"vllm_prefill_tps": 42.90546143454917,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 39.59982313297869
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestMTP counter replayPASS7,228 natural tokens2K 52.13410K 33.819 · diagnostic2K MTP accepted 1107 / 1786 (61.98%) Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T17:46:35.884279265Z",
"FinishedAt": "2026-10-26T18:15:55.378635525Z"
},
"devices_after": [
{
"card": 1,
"render": 129,
"pci": "0000:06:00.0",
"actual_card": "card0",
"busy_percent": 0,
"vram_bytes": 17836716032,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:0a:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 17957421056,
"dpm": "auto"
},
{
"card": 3,
"render": 131,
"pci": "0000:0d:00.0",
"actual_card": "card3",
"busy_percent": 0,
"vram_bytes": 17957421056,
"dpm": "auto"
},
{
"card": 4,
"render": 132,
"pci": "0000:10:00.0",
"actual_card": "card4",
"busy_percent": 0,
"vram_bytes": 17907089408,
"dpm": "auto"
}
],
"finished": "2026-10-26T18:15:55.526176+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 52.13382312317558,
"retest_10K_TPS": 33.81858503497666,
"retest_strict_tokens": 7228,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 7228,
"elapsed_seconds": 201.84848926996347,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 177.56178416404873,
"vllm_decode_tps": 40.70695749104478,
"vllm_generation_tokens_delta": 7228.0,
"vllm_prefill_seconds_delta": 24.27340279892087,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 59.321681477013044,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 38.36281093896832,
"vllm_decode_tps": 52.13382312317558,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 20.947546551004052,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 318.76875816693064,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 295.69539913209155,
"vllm_decode_tps": 33.81858503497666,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 23.061395875993185,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 20.947546551004052,
"retest_2K_total_seconds": 59.321681477013044,
"retest_10K_prefill_seconds": 23.061395875993185,
"retest_10K_total_seconds": 318.76875816693064,
"retest_strict_prefill_seconds": 24.27340279892087,
"retest_strict_total_seconds": 201.84848926996347,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 4373.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 7228.0,
"vllm:request_generation_tokens_sum": 7228.0,
"vllm:spec_decode_num_accepted_tokens_total": 4373.0,
"vllm:spec_decode_num_drafts_total": 2856.0,
"vllm:spec_decode_num_draft_tokens_total": 5712.0
},
"accepted_per_position": {
"0": 2402.0,
"1": 1971.0
},
"accepted_fraction": 0.7655812324929971,
"committed_emitted_tokens": 7228.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "1411456f9ac2de0cf4abed25becd1de45e5fcb81ad379267079da1f541901438",
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "1f754390fbaad7155236e4c71b33c173c3b158180135b13c88b7fe6ec6a3b504"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 1107.0,
"vllm:spec_decode_num_drafts_total": 893.0,
"vllm:spec_decode_num_draft_tokens_total": 1786.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1107.0
},
"accepted_per_position": {
"0": 649.0,
"1": 458.0
},
"accepted_fraction": 0.6198208286674132,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "f08b3a808c79b818598e5ffcea671b6909a8135f90f44bbe34f705ece68ef125",
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "9e4e97ae4f7e360b146b2d710dff319fd670df1e82436d97242a34bcd33a8b1c"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_accepted_tokens_total": 5990.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 5990.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 8022.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:spec_decode_num_drafts_total": 4011.0
},
"accepted_per_position": {
"0": 3306.0,
"1": 2684.0
},
"accepted_fraction": 0.7466965843929195,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "d96d24c021bd72fd6ce7030da03d9a08a49067fb54eacf17fbcc1063e7042ab0",
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "5eb6bf177d61d9131f899ccbf23344e6b01ad427f1062215d80cef034553981e"
}
}
},
"retest_MTP_2K_accepted": 1107.0,
"retest_MTP_2K_proposed": 1786.0,
"retest_MTP_2K_draft_batches": 893.0,
"retest_MTP_2K_acceptance_percent": 61.982082866741315,
"retest_loaded_admission": "Complete record: chart-details/row-10-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 23613620224,
"1": 23630397440,
"2": 23630405632,
"3": 23580045312
},
"retest_memory_observation_count": 29,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w4-tp4-mtp2/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 14.35,
"retest_logged_model_load_seconds": 55.690794,
"retest_logged_available_KV_GiB": 14.41,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 14.41,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 221600,
"retest_logged_context_capacity_x": 6.42,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-w4-tp4-mtp2",
"image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
"container": "4d74b43dece8d4d95c3aa6b916fe0d831886909ec0266e72762306371a15892c",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 14.35,
"retest_logged_model_load_seconds": 55.690794,
"retest_logged_available_KV_GiB": 14.41,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 14.41,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 221600,
"retest_logged_context_capacity_x": 6.42,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w4-tp4-mtp2/service-final.log",
"source_log_sha256": "e464acca61263b85d9bb1d2e0d4fc1b70c5fe21e6d87bd4397926c2df8ec1b38",
"sealed_index_sha256": "9151a652ad0691e375d482b3671dc605a5eef3a2700cb1f444c23f037089644d",
"source_lines": {
"model_load": {
"line": 178,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 17:49:39 [gpu_model_runner.py:4584] Model loading took 14.35 GiB memory and 55.690794 seconds"
},
"automatic_KV_budget": {
"line": 265,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 17:54:52 [gpu_worker.py:453] Available KV cache memory: 14.41 GiB"
},
"grouped_KV_tokens": {
"line": 269,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 17:54:52 [kv_cache_utils.py:1314] GPU KV cache size: 221,600 tokens"
},
"context_capacity": {
"line": 270,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 17:54:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.42x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w4-tp4-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 22.007530212402344
}Complete retest loaded admission | Evidence and limitscompleted matrix cell {
"strict_tokens": 5882,
"source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w4-tp4-mtp2",
"source_revision": "016143b00c7a08e25ff85eb83dc052be148215ab",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"sampled_VRAM_after_ladder_bytes_by_rank": "[23613669376,23630442496,23630434304,23580102656]",
"comparison_limit": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
"recorded_configuration_details": {
"cell": "w4-tp4-mtp2",
"bits": 4,
"group_size": 128,
"TP": 4,
"native_MTP": true,
"num_speculative_tokens": 2,
"strict_valid": true,
"official_2k_eligible": true,
"strict_returned_tokens": 5882,
"strict_decode_TPS": 45.392808024901534,
"strict_prefill_seconds": 23.36222447999171,
"strict_total_seconds": 152.95514036499662,
"fixed_2000_returned_tokens": 2000,
"fixed_2000_decode_TPS": 63.60194464868311,
"fixed_2000_prefill_seconds": 22.624249940010486,
"fixed_2000_total_seconds": 54.08142457698705,
"fixed_10000_returned_tokens": 10000,
"fixed_10000_decode_TPS": 39.246321043123466,
"fixed_10000_prefill_seconds": 23.475374940986512,
"fixed_10000_total_seconds": 278.28769938999903,
"packed_weight_bytes_per_rank": 3162718720,
"packed_weight_bytes_total": 12650874880,
"packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
"render_nodes_by_rank": [
"renderD129",
"renderD130",
"renderD131",
"renderD132"
],
"sampled_VRAM_before_ladder_bytes_by_rank": [
23613669376,
23630442496,
23630434304,
23580102656
],
"sampled_VRAM_after_ladder_bytes_by_rank": [
23613669376,
23630442496,
23630434304,
23580102656
],
"allocated_after_load_bytes_by_rank": [
4805086208,
4805086208,
4805086208,
4805086208
],
"allocated_after_cache_bytes_by_rank": [
20347904000,
20347904000,
20347904000,
20347904000
],
"reserved_after_cache_bytes_by_rank": [
21648900096,
21648900096,
21648900096,
21648900096
],
"sampled_attention_KV_first_layer_views_by_rank": [
{
"rank": 0,
"views": [
{
"shape": [
1110,
2,
800,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 909312000
}
]
},
{
"rank": 1,
"views": [
{
"shape": [
1110,
2,
800,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 909312000
}
]
},
{
"rank": 2,
"views": [
{
"shape": [
1110,
2,
800,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 909312000
}
]
},
{
"rank": 3,
"views": [
{
"shape": [
1110,
2,
800,
1,
256
],
"dtype": "torch.float16",
"logical_bytes": 909312000
}
]
}
],
"KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
"KV_budget": "automatic; actual allocation receipts retained",
"context": 131072,
"greedy": true,
"concurrency": 1,
"activation_dtype": "FP16",
"attention_KV_dtype": "FP16",
"SSM_dtype": "float32",
"native_MTP_head_dtype": "FP16",
"native_MTP_unshared_parameter_bytes_by_rank": [
212388864,
212388864,
212388864,
212388864
],
"model_repository": "Qwen/Qwen3.8-27B",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"serving_source": "016143b00c7a08e25ff85eb83dc052be148215ab",
"image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
"native_MTP_model_source_sha256": "46f57b9c511c9199aa83421fa7fb167b08d5d80c14377948690cdf67d75453c3",
"MTP_request_accounting_by_tier": {
"c1_128_strict": {
"drafts": 2286.0,
"proposed": 4572.0,
"accepted": 3595.0,
"adjusted_unaccepted": 977.0,
"generation": 5882.0,
"prompt": 473.0,
"histogram_count": 2287.0,
"histogram_sum": 6355.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.7787494534324444,
"generation_per_all_observed_iterations": 2.5719282903366856,
"sum_equals_prompt_plus_generation": true
}
},
"c1_2000": {
"drafts": 756.0,
"proposed": 1512.0,
"accepted": 1243.0,
"adjusted_unaccepted": 269.0,
"generation": 2000.0,
"prompt": 473.0,
"histogram_count": 757.0,
"histogram_sum": 2473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 3.2668428005284014,
"generation_per_all_observed_iterations": 2.642007926023778,
"sum_equals_prompt_plus_generation": true
}
},
"c1_10000": {
"drafts": 3433.0,
"proposed": 6866.0,
"accepted": 6567.0,
"adjusted_unaccepted": 299.0,
"generation": 10000.0,
"prompt": 473.0,
"histogram_count": 3434.0,
"histogram_sum": 10473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 3.049796156086197,
"generation_per_all_observed_iterations": 2.9120559114735003,
"sum_equals_prompt_plus_generation": true
}
}
},
"MTP_accounting_scope": "P-A is adjusted unaccepted, not scheduler rollback rejection; iteration means include prefill, not exact decode-only emissions.",
"runtime_difference": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
"MTP_scheduler_rollback_rejection_count": "unobservable",
"host_exclusivity": "unproven",
"serving_CPUs": "0-7,16-23",
"cache_placement": "fresh private admitted rootFS",
"compile_cache_disabled": true,
"official_2k_samples": 1,
"repeatability_established": false,
"FP16_goal_qualifying_ladders": 0,
"evidence": "../../tuning/qwen38-rtn-mi50/w4-tp4-mtp-result-20261022.json"
},
"MTP_accounting_by_tier": {
"c1_128_strict": {
"drafts": 2286.0,
"proposed": 4572.0,
"accepted": 3595.0,
"adjusted_unaccepted": 977.0,
"generation": 5882.0,
"prompt": 473.0,
"histogram_count": 2287.0,
"histogram_sum": 6355.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.7787494534324444,
"generation_per_all_observed_iterations": 2.5719282903366856,
"sum_equals_prompt_plus_generation": true
}
},
"c1_2000": {
"drafts": 756.0,
"proposed": 1512.0,
"accepted": 1243.0,
"adjusted_unaccepted": 269.0,
"generation": 2000.0,
"prompt": 473.0,
"histogram_count": 757.0,
"histogram_sum": 2473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 3.2668428005284014,
"generation_per_all_observed_iterations": 2.642007926023778,
"sum_equals_prompt_plus_generation": true
}
},
"c1_10000": {
"drafts": 3433.0,
"proposed": 6866.0,
"accepted": 6567.0,
"adjusted_unaccepted": 299.0,
"generation": 10000.0,
"prompt": 473.0,
"histogram_count": 3434.0,
"histogram_sum": 10473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 3.049796156086197,
"generation_per_all_observed_iterations": 2.9120559114735003,
"sum_equals_prompt_plus_generation": true
}
}
},
"MTP_2K_draft_batches": 756.0,
"MTP_2K_proposed": 1512.0,
"MTP_2K_accepted": 1243.0,
"MTP_2K_acceptance_percent": 82.20899470899471,
"MTP_2K_committed": 2000.0,
"detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
"detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
"detail_source_pointer": "w4-tp4-mtp2",
"retest_strict_tokens": 7228,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 4373.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 7228.0,
"vllm:request_generation_tokens_sum": 7228.0,
"vllm:spec_decode_num_accepted_tokens_total": 4373.0,
"vllm:spec_decode_num_drafts_total": 2856.0,
"vllm:spec_decode_num_draft_tokens_total": 5712.0
},
"accepted_per_position": {
"0": 2402.0,
"1": 1971.0
},
"accepted_fraction": 0.7655812324929971,
"committed_emitted_tokens": 7228.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "1411456f9ac2de0cf4abed25becd1de45e5fcb81ad379267079da1f541901438",
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "1f754390fbaad7155236e4c71b33c173c3b158180135b13c88b7fe6ec6a3b504"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 1107.0,
"vllm:spec_decode_num_drafts_total": 893.0,
"vllm:spec_decode_num_draft_tokens_total": 1786.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1107.0
},
"accepted_per_position": {
"0": 649.0,
"1": 458.0
},
"accepted_fraction": 0.6198208286674132,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "f08b3a808c79b818598e5ffcea671b6909a8135f90f44bbe34f705ece68ef125",
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "9e4e97ae4f7e360b146b2d710dff319fd670df1e82436d97242a34bcd33a8b1c"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_accepted_tokens_total": 5990.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 5990.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 8022.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:spec_decode_num_drafts_total": 4011.0
},
"accepted_per_position": {
"0": 3306.0,
"1": 2684.0
},
"accepted_fraction": 0.7466965843929195,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "d96d24c021bd72fd6ce7030da03d9a08a49067fb54eacf17fbcc1063e7042ab0",
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp4-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "5eb6bf177d61d9131f899ccbf23344e6b01ad427f1062215d80cef034553981e"
}
}
},
"retest_MTP_2K_accepted": 1107.0,
"retest_MTP_2K_proposed": 1786.0,
"retest_MTP_2K_draft_batches": 893.0,
"retest_MTP_2K_acceptance_percent": 61.982082866741315,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 23613620224,
"1": 23630397440,
"2": 23630405632,
"3": 23580045312
},
"retest_memory_observation_count": 29,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w4-tp4-mtp2/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 14.41,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 14.41,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 221600,
"retest_memory_log_report": {
"profile": "mi50-w4-tp4-mtp2",
"image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
"container": "4d74b43dece8d4d95c3aa6b916fe0d831886909ec0266e72762306371a15892c",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 14.35,
"retest_logged_model_load_seconds": 55.690794,
"retest_logged_available_KV_GiB": 14.41,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 14.41,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 221600,
"retest_logged_context_capacity_x": 6.42,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w4-tp4-mtp2/service-final.log",
"source_log_sha256": "e464acca61263b85d9bb1d2e0d4fc1b70c5fe21e6d87bd4397926c2df8ec1b38",
"sealed_index_sha256": "9151a652ad0691e375d482b3671dc605a5eef3a2700cb1f444c23f037089644d",
"source_lines": {
"model_load": {
"line": 178,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 17:49:39 [gpu_model_runner.py:4584] Model loading took 14.35 GiB memory and 55.690794 seconds"
},
"automatic_KV_budget": {
"line": 265,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 17:54:52 [gpu_worker.py:453] Available KV cache memory: 14.41 GiB"
},
"grouped_KV_tokens": {
"line": 269,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 17:54:52 [kv_cache_utils.py:1314] GPU KV cache size: 221,600 tokens"
},
"context_capacity": {
"line": 270,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 17:54:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.42x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w4-tp4-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 22.007530212402344,
"sampled_peak_VRAM_GiB": 22.007564544677734
} |
| MI50Current Qwen3.8 | 4-bit RTNQwen/Qwen3.8-27B | TP2MTP22K accepted 1092 / 1816 (60.13%) | PASS13252 tokens | 46.966 | 32.753 | 15.057 / 57.65310K: 15.036 / 320.362Strict: 15.042 / 472.320 | 5.89 / 18.28packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:1ee2a65021b0a54e0f8784ea5e7487504cb48741de9af1bad226b5427c4fa96bRuntime image config: sha256:b1450af604a4f69af53a3ac83f783d9572fcfafb3705c44d5730d50ba99baaad Exact compatibility changeReplaced lab-specific PCI address allowlist with distinct visible-device validation. GPU ELF, numerical source, grid, precision, peer and timeout contracts unchanged. The recorded benchmark image and values remain the historical parent; end-to-end compatibility validation is reported separately. | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp2-mtp2 GPU_RENDER_NODES=renderD128,renderD129 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-w4-tp2-mtp2-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-7,16-23 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --user root \ --workdir /tmp \ -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \ -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \ -e MI50_RTN_BITS=4 \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e HF_HUB_OFFLINE=1 \ -e TRANSFORMERS_OFFLINE=1 \ -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \ -e MI50_RTN_TP=2 \ -e HIP_VISIBLE_DEVICES=0,1 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ -e MAX_JOBS=1 \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp2/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp2/cache/.triton,dst=/root/.triton \ --mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp2/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp2,dst=/runwork \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp2-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7Historical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-mi50-r3/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-mi50-r3.tar.gz.sha256 mkdir -p launch-tools-mi50-r3 tar -xzf launch-tools-mi50-r3.tar.gz -C launch-tools-mi50-r3 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp2-mtp2 GPU_RENDER_NODES=renderD128,renderD129 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-mi50-r3/portable/launch.py --config mi50-w4-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 49.476 · eligible10K 38.086 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 9027,
"content_chars": 6853,
"content_has_think_close": false,
"elapsed_seconds": 263.3950856635347,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T21:22:05Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.008886664174497128,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 6853,
"qwen_answer_text_sha256": "7b8ffe2141548b492b29edb8f8d019c0adb81ca7b4dc50bcdf65fd6064a2378f",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 6853,
"qwen_visible_text_sha256": "7b8ffe2141548b492b29edb8f8d019c0adb81ca7b4dc50bcdf65fd6064a2378f",
"raw_has_think_close": false,
"reasoning_chars": 33708,
"text_chars": 40562,
"text_sha256": "4c56847c8958e8315b5f613ed3fe5cd07c7e1a6f515ca50e3e6120e539871a60",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 252.45595039613545,
"vllm_decode_tps": 35.75673295018593,
"vllm_generation_tokens_delta": 9027.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 10.92989128921181,
"vllm_prefill_tps": 43.27581926335057,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 34.27171003308407
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 51.36650285869837,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T21:22:57Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.00833950936794281,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 8988,
"text_chars": 8988,
"text_sha256": "4f354560f326ce7fa34f1b9058c95277a45fee0768e60c45fdbb2871fad810ce",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 40.42383814975619,
"vllm_decode_tps": 49.47575716562834,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 10.934961513616145,
"vllm_prefill_tps": 43.25575352149374,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 38.93588016886615
},
"10K": {
"completion_tokens": 10000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 273.5147092798725,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T21:27:30Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.007087661884725094,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 14636,
"text_chars": 14636,
"text_sha256": "81403098ce625505c2198d17f999d526fbd010f3a8de362f435c4e4952fd0add",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 262.5658295126632,
"vllm_decode_tps": 38.08568700108677,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 10.941470579244196,
"vllm_prefill_tps": 43.230020733892374,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 36.56110498162478
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestMTP counter replayPASS14,111 natural tokens2K 47.78710K 37.319 · diagnostic2K MTP accepted 1109 / 1784 (62.16%) Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T18:18:15.13601314Z",
"FinishedAt": "2026-10-26T18:56:11.307013625Z"
},
"devices_after": [
{
"card": 1,
"render": 129,
"pci": "0000:06:00.0",
"actual_card": "card0",
"busy_percent": 0,
"vram_bytes": 12862812160,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:0a:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 12777545728,
"dpm": "auto"
}
],
"finished": "2026-10-26T18:56:11.450407+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 47.78708170874797,
"retest_10K_TPS": 37.319231865004355,
"retest_strict_tokens": 14111,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 14111,
"elapsed_seconds": 516.6818805149524,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 501.4409740610281,
"vllm_decode_tps": 28.140899387856194,
"vllm_generation_tokens_delta": 14111.0,
"vllm_prefill_seconds_delta": 15.226414316915907,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 57.152446987107396,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 41.8523150710389,
"vllm_decode_tps": 47.78708170874797,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 15.288337494013831,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 283.10880334395915,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 267.95835552492645,
"vllm_decode_tps": 37.319231865004355,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 15.136290078051388,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 15.288337494013831,
"retest_2K_total_seconds": 57.152446987107396,
"retest_10K_prefill_seconds": 15.136290078051388,
"retest_10K_total_seconds": 283.10880334395915,
"retest_strict_prefill_seconds": 15.226414316915907,
"retest_strict_total_seconds": 516.6818805149524,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_sum": 14111.0,
"vllm:spec_decode_num_accepted_tokens_total": 8690.0,
"vllm:spec_decode_num_draft_tokens_total": 10842.0,
"vllm:spec_decode_num_drafts_total": 5421.0,
"vllm:generation_tokens_total": 14111.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 8690.0
},
"accepted_per_position": {
"0": 4679.0,
"1": 4011.0
},
"accepted_fraction": 0.8015126360450101,
"committed_emitted_tokens": 14111.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "ff1459ea939cf973977e7de0e4877ef1a7f3b7ed5a7381611bb0583fbd95b814",
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "f4f791c56c4a24e726d34177403687315b177e29ec4638a4e3c350792aa44b99"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 1784.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:spec_decode_num_accepted_tokens_total": 1109.0,
"vllm:spec_decode_num_drafts_total": 892.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1109.0
},
"accepted_per_position": {
"0": 660.0,
"1": 449.0
},
"accepted_fraction": 0.6216367713004485,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "6147a60b6fc55ffe8bd73699a6e21cdb051e35aa4381a03802c8ea7cfb9f84b9",
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "ec311d29ffa90c0544ecfc89a16dc2853c2850a5cb19fa86fdb0c4c4b7dacf7b"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 6880.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6561.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_drafts_total": 3440.0,
"vllm:spec_decode_num_accepted_tokens_total": 6561.0
},
"accepted_per_position": {
"0": 3329.0,
"1": 3232.0
},
"accepted_fraction": 0.9536337209302326,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "b7ec2069b7cb5c29289e1dc5bf3d7d6e4d61f7ab90f67de5c7bb74de89fc14ea",
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "4669652226a155c0001a247d6d68276dff4f6419210d5cd06f367b4ddf6bbc4b"
}
}
},
"retest_MTP_2K_accepted": 1109.0,
"retest_MTP_2K_proposed": 1784.0,
"retest_MTP_2K_draft_batches": 892.0,
"retest_MTP_2K_acceptance_percent": 62.163677130044846,
"retest_loaded_admission": "Complete record: chart-details/row-11-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30614081536,
"1": 30599421952
},
"retest_memory_observation_count": 36,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w4-tp2-mtp2/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 27.94,
"retest_logged_model_load_seconds": 175.217768,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 45600,
"retest_logged_context_capacity_x": 1.34,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-w4-tp2-mtp2",
"image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
"container": "e2fbe3010395831aa4c2972d9c94dbc1ccae461c86819aa28afb76f3f9e3ebd6",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 27.94,
"retest_logged_model_load_seconds": 175.217768,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 45600,
"retest_logged_context_capacity_x": 1.34,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w4-tp2-mtp2/service-final.log",
"source_log_sha256": "59412ba4a8cd81636f00030fe647d14f836027e24663bf27f338e3a657c67e2f",
"sealed_index_sha256": "a258b5d8f94c32526dc0aa359b7bde709b0e9177620b565088dad25c4b98c5a7",
"source_lines": {
"model_load": {
"line": 144,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 18:24:01 [gpu_model_runner.py:4584] Model loading took 27.94 GiB memory and 175.217768 seconds"
},
"explicit_KV_budget": {
"line": 7,
"text": "(APIServer pid=1) INFO 10-26 18:18:53 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 4, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 2}, 'optimization_level': '3'}"
},
"explicit_budget_worker_reports": [
{
"line": 157,
"text": "(Worker pid=217) (Worker_TP1 pid=217) INFO 10-26 18:25:14 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
},
{
"line": 158,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 18:25:14 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
}
],
"grouped_KV_tokens": {
"line": 160,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 18:25:14 [kv_cache_utils.py:1314] GPU KV cache size: 45,600 tokens"
},
"context_capacity": {
"line": 161,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 18:25:14 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.34x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w4-tp2-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.511585235595703
}Complete retest loaded admission | Evidence and limitscompleted matrix cell {
"strict_tokens": 13252,
"source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv#cell=w4-tp2-mtp2",
"source_revision": "016143b00c7a08e25ff85eb83dc052be148215ab",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"sampled_VRAM_after_ladder_bytes_by_rank": "[19633012736,19633037312]",
"comparison_limit": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
"recorded_configuration_details": {
"cell": "w4-tp2-mtp2",
"bits": 4,
"group_size": 128,
"TP": 2,
"native_MTP": true,
"num_speculative_tokens": 2,
"strict_valid": true,
"official_2k_eligible": true,
"strict_returned_tokens": 13252,
"strict_decode_TPS": 28.981154832388903,
"strict_prefill_seconds": 15.042166554019786,
"strict_total_seconds": 472.3203631230281,
"fixed_2000_returned_tokens": 2000,
"fixed_2000_decode_TPS": 46.966211495697614,
"fixed_2000_prefill_seconds": 15.056974492967129,
"fixed_2000_total_seconds": 57.652922383043915,
"fixed_10000_returned_tokens": 10000,
"fixed_10000_decode_TPS": 32.75325418710531,
"fixed_10000_prefill_seconds": 15.036388729000464,
"fixed_10000_total_seconds": 320.3623473200132,
"packed_weight_bytes_per_rank": 6325437440,
"packed_weight_bytes_total": 12650874880,
"packed_scope": "RTN packed weights/zero points/scales/index buffers; excludes unquantized embeddings/head and native draft head",
"render_nodes_by_rank": [
"renderD129",
"renderD130"
],
"sampled_VRAM_before_ladder_bytes_by_rank": [
19633012736,
19633037312
],
"sampled_VRAM_after_ladder_bytes_by_rank": [
19633012736,
19633037312
],
"allocated_after_load_bytes_by_rank": [
9456126976,
9456126976
],
"allocated_after_cache_bytes_by_rank": [
15928451072,
15928451072
],
"reserved_after_cache_bytes_by_rank": [
17576230912,
17576230912
],
"sampled_attention_KV_first_layer_views_by_rank": [
{
"rank": 0,
"views": [
{
"shape": [
231,
2,
800,
2,
256
],
"dtype": "torch.float16",
"logical_bytes": 378470400
}
]
},
{
"rank": 1,
"views": [
{
"shape": [
231,
2,
800,
2,
256
],
"dtype": "torch.float16",
"logical_bytes": 378470400
}
]
}
],
"KV_view_scope": "Logical hybrid-cache views may alias/pad common pools; do not sum as independent allocation.",
"KV_budget": "6442450944 bytes explicit",
"context": 131072,
"greedy": true,
"concurrency": 1,
"activation_dtype": "FP16",
"attention_KV_dtype": "FP16",
"SSM_dtype": "float32",
"native_MTP_head_dtype": "FP16",
"native_MTP_unshared_parameter_bytes_by_rank": [
424725504,
424725504
],
"model_repository": "Qwen/Qwen3.8-27B",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"model_path": "/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"serving_source": "016143b00c7a08e25ff85eb83dc052be148215ab",
"image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
"native_MTP_model_source_sha256": "46f57b9c511c9199aa83421fa7fb167b08d5d80c14377948690cdf67d75453c3",
"MTP_request_accounting_by_tier": {
"c1_128_strict": {
"drafts": 5135.0,
"proposed": 10270.0,
"accepted": 8117.0,
"adjusted_unaccepted": 2153.0,
"generation": 13252.0,
"prompt": 473.0,
"histogram_count": 5136.0,
"histogram_sum": 13725.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.6723130841121496,
"generation_per_all_observed_iterations": 2.5802180685358254,
"sum_equals_prompt_plus_generation": true
}
},
"c1_2000": {
"drafts": 908.0,
"proposed": 1816.0,
"accepted": 1092.0,
"adjusted_unaccepted": 724.0,
"generation": 2000.0,
"prompt": 473.0,
"histogram_count": 909.0,
"histogram_sum": 2473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.7205720572057204,
"generation_per_all_observed_iterations": 2.2002200220022003,
"sum_equals_prompt_plus_generation": true
}
},
"c1_10000": {
"drafts": 3808.0,
"proposed": 7616.0,
"accepted": 6191.0,
"adjusted_unaccepted": 1425.0,
"generation": 10000.0,
"prompt": 473.0,
"histogram_count": 3809.0,
"histogram_sum": 10473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.749540561827251,
"generation_per_all_observed_iterations": 2.625360987135731,
"sum_equals_prompt_plus_generation": true
}
}
},
"MTP_accounting_scope": "P-A is adjusted unaccepted, not scheduler rollback rejection; iteration means include prefill, not exact decode-only emissions.",
"runtime_difference": "NativeMTP with generic convolution route; source/runtime differences explicit; no causal MTP-only comparison.",
"MTP_scheduler_rollback_rejection_count": "unobservable",
"host_exclusivity": "unproven",
"serving_CPUs": "0-7,16-23",
"cache_placement": "fresh private admitted rootFS",
"compile_cache_disabled": true,
"official_2k_samples": 1,
"repeatability_established": false,
"FP16_goal_qualifying_ladders": 0,
"evidence": "../../tuning/qwen38-rtn-mi50/w4-tp2-mtp-result-20261023.json"
},
"MTP_accounting_by_tier": {
"c1_128_strict": {
"drafts": 5135.0,
"proposed": 10270.0,
"accepted": 8117.0,
"adjusted_unaccepted": 2153.0,
"generation": 13252.0,
"prompt": 473.0,
"histogram_count": 5136.0,
"histogram_sum": 13725.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.6723130841121496,
"generation_per_all_observed_iterations": 2.5802180685358254,
"sum_equals_prompt_plus_generation": true
}
},
"c1_2000": {
"drafts": 908.0,
"proposed": 1816.0,
"accepted": 1092.0,
"adjusted_unaccepted": 724.0,
"generation": 2000.0,
"prompt": 473.0,
"histogram_count": 909.0,
"histogram_sum": 2473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.7205720572057204,
"generation_per_all_observed_iterations": 2.2002200220022003,
"sum_equals_prompt_plus_generation": true
}
},
"c1_10000": {
"drafts": 3808.0,
"proposed": 7616.0,
"accepted": 6191.0,
"adjusted_unaccepted": 1425.0,
"generation": 10000.0,
"prompt": 473.0,
"histogram_count": 3809.0,
"histogram_sum": 10473.0,
"means": {
"prompt_plus_generation_per_observed_iteration": 2.749540561827251,
"generation_per_all_observed_iterations": 2.625360987135731,
"sum_equals_prompt_plus_generation": true
}
}
},
"MTP_2K_draft_batches": 908.0,
"MTP_2K_proposed": 1816.0,
"MTP_2K_accepted": 1092.0,
"MTP_2K_acceptance_percent": 60.13215859030837,
"MTP_2K_committed": 2000.0,
"detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/quantized-matrix.csv",
"detail_source_sha256": "c97f12e890c2817bf84fdf70bfc074750219143c00cfc2f93f9eedb8b9cc949c",
"detail_source_pointer": "w4-tp2-mtp2",
"retest_strict_tokens": 14111,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_sum": 14111.0,
"vllm:spec_decode_num_accepted_tokens_total": 8690.0,
"vllm:spec_decode_num_draft_tokens_total": 10842.0,
"vllm:spec_decode_num_drafts_total": 5421.0,
"vllm:generation_tokens_total": 14111.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 8690.0
},
"accepted_per_position": {
"0": 4679.0,
"1": 4011.0
},
"accepted_fraction": 0.8015126360450101,
"committed_emitted_tokens": 14111.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-before.txt": "ff1459ea939cf973977e7de0e4877ef1a7f3b7ed5a7381611bb0583fbd95b814",
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_128_strict_proxy/metrics-after.txt": "f4f791c56c4a24e726d34177403687315b177e29ec4638a4e3c350792aa44b99"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 1784.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:spec_decode_num_accepted_tokens_total": 1109.0,
"vllm:spec_decode_num_drafts_total": 892.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1109.0
},
"accepted_per_position": {
"0": 660.0,
"1": 449.0
},
"accepted_fraction": 0.6216367713004485,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-before.txt": "6147a60b6fc55ffe8bd73699a6e21cdb051e35aa4381a03802c8ea7cfb9f84b9",
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_2000_proxy/metrics-after.txt": "ec311d29ffa90c0544ecfc89a16dc2853c2850a5cb19fa86fdb0c4c4b7dacf7b"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 6880.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6561.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_drafts_total": 3440.0,
"vllm:spec_decode_num_accepted_tokens_total": 6561.0
},
"accepted_per_position": {
"0": 3329.0,
"1": 3232.0
},
"accepted_fraction": 0.9536337209302326,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-before.txt": "b7ec2069b7cb5c29289e1dc5bf3d7d6e4d61f7ab90f67de5c7bb74de89fc14ea",
"/home/ai/qwen38-release-retests-20261026/mi50-w4-tp2-mtp2-r2/ladder/c1_10000_proxy/metrics-after.txt": "4669652226a155c0001a247d6d68276dff4f6419210d5cd06f367b4ddf6bbc4b"
}
}
},
"retest_MTP_2K_accepted": 1109.0,
"retest_MTP_2K_proposed": 1784.0,
"retest_MTP_2K_draft_batches": 892.0,
"retest_MTP_2K_acceptance_percent": 62.163677130044846,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30614081536,
"1": 30599421952
},
"retest_memory_observation_count": 36,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w4-tp2-mtp2/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 45600,
"retest_memory_log_report": {
"profile": "mi50-w4-tp2-mtp2",
"image": "sha256:b1a99e7c6968d27022bfb22ad547a38e6be2b8c760db2e708e2c3d695546cbe7",
"container": "e2fbe3010395831aa4c2972d9c94dbc1ccae461c86819aa28afb76f3f9e3ebd6",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 27.94,
"retest_logged_model_load_seconds": 175.217768,
"retest_logged_available_KV_GiB": null,
"retest_explicit_KV_budget_bytes": 6442450944,
"retest_KV_budget_GiB": 6.0,
"retest_KV_budget_scope": "Explicit byte budget; profiling skipped",
"retest_logged_GPU_KV_tokens": 45600,
"retest_logged_context_capacity_x": 1.34,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w4-tp2-mtp2/service-final.log",
"source_log_sha256": "59412ba4a8cd81636f00030fe647d14f836027e24663bf27f338e3a657c67e2f",
"sealed_index_sha256": "a258b5d8f94c32526dc0aa359b7bde709b0e9177620b565088dad25c4b98c5a7",
"source_lines": {
"model_load": {
"line": 144,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 18:24:01 [gpu_model_runner.py:4584] Model loading took 27.94 GiB memory and 175.217768 seconds"
},
"explicit_KV_budget": {
"line": 7,
"text": "(APIServer pid=1) INFO 10-26 18:18:53 [utils.py:228] non-default args: {'model_tag': '/opt/local-models/Qwen3.8-27B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 8001, 'model': '/opt/local-models/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'half', 'max_model_len': 131072, 'quantization': 'vnext_rtn', 'served_model_name': ['Qwen/Qwen3.8-27B'], 'hf_overrides': {'architectures': ['Qwen3_5ForCausalLM'], 'quantization_config': {'quant_method': 'vnext_rtn', 'bits': 4, 'group_size': 128}}, 'generation_config': 'vllm', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'worker_cls': 'vllm.vnext_rtn_worker.Worker', 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 6442450944, 'enable_prefix_caching': False, 'mamba_ssm_cache_dtype': 'float32', 'language_model_only': True, 'max_num_batched_tokens': 4, 'max_num_seqs': 2, 'async_scheduling': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 2}, 'optimization_level': '3'}"
},
"explicit_budget_worker_reports": [
{
"line": 157,
"text": "(Worker pid=217) (Worker_TP1 pid=217) INFO 10-26 18:25:14 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
},
{
"line": 158,
"text": "(Worker pid=216) (Worker_TP0 pid=216) INFO 10-26 18:25:14 [gpu_worker.py:380] Initial free memory 31.71 GiB, reserved 6.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly."
}
],
"grouped_KV_tokens": {
"line": 160,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 18:25:14 [kv_cache_utils.py:1314] GPU KV cache size: 45,600 tokens"
},
"context_capacity": {
"line": 161,
"text": "(EngineCore_DP0 pid=198) INFO 10-26 18:25:14 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 1.34x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w4-tp2-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.511585235595703,
"sampled_peak_VRAM_GiB": 18.284690856933594
} |
| V620Current Qwen3.8 | FP16Qwen/Qwen3.8-27B | TP4MTP off | PASS9102 tokens | 33.451 | 32.283 | 3.692 / 63.49410K: 3.717 / 313.490Strict: — / — | 1.19 / 28.88retained FP16 parameter extents; projection buffers separate / VRAM GiBSeparate packed-buffer extents: 17.509 GiB/rank; includes alternate views, not additive physical storage. | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:7045fec47d4265247bd61a45ec50e73d13d2def440121f047e9aaa4b530e24d4Runtime image config: sha256:71f0e460e6327c9b3c5edcc90ff104e43ac14089883164c8bad20e9b141d1aeb | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256 mkdir -p launch-tools-v620-serve-r5 tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp4-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-fp16-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Preflight cleanup compatibilityAfter its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged. Retained lab Docker command · not portabledocker run -d \
--name serve-v620-fp16-tp4-mtp0-20261028 \
--network host \
--ipc host \
--cpuset-cpus 0-7,16-23 \
--cpuset-mems 0 \
--security-opt label=disable \
--shm-size 67108864 \
--device /dev/kfd:/dev/kfd:rwm \
--device /dev/dri/card5:/dev/dri/card5:rwm \
--device /dev/dri/card6:/dev/dri/card6:rwm \
--device /dev/dri/card7:/dev/dri/card7:rwm \
--device /dev/dri/card8:/dev/dri/card8:rwm \
--device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
--device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
--device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
--device /dev/dri/renderD136:/dev/dri/renderD136:rwm \
--user root \
--workdir /tmp \
-e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
-e ROCM_VERSION=7.14.0 \
-e TORCH_VERSION=2.12.0+rocm7.14.0 \
-e VLLM_TARGET_DEVICE=rocm \
-e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
-e PYTORCH_ROCM_ARCH=gfx1030 \
-e GPU_ARCHS=gfx1030 \
-e AMDGPU_TARGETS=gfx1030 \
-e CMAKE_HIP_ARCHITECTURES=gfx1030 \
-e PYTHONDONTWRITEBYTECODE=1 \
-e PIP_NO_CACHE_DIR=1 \
-e PIP_DISABLE_PIP_VERSION_CHECK=1 \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp0/cache/vllm,dst=/root/.cache/vllm \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp0/cache/torch_extensions,dst=/root/.cache/torch_extensions \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp0/cache/.triton,dst=/root/.triton \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
--mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
--entrypoint /usr/bin/env \
sha256:71f0e460e6327c9b3c5edcc90ff104e43ac14089883164c8bad20e9b141d1aeb \
V756F94_BOUNDARY_MODE=posted \
DO_NOT_TRACK=1 \
GPU_DEVICE_ORDINAL=0,1,2,3 \
GPU_MAX_HW_QUEUES=1 \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
OMP_NUM_THREADS=4 \
TEMP=/tmp/torchinductor_root \
TMPDIR=/tmp/torchinductor_root \
TMP=/tmp/torchinductor_root \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
TRITON_CACHE_DIR=/root/.triton \
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
VLLM_GPU_MEMORY_UTILIZATION=0.95 \
VLLM_LOGGING_LEVEL=INFO \
VLLM_PORT=9002 \
VLLM_RDNA_FORCE_FP16=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
NCCL_P2P_DISABLE=0 \
NCCL_ALGO=Tree \
NCCL_PROTO=LL \
NCCL_MIN_NCHANNELS=4 \
NCCL_MAX_NCHANNELS=4 \
NCCL_NTHREADS=256 \
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
HIP_VISIBLE_DEVICES=0,1,2,3 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
ROCR_VISIBLE_DEVICES=0,1,2,3 \
VLLM_CACHE_ROOT=/root/.cache/vllm \
PYTORCH_ROCM_ARCH=gfx1030 \
PYTHONHASHSEED=0 \
/opt/venv/bin/vllm \
serve \
/opt/local-models/qwen38-27b-hf \
--served-model-name \
Qwen/Qwen3.8-27B \
--enable-auto-tool-choice \
--tool-call-parser \
qwen3_coder \
--dtype \
half \
--host \
0.0.0.0 \
--port \
9002 \
--tensor-parallel-size \
4 \
--max-model-len \
131072 \
--gpu-memory-utilization \
0.95 \
--trust-remote-code \
--generation-config \
vllm \
-O=3 \
--reasoning-parser \
qwen3 \
--async-scheduling \
--hf-overrides \
'{"architectures":["Qwen3_5ForCausalLM"]}' \
--language-model-only \
--max-num-seqs \
2 \
--max-num-batched-tokens \
2048 \
--mamba-ssm-cache-dtype \
float32 \
--no-enable-prefix-caching \
--no-enforce-eager \
--attention-backend \
TRITON_ATTN \
--disable-custom-all-reduce \
--seed \
0 \
--worker-cls \
vllm.vnext_qwen38_worker.WorkerHistorical host paths and device addresses are retained as evidence. Use the serving command above.Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp4-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-r2/portable/launch.py --config v620-fp16-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 33.089 · eligible10K 31.895 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 8403,
"content_chars": 18748,
"content_has_think_close": false,
"elapsed_seconds": 265.0742276192177,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T14:51:04Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.010981950908899307,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 18748,
"qwen_answer_text_sha256": "9442102c75db56b3781c1291bf8560af55e51ce1e430bb25972fec0512ee9988",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 18748,
"qwen_visible_text_sha256": "9442102c75db56b3781c1291bf8560af55e51ce1e430bb25972fec0512ee9988",
"raw_has_think_close": false,
"reasoning_chars": 24559,
"text_chars": 43308,
"text_sha256": "5ae0a353a6f8f0bb582929b4301027c8eea4f03635e43e8c1310d52e1b571406",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 261.28218361292966,
"vllm_decode_tps": 32.16063140550152,
"vllm_generation_tokens_delta": 8403.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 3.763594701187685,
"vllm_prefill_tps": 125.67771972118423,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 31.700554503061724
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 64.21472008898854,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T14:52:08Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.010160903912037611,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 10359,
"text_chars": 10359,
"text_sha256": "ff423fe2afad7ed6d76abe61f2c7d0294d6c883fc7e7549ae7b4039b5aadd9cb",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 60.44229609705508,
"vllm_decode_tps": 33.08941137491707,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 3.7466725350823253,
"vllm_prefill_tps": 126.24535386293289,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 31.145506781442116
},
"10K": {
"completion_tokens": 10000,
"content_chars": 27196,
"content_has_think_close": false,
"elapsed_seconds": 317.32863993081264,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T14:57:26Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.009598379023373127,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 27196,
"qwen_answer_text_sha256": "1d6586c92055be911c8df924fd3b36800958433ffca444602a0e3b8d2fc11a91",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 27196,
"qwen_visible_text_sha256": "1d6586c92055be911c8df924fd3b36800958433ffca444602a0e3b8d2fc11a91",
"raw_has_think_close": false,
"reasoning_chars": 24559,
"text_chars": 51756,
"text_sha256": "4c8b6331b15aaf31132fce62e6e77116c74c3caf46315b9ab0c7b10807c345ad",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 313.5314573459327,
"vllm_decode_tps": 31.894726241030963,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 3.777682553976774,
"vllm_prefill_tps": 125.2090384095593,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 31.513071124561293
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS5,674 natural tokens2K 33.44610K 32.252 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T21:35:28.639248975Z",
"FinishedAt": "2026-10-26T22:21:26.85186675Z"
},
"devices_after": [
{
"card": 5,
"render": 133,
"pci": "0000:16:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:19:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 7,
"render": 135,
"pci": "0000:1c:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 8,
"render": 136,
"pci": "0000:1f:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
}
],
"finished": "2026-10-26T22:21:26.998315+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 33.44616375717536,
"retest_10K_TPS": 32.25177356684413,
"retest_strict_tokens": 5674,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 5674,
"elapsed_seconds": 176.00342907500453,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 172.4922093800269,
"vllm_decode_tps": 32.89423922618617,
"vllm_generation_tokens_delta": 5674.0,
"vllm_prefill_seconds_delta": 3.498497008928098,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 63.30487280106172,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 59.79759037599433,
"vllm_decode_tps": 33.44616375717536,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 3.495661307941191,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 313.58174517401494,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 310.06046781502664,
"vllm_decode_tps": 32.25177356684413,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 3.50971574394498,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 3.495661307941191,
"retest_2K_total_seconds": 63.30487280106172,
"retest_10K_prefill_seconds": 3.50971574394498,
"retest_10K_total_seconds": 313.58174517401494,
"retest_strict_prefill_seconds": 3.498497008928098,
"retest_strict_total_seconds": 176.00342907500453,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": null,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30992297984,
"1": 31004942336,
"2": 31005011968,
"3": 30950563840
},
"retest_memory_observation_count": 44,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-fp16-tp4-mtp0/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 19.34,
"retest_logged_model_load_seconds": 102.409229,
"retest_logged_available_KV_GiB": 7.92,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 7.92,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 129360,
"retest_logged_context_capacity_x": 3.87,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "v620-fp16-tp4-mtp0",
"image": "sha256:71f0e460e6327c9b3c5edcc90ff104e43ac14089883164c8bad20e9b141d1aeb",
"container": "7d983ace8802d4bb9a7baa53f5380cf5063ccfa287413bae5200b7410067fd7b",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 19.34,
"retest_logged_model_load_seconds": 102.409229,
"retest_logged_available_KV_GiB": 7.92,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 7.92,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 129360,
"retest_logged_context_capacity_x": 3.87,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-fp16-tp4-mtp0/service-final.log",
"source_log_sha256": "2105e2e2c71b1f1c486467df44c1e3bb7f21fcf1728f97336882c12f7f9fca2f",
"sealed_index_sha256": "0e7d2dc4efd36ed79cc775fbbcb19e362de1cd83ef6d074fe33e79bd4bcd296a",
"source_lines": {
"model_load": {
"line": 851,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 21:39:20 [gpu_model_runner.py:4579] Model loading took 19.34 GiB memory and 102.409229 seconds"
},
"automatic_KV_budget": {
"line": 938,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 21:43:38 [gpu_worker.py:453] Available KV cache memory: 7.92 GiB"
},
"grouped_KV_tokens": {
"line": 943,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 21:43:39 [kv_cache_utils.py:1314] GPU KV cache size: 129,360 tokens"
},
"context_capacity": {
"line": 944,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 21:43:39 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 3.87x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-fp16-tp4-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.875667572021484
} | Evidence and limitsselected FP16 reference {
"strict_tokens": 9102,
"source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-control-result.json#$.measurements",
"source_revision": "8ae761a65953ca037494e287f20e03b680503fb4",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"latency_sources": "[{\"path\": \"/home/joe/PhoneSync/Documents/gfx1030/.work/qwen38-migration/retained-control-r0/evidence/run/ladder1/c1_2000_proxy/summary.json\", \"sha256\": \"a32602f1ab06bb87ac7957e783b6bfbaff4deb08c29dca645847ad7cb21241fa\"}, {\"path\": \"/home/joe/PhoneSync/Documents/gfx1030/.work/qwen38-migration/retained-control-r0/evidence/run/ladder1/c1_10000_proxy/summary.json\", \"sha256\": \"2d98720ac31ea42ada1d2b70bd6f4dc25dbde1e9164a62658664d2c0122a5032\"}]",
"sampled_peak_VRAM_bytes_by_PCI": {
"0000:16:00.0": 30992314368,
"0000:19:00.0": 31004827648,
"0000:1c:00.0": 31004913664,
"0000:1f:00.0": 30950211584
},
"memory_accounting_limit": "All retained integer-buffer logical extents include alternate representations and are not unique physical allocations or measured memory traffic. VRAM is the maximum of retained startup samples.",
"memory_source": "/home/joe/PhoneSync/Documents/gfx1030/.work/qwen38-migration/retained-control-r0/evidence.tar.gz",
"memory_source_sha256": "10368e0004f8558f53363a614f289d6e6695e5c26daa2def9a008e2dcb0463ca",
"retest_strict_tokens": 5674,
"retest_MTP_accounting_by_tier": {},
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30992297984,
"1": 31004942336,
"2": 31005011968,
"3": 30950563840
},
"retest_memory_observation_count": 44,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-fp16-tp4-mtp0/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 7.92,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 7.92,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 129360,
"retest_memory_log_report": {
"profile": "v620-fp16-tp4-mtp0",
"image": "sha256:71f0e460e6327c9b3c5edcc90ff104e43ac14089883164c8bad20e9b141d1aeb",
"container": "7d983ace8802d4bb9a7baa53f5380cf5063ccfa287413bae5200b7410067fd7b",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 19.34,
"retest_logged_model_load_seconds": 102.409229,
"retest_logged_available_KV_GiB": 7.92,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 7.92,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 129360,
"retest_logged_context_capacity_x": 3.87,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-fp16-tp4-mtp0/service-final.log",
"source_log_sha256": "2105e2e2c71b1f1c486467df44c1e3bb7f21fcf1728f97336882c12f7f9fca2f",
"sealed_index_sha256": "0e7d2dc4efd36ed79cc775fbbcb19e362de1cd83ef6d074fe33e79bd4bcd296a",
"source_lines": {
"model_load": {
"line": 851,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 21:39:20 [gpu_model_runner.py:4579] Model loading took 19.34 GiB memory and 102.409229 seconds"
},
"automatic_KV_budget": {
"line": 938,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 21:43:38 [gpu_worker.py:453] Available KV cache memory: 7.92 GiB"
},
"grouped_KV_tokens": {
"line": 943,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 21:43:39 [kv_cache_utils.py:1314] GPU KV cache size: 129,360 tokens"
},
"context_capacity": {
"line": 944,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 21:43:39 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 3.87x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-fp16-tp4-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.875667572021484,
"sampled_peak_VRAM_GiB": 28.87557601928711
} |
| V620Current Qwen3.8 | FP16Qwen/Qwen3.8-27B | TP2MTP off | PASS8355 tokens | 13.068 | 12.694 | 0.798 / 153.85910K: 0.799 / 788.600Strict: 0.792 / 655.029 | 25.17 / 28.32FP16 parameter extents / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:afc254a3459d27e4d4fb5b350cbd8c081074da3b696305a7cc8ef7e81bfe476bRuntime image config: sha256:83a139552fe2e33ba48844e32e69b83ac6d10c592cd0547dfdd1643eaad2408f | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256 mkdir -p launch-tools-v620-serve-r5 tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp2-mtp0 GPU_RENDER_NODES=renderD128,renderD129 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-fp16-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Preflight cleanup compatibilityAfter its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged. Retained lab Docker command · not portabledocker run -d \
--name serve-v620-fp16-tp2-mtp0-20261028 \
--network host \
--ipc host \
--cpuset-cpus 0-7,16-23 \
--cpuset-mems 0 \
--security-opt label=disable \
--shm-size 67108864 \
--device /dev/kfd:/dev/kfd:rwm \
--device /dev/dri/card5:/dev/dri/card5:rwm \
--device /dev/dri/card6:/dev/dri/card6:rwm \
--device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
--device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
--user root \
--workdir /tmp \
-e VLLM_DISABLE_COMPILE_CACHE=1 \
-e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
-e ROCM_VERSION=7.14.0 \
-e TORCH_VERSION=2.12.0+rocm7.14.0 \
-e VLLM_TARGET_DEVICE=rocm \
-e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
-e PYTORCH_ROCM_ARCH=gfx1030 \
-e GPU_ARCHS=gfx1030 \
-e AMDGPU_TARGETS=gfx1030 \
-e CMAKE_HIP_ARCHITECTURES=gfx1030 \
-e PYTHONDONTWRITEBYTECODE=1 \
-e PIP_NO_CACHE_DIR=1 \
-e PIP_DISABLE_PIP_VERSION_CHECK=1 \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp0/cache/vllm,dst=/root/.cache/vllm \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp0/cache/torch_extensions,dst=/root/.cache/torch_extensions \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp0/cache/.triton,dst=/root/.triton \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
--mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
--entrypoint /usr/bin/env \
sha256:83a139552fe2e33ba48844e32e69b83ac6d10c592cd0547dfdd1643eaad2408f \
V756F94_BOUNDARY_MODE=posted \
DO_NOT_TRACK=1 \
GPU_DEVICE_ORDINAL=0,1 \
GPU_MAX_HW_QUEUES=1 \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
OMP_NUM_THREADS=4 \
TEMP=/tmp/torchinductor_root \
TMPDIR=/tmp/torchinductor_root \
TMP=/tmp/torchinductor_root \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
TRITON_CACHE_DIR=/root/.triton \
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
VLLM_GPU_MEMORY_UTILIZATION=0.95 \
VLLM_LOGGING_LEVEL=INFO \
VLLM_PORT=9002 \
VLLM_RDNA_FORCE_FP16=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
NCCL_P2P_DISABLE=0 \
NCCL_ALGO=Tree \
NCCL_PROTO=LL \
NCCL_MIN_NCHANNELS=4 \
NCCL_MAX_NCHANNELS=4 \
NCCL_NTHREADS=256 \
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
HIP_VISIBLE_DEVICES=0,1 \
CUDA_VISIBLE_DEVICES=0,1 \
ROCR_VISIBLE_DEVICES=0,1 \
VLLM_CACHE_ROOT=/root/.cache/vllm \
PYTORCH_ROCM_ARCH=gfx1030 \
PYTHONHASHSEED=0 \
/opt/venv/bin/vllm \
serve \
/opt/local-models/qwen38-27b-hf \
--served-model-name \
Qwen/Qwen3.8-27B \
--enable-auto-tool-choice \
--tool-call-parser \
qwen3_coder \
--dtype \
half \
--host \
0.0.0.0 \
--port \
9002 \
--tensor-parallel-size \
2 \
--max-model-len \
131072 \
--gpu-memory-utilization \
0.95 \
--trust-remote-code \
--generation-config \
vllm \
-O=3 \
--reasoning-parser \
qwen3 \
--async-scheduling \
--hf-overrides \
'{"architectures":["Qwen3_5ForCausalLM"]}' \
--language-model-only \
--max-num-seqs \
2 \
--max-num-batched-tokens \
2048 \
--mamba-ssm-cache-dtype \
float32 \
--no-enable-prefix-caching \
--no-enforce-eager \
--attention-backend \
TRITON_ATTN \
--disable-custom-all-reduce \
--seed \
0 \
--worker-cls \
vllm.vnext_qwen38_worker.WorkerHistorical host paths and device addresses are retained as evidence. Use the serving command above.Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp2-mtp0 GPU_RENDER_NODES=renderD128,renderD129 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-r2/portable/launch.py --config v620-fp16-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 13.092 · eligible10K 12.714 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 5745,
"content_chars": 12238,
"content_has_think_close": false,
"elapsed_seconds": 445.8737674509175,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T16:22:26Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.019420526921749115,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 12238,
"qwen_answer_text_sha256": "3a79027ad7fed9ab4a8ecdf167ff2ee2cbecab9c7326e7bec78ffa1b0c124d5c",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 12238,
"qwen_visible_text_sha256": "3a79027ad7fed9ab4a8ecdf167ff2ee2cbecab9c7326e7bec78ffa1b0c124d5c",
"raw_has_think_close": false,
"reasoning_chars": 17602,
"text_chars": 29841,
"text_sha256": "b0bde25f1140d16eebe07f1b6aa3ec0c00c2d286d99f4a858096c553b6213840",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 445.05560585297644,
"vllm_decode_tps": 12.908499352545743,
"vllm_generation_tokens_delta": 5745.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.7985437789466232,
"vllm_prefill_tps": 592.3282009959989,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 12.884812741607227
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 153.58387053897604,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T16:24:59Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.017818843945860863,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 10439,
"text_chars": 10439,
"text_sha256": "ac4d2fe9d9291b42059c546ee942aa7a4e1e96c2f2aa522b5db331457e9aa1ce",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 152.76036863587797,
"vllm_decode_tps": 13.092400979780507,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8026171950623393,
"vllm_prefill_tps": 589.3220365946211,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 13.022200788281646
},
"10K": {
"completion_tokens": 10000,
"content_chars": 35044,
"content_has_think_close": false,
"elapsed_seconds": 787.3251471030526,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T16:38:07Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.014115805039182305,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 35044,
"qwen_answer_text_sha256": "02b0dc02851bd7baf8e137f4e7b9e4c308948ffc73da1c1da4d5a61edd167ead",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 35044,
"qwen_visible_text_sha256": "02b0dc02851bd7baf8e137f4e7b9e4c308948ffc73da1c1da4d5a61edd167ead",
"raw_has_think_close": false,
"reasoning_chars": 17602,
"text_chars": 52647,
"text_sha256": "fb7a87c2a39821a2b73c9834aa37be8ce25942b70a75dfa28ce7dcaf0fc12673",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 786.5056542060338,
"vllm_decode_tps": 12.714466763871973,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.7967424760572612,
"vllm_prefill_tps": 593.6673570369629,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 12.701232822036491
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS11,009 natural tokens2K 13.12510K 12.752 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T22:23:38.017976016Z",
"FinishedAt": "2026-10-26T23:40:00.366118153Z"
},
"devices_after": [
{
"card": 5,
"render": 133,
"pci": "0000:16:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:19:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
}
],
"finished": "2026-10-26T23:40:00.522295+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 13.125058049552255,
"retest_10K_TPS": 12.75176962944575,
"retest_strict_tokens": 11009,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 11009,
"elapsed_seconds": 867.0248347370652,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 866.2204551490722,
"vllm_decode_tps": 12.709235777750603,
"vllm_generation_tokens_delta": 11009.0,
"vllm_prefill_seconds_delta": 0.7914496649755165,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 153.18691095500253,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 152.3802784299478,
"vllm_decode_tps": 13.125058049552255,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 0.7956740560475737,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 785.0079319690121,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 784.2048821920762,
"vllm_decode_tps": 12.75176962944575,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 0.790560997906141,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 0.7956740560475737,
"retest_2K_total_seconds": 153.18691095500253,
"retest_10K_prefill_seconds": 0.790560997906141,
"retest_10K_total_seconds": 785.0079319690121,
"retest_strict_prefill_seconds": 0.7914496649755165,
"retest_strict_total_seconds": 867.0248347370652,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": null,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30409326592,
"1": 30409351168
},
"retest_memory_observation_count": 72,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-fp16-tp2-mtp0/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 25.39,
"retest_logged_model_load_seconds": 15.697194,
"retest_logged_available_KV_GiB": 1.24,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 1.24,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 9408,
"retest_logged_context_capacity_x": 0.3,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "v620-fp16-tp2-mtp0",
"image": "sha256:83a139552fe2e33ba48844e32e69b83ac6d10c592cd0547dfdd1643eaad2408f",
"container": "15339a206d6624e7abc9caab614f44169f6673d584ba35e9b55bb74a045adbff",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 25.39,
"retest_logged_model_load_seconds": 15.697194,
"retest_logged_available_KV_GiB": 1.24,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 1.24,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 9408,
"retest_logged_context_capacity_x": 0.3,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-fp16-tp2-mtp0/service-final.log",
"source_log_sha256": "f341d1fc02158cf2480d0571daa0c0d94ea5331083628a0a5fbf9f801b2669a9",
"sealed_index_sha256": "cf2eb4f5955d369cad6634fb9728c17c9d89e4376c3fad6427c6d195add6bad7",
"source_lines": {
"model_load": {
"line": 91,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 22:25:31 [gpu_model_runner.py:4579] Model loading took 25.39 GiB memory and 15.697194 seconds"
},
"automatic_KV_budget": {
"line": 102,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 22:26:47 [gpu_worker.py:453] Available KV cache memory: 1.24 GiB"
},
"grouped_KV_tokens": {
"line": 104,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 22:26:47 [kv_cache_utils.py:1314] GPU KV cache size: 9,408 tokens"
},
"context_capacity": {
"line": 105,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 22:26:47 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 0.30x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-fp16-tp2-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.32091522216797
} | Evidence and limitsCompleted, independently reviewed FP16 TP2 matrix cell; no FP16 release qualification or promotion. {
"strict_tokens": 8355,
"source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.FP16_W8_TP_MTP_matrix.cells[0].result",
"source_revision": "1f00a8f7e264ad38c8997d33807e2395bd8e0dd4",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"attention_KV_dtype": "FP16",
"sampled_peak_VRAM_bytes_by_PCI": "{\"0000:16:00.0\": 30409351168, \"0000:19:00.0\": 30409719808}",
"KV_capacity": "{\"available_GiB\": 1.24, \"concurrency_at131072\": 0.3, \"tokens\": 9408}",
"attention_KV_logical_view_bytes_per_rank": 1310195712,
"post_KV_allocator_allocated_bytes_per_rank": 28462785536,
"post_KV_allocator_reserved_bytes_per_rank": 28672262144,
"comparison_limit": "TP2 ordinary inherited FP16 decoder, linear/allreduce, finalnorm and LMhead replace TP4 lossless-packed/posted paths; not an isolated TP change. | Fresh rootSSD caches; selected TP4 reference storage placement retained separately. | Runtime reports 9408 GPU KV tokens and0.30x concurrency at configured131072; full-context residency was not demonstrated. | One sample/cell, no repeatability or FP16qualification; official2K below44. | TP2 ordinary FP16 decoder/fallback and dense loaded storage differ materially from selected TP4 packed control; no causal attribution of speed to TP alone. | Logical tensor/KV views may alias. PyTorchallocated/reserved, sampledcardVRAM and runtimeavailableKV are distinct observations; no fullcontext residency proof. | Inherited closure mode saysquantized-complete but actualbaseline_kindFP16_TP2/checks/image identify thisFP16cell; no quantized result relabel. | W8card7 diagnostic overlapped excludedwarmups only, not official samples.",
"recorded_configuration_details": {
"source": "1f00a8f7e264ad38c8997d33807e2395bd8e0dd4",
"image": "sha256:83a139552fe2e33ba48844e32e69b83ac6d10c592cd0547dfdd1643eaad2408f",
"strict_valid": true,
"strict_tokens": 8355,
"strict_diagnostic_TPS": 12.770910773696434,
"official_fixed2K_TPS": 13.067638634805824,
"diagnostic_fixed10K_TPS": 12.693775456156407,
"metrics": {
"strict": {
"completion_tokens": 8355,
"finish_reason": "stop",
"elapsed_seconds": 655.0289586309809,
"vllm_prefill_seconds_delta": 0.7922419030219316,
"vllm_decode_seconds_delta": 654.2211552529479,
"vllm_decode_tps": 12.770910773696434,
"vllm_generation_tokens_delta": 8355.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "f1c67a3a47e860b7bba9f8ff7be2b0266d50b2fff6cb0bb0307c4c8430c39059"
},
"fixed2K": {
"completion_tokens": 2000,
"finish_reason": "length",
"elapsed_seconds": 153.8593644410139,
"vllm_prefill_seconds_delta": 0.7977076760143973,
"vllm_decode_seconds_delta": 153.0498398289783,
"vllm_decode_tps": 13.067638634805824,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "7910f673846d91314be1e19fef6cf2b0e8e8b2dc3b25d877365eaf7de2dbb31b"
},
"fixed10K": {
"completion_tokens": 10000,
"finish_reason": "length",
"elapsed_seconds": 788.5997820449993,
"vllm_prefill_seconds_delta": 0.79924124001991,
"vllm_decode_seconds_delta": 787.7876865349826,
"vllm_decode_tps": 12.693775456156407,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "40e6f56b389ce2d9681ea939d167f6bf89ed490bd10ba821cac09bf4b49c47cc"
}
},
"weight_dtype": "native FP16; no quantization",
"activation_dtype": "FP16",
"attention_KV_dtype": "FP16",
"TP": 2,
"MTP_tokens": 0,
"packed_weight_bytes_per_rank": 0,
"FP16_logical_tensor_bytes_per_rank": 27030891008,
"sampled_peak_VRAM_bytes": {
"0000:16:00.0": 30409351168,
"0000:19:00.0": 30409719808
},
"KV_capacity": {
"available_GiB": 1.24,
"tokens": 9408,
"concurrency_at131072": 0.3
},
"configured_context": 131072,
"complete_ladders": 1,
"sample_count": 1,
"repeatability_established": false,
"FP16_release_qualification": false,
"archive": {
"path": "/home/ai/vnext-v620-qwen38-fp16-tp2-service-r0-20261023/raw-evidence.tar.gz",
"bytes": 381935,
"sha256": "86afe4421e7be303028b88b503517b15399cdde69a8831b27ce4b1582a11e260",
"files": 231,
"index_sha256": "fea34562414c93ead3ef2db5fa65d85b3163d9cb7cd950701fa402b9973efaec"
},
"local_primary_record": ".work/fp16-w8-tp-mtp-matrix/fp16-tp2-service-r0/sealed-mirror/retained/evidence/measurement-closure.json",
"review": {
"path": ".work/fp16-w8-tp-mtp-matrix/reviews/FP16-TP2-final-cell-review.json",
"sha256": "43a550344eb2b3059ef139536e3740ab33e7cf4be39d2cd9770f5669ffd5a7e8"
},
"closure": {
"container_id": "eeec72e25a4ac42331400e4525ee45261cc72371a50606f7f6bd94d6b4640d3c",
"image_id": "sha256:83a139552fe2e33ba48844e32e69b83ac6d10c592cd0547dfdd1643eaad2408f",
"exit_code": 0,
"OOMKilled": false,
"restart_count": 0,
"bank_idle": true,
"devices": [
{
"pci": "0000:16:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
},
{
"pci": "0000:19:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
}
],
"started_at": "2026-10-23T10:34:57.477000767Z",
"finished_at": "2026-10-23T11:48:28.181764449Z",
"closed_at": "2026-10-23T11:48:28.337427+00:00",
"kernel_fault_matches": [],
"startup_requests_completed_and_drained": 3,
"official_phase_samples": 11,
"complete_ladders": 1,
"strict_valid": true,
"official_fixed_2000_tps": 13.067638634805824,
"qualifying_ladders": 0,
"FP16_qualification": false,
"baseline_kind": "FP16_TP2"
},
"final_counters": {
"vllm:prompt_tokens_total": 6622,
"vllm:generation_tokens_total": 42355,
"vllm:request_decode_time_seconds_count": 14,
"vllm:request_prefill_time_seconds_count": 14,
"vllm:num_requests_running": 0,
"vllm:num_requests_waiting": 0
},
"runtime_differences": [
"TP2 ordinary inherited FP16 decoder, linear/allreduce, finalnorm and LMhead replace TP4 lossless-packed/posted paths; not an isolated TP change.",
"Fresh rootSSD caches; selected TP4 reference storage placement retained separately.",
"Runtime reports 9408 GPU KV tokens and0.30x concurrency at configured131072; full-context residency was not demonstrated."
],
"limits": [
"One sample/cell, no repeatability or FP16qualification; official2K below44.",
"TP2 ordinary FP16 decoder/fallback and dense loaded storage differ materially from selected TP4 packed control; no causal attribution of speed to TP alone.",
"Logical tensor/KV views may alias. PyTorchallocated/reserved, sampledcardVRAM and runtimeavailableKV are distinct observations; no fullcontext residency proof.",
"Inherited closure mode saysquantized-complete but actualbaseline_kindFP16_TP2/checks/image identify thisFP16cell; no quantized result relabel.",
"W8card7 diagnostic overlapped excludedwarmups only, not official samples."
]
},
"detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
"detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
"detail_source_pointer": "FP16_W8_TP_MTP_matrix.cells[0]",
"retest_strict_tokens": 11009,
"retest_MTP_accounting_by_tier": {},
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30409326592,
"1": 30409351168
},
"retest_memory_observation_count": 72,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-fp16-tp2-mtp0/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 1.24,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 1.24,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 9408,
"retest_memory_log_report": {
"profile": "v620-fp16-tp2-mtp0",
"image": "sha256:83a139552fe2e33ba48844e32e69b83ac6d10c592cd0547dfdd1643eaad2408f",
"container": "15339a206d6624e7abc9caab614f44169f6673d584ba35e9b55bb74a045adbff",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 25.39,
"retest_logged_model_load_seconds": 15.697194,
"retest_logged_available_KV_GiB": 1.24,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 1.24,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 9408,
"retest_logged_context_capacity_x": 0.3,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-fp16-tp2-mtp0/service-final.log",
"source_log_sha256": "f341d1fc02158cf2480d0571daa0c0d94ea5331083628a0a5fbf9f801b2669a9",
"sealed_index_sha256": "cf2eb4f5955d369cad6634fb9728c17c9d89e4376c3fad6427c6d195add6bad7",
"source_lines": {
"model_load": {
"line": 91,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 22:25:31 [gpu_model_runner.py:4579] Model loading took 25.39 GiB memory and 15.697194 seconds"
},
"automatic_KV_budget": {
"line": 102,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 22:26:47 [gpu_worker.py:453] Available KV cache memory: 1.24 GiB"
},
"grouped_KV_tokens": {
"line": 104,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 22:26:47 [kv_cache_utils.py:1314] GPU KV cache size: 9,408 tokens"
},
"context_capacity": {
"line": 105,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 22:26:47 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 0.30x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-fp16-tp2-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.32091522216797,
"sampled_peak_VRAM_GiB": 28.321258544921875
} |
| V620Current Qwen3.8 | FP16Qwen/Qwen3.8-27B | TP4MTP22K accepted 1266 / 1468 (86.24%) | FAIL4212 tokens | 39.586* | 28.223 | 3.188 / 53.72310K: 3.243 / 357.569Strict: 3.436 / 118.761 | 1.19 / 27.56retained FP16 parameter extents; projection buffers separate / VRAM GiBSeparate packed-buffer extents: 17.509 GiB/rank; includes alternate views, not additive physical storage. | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:ad86e1b5181d4a8b4a45d24bce14b96f691c0927ca1409e4d43c6eda40764e0dRuntime image config: sha256:470486980ed01b78334c8e0b3ae36c9396d0ac9d5dba1183d581a857fe95674d | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256 mkdir -p launch-tools-v620-serve-r5 tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp4-mtp2 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-fp16-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Preflight cleanup compatibilityAfter its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged. Retained lab Docker command · not portabledocker run -d \
--name serve-v620-fp16-tp4-mtp2-20261028 \
--network host \
--ipc host \
--cpuset-cpus 0-7,16-23 \
--cpuset-mems 0 \
--security-opt label=disable \
--shm-size 67108864 \
--device /dev/kfd:/dev/kfd:rwm \
--device /dev/dri/card5:/dev/dri/card5:rwm \
--device /dev/dri/card6:/dev/dri/card6:rwm \
--device /dev/dri/card7:/dev/dri/card7:rwm \
--device /dev/dri/card8:/dev/dri/card8:rwm \
--device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
--device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
--device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
--device /dev/dri/renderD136:/dev/dri/renderD136:rwm \
--user root \
--workdir /tmp \
-e VLLM_DISABLE_COMPILE_CACHE=1 \
-e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
-e ROCM_VERSION=7.14.0 \
-e TORCH_VERSION=2.12.0+rocm7.14.0 \
-e VLLM_TARGET_DEVICE=rocm \
-e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
-e PYTORCH_ROCM_ARCH=gfx1030 \
-e GPU_ARCHS=gfx1030 \
-e AMDGPU_TARGETS=gfx1030 \
-e CMAKE_HIP_ARCHITECTURES=gfx1030 \
-e PYTHONDONTWRITEBYTECODE=1 \
-e PIP_NO_CACHE_DIR=1 \
-e PIP_DISABLE_PIP_VERSION_CHECK=1 \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp2/cache/vllm,dst=/root/.cache/vllm \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp2/cache/torch_extensions,dst=/root/.cache/torch_extensions \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp2/cache/.triton,dst=/root/.triton \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp4-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
--mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
--entrypoint /usr/bin/env \
sha256:470486980ed01b78334c8e0b3ae36c9396d0ac9d5dba1183d581a857fe95674d \
V756F94_BOUNDARY_MODE=posted \
DO_NOT_TRACK=1 \
GPU_DEVICE_ORDINAL=0,1,2,3 \
GPU_MAX_HW_QUEUES=1 \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
OMP_NUM_THREADS=4 \
TEMP=/tmp/torchinductor_root \
TMPDIR=/tmp/torchinductor_root \
TMP=/tmp/torchinductor_root \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
TRITON_CACHE_DIR=/root/.triton \
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
VLLM_GPU_MEMORY_UTILIZATION=0.95 \
VLLM_LOGGING_LEVEL=INFO \
VLLM_PORT=9002 \
VLLM_RDNA_FORCE_FP16=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
NCCL_P2P_DISABLE=0 \
NCCL_ALGO=Tree \
NCCL_PROTO=LL \
NCCL_MIN_NCHANNELS=4 \
NCCL_MAX_NCHANNELS=4 \
NCCL_NTHREADS=256 \
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
HIP_VISIBLE_DEVICES=0,1,2,3 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
ROCR_VISIBLE_DEVICES=0,1,2,3 \
VLLM_CACHE_ROOT=/root/.cache/vllm \
PYTORCH_ROCM_ARCH=gfx1030 \
PYTHONHASHSEED=0 \
/opt/venv/bin/vllm \
serve \
/opt/local-models/qwen38-27b-hf \
--served-model-name \
Qwen/Qwen3.8-27B \
--enable-auto-tool-choice \
--tool-call-parser \
qwen3_coder \
--dtype \
half \
--host \
0.0.0.0 \
--port \
9002 \
--tensor-parallel-size \
4 \
--max-model-len \
131072 \
--gpu-memory-utilization \
0.95 \
--trust-remote-code \
--generation-config \
vllm \
-O=3 \
--reasoning-parser \
qwen3 \
--async-scheduling \
--hf-overrides \
'{"architectures":["Qwen3_5ForCausalLM"]}' \
--language-model-only \
--max-num-seqs \
2 \
--max-num-batched-tokens \
2048 \
--mamba-ssm-cache-dtype \
float32 \
--no-enable-prefix-caching \
--no-enforce-eager \
--attention-backend \
TRITON_ATTN \
--disable-custom-all-reduce \
--seed \
0 \
--worker-cls \
vllm.vnext_qwen38_worker.Worker \
--speculative-config \
'{"method":"mtp","num_speculative_tokens":2}'Historical host paths and device addresses are retained as evidence. Use the serving command above.Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp4-mtp2 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-r2/portable/launch.py --config v620-fp16-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 32.783 · eligible10K 25.989 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 8503,
"content_chars": 18777,
"content_has_think_close": false,
"elapsed_seconds": 313.99245239887387,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T17:52:11Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.013237901031970978,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 18777,
"qwen_answer_text_sha256": "0efa285a05756d4fa88517b1757968c2239a589403da2f8f23b3765d95c9d4e2",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 18777,
"qwen_visible_text_sha256": "0efa285a05756d4fa88517b1757968c2239a589403da2f8f23b3765d95c9d4e2",
"raw_has_think_close": false,
"reasoning_chars": 25187,
"text_chars": 43965,
"text_sha256": "2faa6a681bead4a90b43fa84dc20b179f7bb62ae038ed66bf007960f41c5e720",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 310.6009135739878,
"vllm_decode_tps": 27.375965840404756,
"vllm_generation_tokens_delta": 8503.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 3.370070411125198,
"vllm_prefill_tps": 140.35315061624334,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 27.08026876135987
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 64.3155171200633,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T17:53:16Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.014873459935188293,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 10388,
"text_chars": 10388,
"text_sha256": "814c78cff59e5fe898f376561d8a7d6c88fe49407d365e6dfa2b9968ccd3efa1",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 61.00775268115103,
"vllm_decode_tps": 32.78271878744225,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 3.2730508069507778,
"vllm_prefill_tps": 144.51349150936454,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 31.09669469447673
},
"10K": {
"completion_tokens": 10000,
"content_chars": 26696,
"content_has_think_close": false,
"elapsed_seconds": 388.10585302114487,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T17:59:44Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.0162037571426481,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 26696,
"qwen_answer_text_sha256": "1395c5317617a065b003c9214deb93a08142c96d81125b0968c7a5c22c77dd41",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 26696,
"qwen_visible_text_sha256": "1395c5317617a065b003c9214deb93a08142c96d81125b0968c7a5c22c77dd41",
"raw_has_think_close": false,
"reasoning_chars": 25187,
"text_chars": 51884,
"text_sha256": "22fb1bdfc8331e84906af8185b7249afaff80ccd51cc9a6bcb763122a674d672",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 384.77785909199156,
"vllm_decode_tps": 25.9890213631269,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 3.313091953052208,
"vllm_prefill_tps": 142.7669399770947,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 25.7661664263929
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestMTP counter replayPASS9,359 natural tokens2K 32.92610K 25.631 · diagnostic2K MTP accepted 1108 / 1782 (62.18%) Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T23:42:11.23673073Z",
"FinishedAt": "2026-10-27T00:29:51.078728116Z"
},
"devices_after": [
{
"card": 5,
"render": 133,
"pci": "0000:16:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:19:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 7,
"render": 135,
"pci": "0000:1c:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 8,
"render": 136,
"pci": "0000:1f:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
}
],
"finished": "2026-10-27T00:29:51.231343+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 32.925748283050446,
"retest_10K_TPS": 25.630758150290323,
"retest_strict_tokens": 9359,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 9359,
"elapsed_seconds": 357.20782125892583,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 353.9098056239309,
"vllm_decode_tps": 26.44459082872938,
"vllm_generation_tokens_delta": 9359.0,
"vllm_prefill_seconds_delta": 3.2840392630314454,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 64.00938890292309,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 60.74273492000066,
"vllm_decode_tps": 32.925748283050446,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 3.2527504609897733,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 393.44662721501663,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 390.1562310940353,
"vllm_decode_tps": 25.630758150290323,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 3.2780255859252065,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 3.2527504609897733,
"retest_2K_total_seconds": 64.00938890292309,
"retest_10K_prefill_seconds": 3.2780255859252065,
"retest_10K_total_seconds": 393.44662721501663,
"retest_strict_prefill_seconds": 3.2840392630314454,
"retest_strict_total_seconds": 357.20782125892583,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 7194.0,
"vllm:spec_decode_num_drafts_total": 3597.0,
"vllm:spec_decode_num_accepted_tokens_total": 5761.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 9359.0,
"vllm:request_generation_tokens_sum": 9359.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 5761.0
},
"accepted_per_position": {
"0": 3109.0,
"1": 2652.0
},
"accepted_fraction": 0.800806227411732,
"committed_emitted_tokens": 9359.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "2276ca23df0bed94b682de122f7bd0c2b00260cb47bab2037e2f43969ea70a4d",
"/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "e556fe48dbe5c75807870feb3d7fa9d231f189e83256bd5b0205ac4e3a6c6bb8"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 1108.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_drafts_total": 891.0,
"vllm:spec_decode_num_draft_tokens_total": 1782.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1108.0
},
"accepted_per_position": {
"0": 658.0,
"1": 450.0
},
"accepted_fraction": 0.621773288439955,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "beff10249012b5c21c4b4f08a002d9d1b1942567e62040c29fc81c3c7f18886f",
"/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "9edb860863c37ec9015da0b8a76e971b6c6b6591fbc394f34c7a05d88e4b116f"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 7690.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_drafts_total": 3845.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6155.0,
"vllm:spec_decode_num_accepted_tokens_total": 6155.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:request_generation_tokens_count": 1.0
},
"accepted_per_position": {
"0": 3324.0,
"1": 2831.0
},
"accepted_fraction": 0.8003901170351105,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "d7b34f54ae7309059cd34e1d1e1c711375863431290ef5ec2011db877332c33a",
"/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "b7472565372751ecbb75b3d7944f94907595198288e1e00dce240ad47304310b"
}
}
},
"retest_MTP_2K_accepted": 1108.0,
"retest_MTP_2K_proposed": 1782.0,
"retest_MTP_2K_draft_batches": 891.0,
"retest_MTP_2K_acceptance_percent": 62.177328843995504,
"retest_loaded_admission": null,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 29542203392,
"1": 29554749440,
"2": 29552521216,
"3": 29499723776
},
"retest_memory_observation_count": 46,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-fp16-tp4-mtp2/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 20.64,
"retest_logged_model_load_seconds": 68.668489,
"retest_logged_available_KV_GiB": 6.11,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 6.11,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 93600,
"retest_logged_context_capacity_x": 2.72,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "v620-fp16-tp4-mtp2",
"image": "sha256:470486980ed01b78334c8e0b3ae36c9396d0ac9d5dba1183d581a857fe95674d",
"container": "ecc08d8c02c333ce0a0cbdc6b46ac5a15fc230cb1189ca255b88a68f07f03fd9",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 20.64,
"retest_logged_model_load_seconds": 68.668489,
"retest_logged_available_KV_GiB": 6.11,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 6.11,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 93600,
"retest_logged_context_capacity_x": 2.72,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-fp16-tp4-mtp2/service-final.log",
"source_log_sha256": "7b07a9c59e06465fab4a524d823a544a019ffd6d1da77b716d1da525392aaa3c",
"sealed_index_sha256": "77ccd32707a78a1145f1917f797db402a47ad46148bc20fb5ff6f1af0a7821aa",
"source_lines": {
"model_load": {
"line": 893,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 23:45:18 [gpu_model_runner.py:4579] Model loading took 20.64 GiB memory and 68.668489 seconds"
},
"automatic_KV_budget": {
"line": 986,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-27 00:02:07 [gpu_worker.py:453] Available KV cache memory: 6.11 GiB"
},
"grouped_KV_tokens": {
"line": 991,
"text": "(EngineCore_DP0 pid=85) INFO 10-27 00:02:07 [kv_cache_utils.py:1314] GPU KV cache size: 93,600 tokens"
},
"context_capacity": {
"line": 992,
"text": "(EngineCore_DP0 pid=85) INFO 10-27 00:02:07 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 2.72x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-fp16-tp4-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 27.525005340576172
} | Evidence and limitsCompleted strict-failed: no final answer. Fixed 2K and 10K speeds are diagnostic and ineligible; no strict rerun. {
"strict_tokens": 4212,
"source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.FP16_W8_TP_MTP_matrix.cells[1].result",
"source_revision": "5af8a8e02d310edaf1fbb289bcf430ea2fe75791",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"attention_KV_dtype": "FP16",
"memory_accounting_limit": "Retained FP16 parameter extents are not all model weights: TP4 projection tensors use lossless integer buffers. All packed-buffer extents include alternate representations. Draft tensors can share target embeddings/head. None of these logical sums is a unique physical allocation or measured memory traffic.",
"load_and_KV_allocator_bytes_per_rank": "[{\"KV_allocated_bytes\": 27199667712, \"KV_reserved_bytes\": 27770486784, \"draft_KV_logical_view_sum_bytes\": 387481600, \"load_allocated_bytes\": 20569802752, \"load_reserved_bytes\": 22196256768, \"rank\": 0, \"target_KV_logical_view_sum_bytes\": 6199705600}, {\"KV_allocated_bytes\": 27201705984, \"KV_reserved_bytes\": 27766292480, \"draft_KV_logical_view_sum_bytes\": 387481600, \"load_allocated_bytes\": 20571841024, \"load_reserved_bytes\": 22192062464, \"rank\": 1, \"target_KV_logical_view_sum_bytes\": 6199705600}, {\"KV_allocated_bytes\": 27201809408, \"KV_reserved_bytes\": 27764195328, \"draft_KV_logical_view_sum_bytes\": 387481600, \"load_allocated_bytes\": 20571944448, \"load_reserved_bytes\": 22189965312, \"rank\": 2, \"target_KV_logical_view_sum_bytes\": 6199705600}, {\"KV_allocated_bytes\": 27202460672, \"KV_reserved_bytes\": 27762098176, \"draft_KV_logical_view_sum_bytes\": 387481600, \"load_allocated_bytes\": 20572595712, \"load_reserved_bytes\": 22187868160, \"rank\": 3, \"target_KV_logical_view_sum_bytes\": 6199705600}]",
"sampled_peak_VRAM_bytes_by_PCI": "{\"0000:16:00.0\": 29575184384, \"0000:19:00.0\": 29587857408, \"0000:1c:00.0\": 29585731584, \"0000:1f:00.0\": 29533675520}",
"KV_capacity": "{\"GPU_KV_tokens\": 94400, \"actual_KV_shape\": [473, 2, 800, 1, 256], \"available_KV_GiB\": 6.15, \"graph_actual_GiB_each_rank\": 0.27, \"max_concurrency_at131072\": 2.73, \"service_ready_log_sha256\": \"881258f0123ae8f7314137c015c2fb15188bc00616dd64750de5cdc7b713a771\"}",
"attention_KV_logical_view_shape": "[473, 2, 800, 1, 256]",
"post_KV_allocator_allocated_bytes_by_rank": "{\"0\": 27199667712, \"1\": 27201705984, \"2\": 27201809408, \"3\": 27202460672}",
"post_KV_allocator_reserved_bytes_by_rank": "{\"0\": 27770486784, \"1\": 27766292480, \"2\": 27764195328, \"3\": 27762098176}",
"MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_sum": 4212.0,
"vllm:generation_tokens_total": 4212.0,
"vllm:spec_decode_num_drafts_total": 1471.0,
"vllm:spec_decode_num_draft_tokens_total": 2942.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_total": 2740.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 2740.0
},
"accepted_per_position": {
"0": 1403.0,
"1": 1337.0
},
"accepted_fraction": 0.9313392250169953,
"committed_emitted_tokens": 4212.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-before.txt": "f56d64dad72696f5f4697a23288e544e9fdce350bae5d41cb590eced112a31b2",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-after.txt": "6c6096b79e7766569d8f8b3a85d93317babe945918808cd1a82a042f605cee86"
}
},
"fixed2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:generation_tokens_total": 2000.0,
"vllm:spec_decode_num_drafts_total": 734.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 1468.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1266.0,
"vllm:spec_decode_num_accepted_tokens_total": 1266.0
},
"accepted_per_position": {
"0": 666.0,
"1": 600.0
},
"accepted_fraction": 0.8623978201634878,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-before.txt": "9412ba3b6d627fce6606a9d9b2ecbc8ede1018cdf4187ced3bb1f9c4c0b9ce36",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-after.txt": "267c4ee5121579bc10250e527b2f98a8bfa68eab7f732aede2e1218ab9355c26"
}
},
"fixed10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 6802.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:spec_decode_num_accepted_tokens_total": 6600.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6600.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_drafts_total": 3401.0
},
"accepted_per_position": {
"0": 3333.0,
"1": 3267.0
},
"accepted_fraction": 0.9703028521023228,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-before.txt": "b25e84e356c2d6c146b0b48a94b21be683ff3b00fda0db153f4d0c8e1077f853",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-after.txt": "617f37fd48e4cf04df1ec067066442637258348978916934fe3477d6b600e3d1"
}
}
},
"MTP_accounting_limit": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"source_evidence_commit": "7221c11",
"comparison_limit": "Native MTP requires inherited ordinary target decoder/residual/norm/collectives instead of the unsupported TP4 posted boundary. The r0 pre-HTTP initialization failure remains retained. | Actual800-block attentionKV uses inherited fallback; historical non-MTP uses784 native route. No isolated MTP gain is established. | Same pinned official checkpoint, FP16 activation/attentionKV, greedy profile/context131072 and fresh private rootSSD caches. | Original ladder exits2 on strict failure. Only the remaining fixed2K/10K were continued with original proxy/settings on unchanged loaded model; idle gap and actual parent exits retained. | StrictFAIL remainssolecorrectnessverdict. No repeatstrict, passfishing, officialspeedeligibility orFP16qualification. | MTPaccepted counts precede finalstoptrimming; committedtokens only speednumerator. E/draftbatches is descriptive, not measuredtokensperGPUstep. | Targetintegerpackcounts include retainedmultipleformats; targetplainFP16 sums are not totaluniquephysicalmemory. Draftsharedembedding/head overlaptarget; KVviews mayalias; allocator/cardVRAM/runtimecapacity are distinct. | Onecohort; ordinarytargetcollective/norm and800attentionfallback materially differ fromTP4offreference. No isolatedMTPgain orcausalclaim.",
"recorded_configuration_details": {
"source": "5af8a8e02d310edaf1fbb289bcf430ea2fe75791",
"image": "sha256:470486980ed01b78334c8e0b3ae36c9396d0ac9d5dba1183d581a857fe95674d",
"manifest": "f3db159511bad243ff7939343846e5febc89327f5774cae9419cbafcde6c1b59",
"strict_valid": false,
"strict_tokens": 4212,
"strict_failure": "4212 naturalstop;missing_think_close_or_parser_answer;no finalanswer",
"strict_diagnostic_TPS": 36.52689113332412,
"official_fixed2K_TPS": null,
"official_speed_eligible": false,
"diagnostic_fixed2K_TPS": 39.58600405759037,
"diagnostic_fixed10K_TPS": 28.223489817950945,
"metrics": {
"strict": {
"completion_tokens": 4212,
"finish_reason": "stop",
"elapsed_seconds": 118.7605559109943,
"vllm_prefill_seconds_delta": 3.4359108269563876,
"vllm_decode_seconds_delta": 115.31230469699949,
"vllm_decode_tps": 36.52689113332412,
"vllm_generation_tokens_delta": 4212.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "88d2ed6ee05e0b1aa5ba8823cbc24b5a88d80c8266b6d5fc74dc55902d3d6f23",
"server_e2e_seconds": 118.75620102882385,
"MTP": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_sum": 4212.0,
"vllm:generation_tokens_total": 4212.0,
"vllm:spec_decode_num_drafts_total": 1471.0,
"vllm:spec_decode_num_draft_tokens_total": 2942.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_total": 2740.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 2740.0
},
"accepted_per_position": {
"0": 1403.0,
"1": 1337.0
},
"accepted_fraction": 0.9313392250169953,
"committed_emitted_tokens": 4212.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-before.txt": "f56d64dad72696f5f4697a23288e544e9fdce350bae5d41cb590eced112a31b2",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-after.txt": "6c6096b79e7766569d8f8b3a85d93317babe945918808cd1a82a042f605cee86"
}
}
},
"fixed2K": {
"completion_tokens": 2000,
"finish_reason": "length",
"elapsed_seconds": 53.72339402302168,
"vllm_prefill_seconds_delta": 3.1883261629845947,
"vllm_decode_seconds_delta": 50.522906962025445,
"vllm_decode_tps": 39.58600405759037,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "8194a75ebf601b18347c55e6d90a806ab947b2498a3c2b33df8928ead3e02028",
"server_e2e_seconds": 53.71901798248291,
"MTP": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:generation_tokens_total": 2000.0,
"vllm:spec_decode_num_drafts_total": 734.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 1468.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1266.0,
"vllm:spec_decode_num_accepted_tokens_total": 1266.0
},
"accepted_per_position": {
"0": 666.0,
"1": 600.0
},
"accepted_fraction": 0.8623978201634878,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-before.txt": "9412ba3b6d627fce6606a9d9b2ecbc8ede1018cdf4187ced3bb1f9c4c0b9ce36",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-after.txt": "267c4ee5121579bc10250e527b2f98a8bfa68eab7f732aede2e1218ab9355c26"
}
}
},
"fixed10K": {
"completion_tokens": 10000,
"finish_reason": "length",
"elapsed_seconds": 357.5690232060151,
"vllm_prefill_seconds_delta": 3.242719638044946,
"vllm_decode_seconds_delta": 354.31479467998724,
"vllm_decode_tps": 28.223489817950945,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "4257fd9708b76c87e6eee072f2051550b27816ce3500f4812bc0e797317f66d9",
"server_e2e_seconds": 357.5645079612732,
"MTP": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 6802.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:spec_decode_num_accepted_tokens_total": 6600.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6600.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_drafts_total": 3401.0
},
"accepted_per_position": {
"0": 3333.0,
"1": 3267.0
},
"accepted_fraction": 0.9703028521023228,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-before.txt": "b25e84e356c2d6c146b0b48a94b21be683ff3b00fda0db153f4d0c8e1077f853",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-after.txt": "617f37fd48e4cf04df1ec067066442637258348978916934fe3477d6b600e3d1"
}
}
}
},
"weight_dtype": "Native FP16 with selected lossless integer packs; no quantization",
"activation_dtype": "FP16",
"attention_KV_dtype": "FP16",
"TP": 4,
"MTP_tokens": 2,
"MTP_precision": "Five native Qwen MTP projections FP16, same official revision; embedding/head shared with target",
"packed_logical_integer_bytes_per_rank": [
18799031148,
18800533083,
18800415543,
18799381732
],
"packed_bytes_limit": "Logical tensor extents include multiple retained lossless representations; not unique physical allocations or quantized weights. Draft shared embedding/head must not be added twice.",
"draft_five_projection_FP16_bytes_per_rank": 212336640,
"sampled_peak_VRAM_bytes": {
"0000:16:00.0": 29575184384,
"0000:19:00.0": 29587857408,
"0000:1c:00.0": 29585731584,
"0000:1f:00.0": 29533675520
},
"KV_capacity": {
"service_ready_log_sha256": "881258f0123ae8f7314137c015c2fb15188bc00616dd64750de5cdc7b713a771",
"available_KV_GiB": 6.15,
"GPU_KV_tokens": 94400,
"max_concurrency_at131072": 2.73,
"graph_actual_GiB_each_rank": 0.27,
"actual_KV_shape": [
473,
2,
800,
1,
256
]
},
"load_and_KV_allocator_bytes_per_rank": [
{
"rank": 0,
"load_allocated_bytes": 20569802752,
"load_reserved_bytes": 22196256768,
"KV_allocated_bytes": 27199667712,
"KV_reserved_bytes": 27770486784,
"target_KV_logical_view_sum_bytes": 6199705600,
"draft_KV_logical_view_sum_bytes": 387481600
},
{
"rank": 1,
"load_allocated_bytes": 20571841024,
"load_reserved_bytes": 22192062464,
"KV_allocated_bytes": 27201705984,
"KV_reserved_bytes": 27766292480,
"target_KV_logical_view_sum_bytes": 6199705600,
"draft_KV_logical_view_sum_bytes": 387481600
},
{
"rank": 2,
"load_allocated_bytes": 20571944448,
"load_reserved_bytes": 22189965312,
"KV_allocated_bytes": 27201809408,
"KV_reserved_bytes": 27764195328,
"target_KV_logical_view_sum_bytes": 6199705600,
"draft_KV_logical_view_sum_bytes": 387481600
},
{
"rank": 3,
"load_allocated_bytes": 20572595712,
"load_reserved_bytes": 22187868160,
"KV_allocated_bytes": 27202460672,
"KV_reserved_bytes": 27762098176,
"target_KV_logical_view_sum_bytes": 6199705600,
"draft_KV_logical_view_sum_bytes": 387481600
}
],
"configured_context": 131072,
"complete_ladders": 1,
"sample_count": 1,
"repeatability_established": false,
"FP16_release_qualification": false,
"archive": {
"path": "/home/ai/vnext-v620-qwen38-fp16-tp4-mtp2-service-r1-20261023/raw-evidence.tar.gz",
"bytes": 447218,
"sha256": "002994242cea527782ecaf7bbc3da98b6f15755f28f38338bc3c213fb50ab02d",
"files": 249,
"index_sha256": "c707c0df10917b1a3a4a5a328cf66ac149f5e67622d686b5997edce0c1f6e89a"
},
"local_primary_record": ".work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/measurement-closure.json",
"review": {
"path": ".work/fp16-w8-tp-mtp-matrix/reviews/FP16-TP4-MTP2-r1-final-cell-review.json",
"sha256": "7590389df32af23bad89dfcb365a3947f274f148ec871c47ce91c0465596c987"
},
"closure": {
"container_id": "b5d65c39ad36036f9d62f935335427adf830338159429b9789b2f31ca2459029",
"image_id": "sha256:470486980ed01b78334c8e0b3ae36c9396d0ac9d5dba1183d581a857fe95674d",
"exit_code": 0,
"OOMKilled": false,
"restart_count": 0,
"bank_idle": true,
"devices": [
{
"pci": "0000:16:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
},
{
"pci": "0000:19:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
},
{
"pci": "0000:1c:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
},
{
"pci": "0000:1f:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
}
],
"started_at": "2026-10-23T12:53:46.083011206Z",
"finished_at": "2026-10-23T13:42:15.417744232Z",
"closed_at": "2026-10-23T13:42:15.583708+00:00",
"kernel_fault_matches": [],
"startup_requests_completed_and_drained": 3,
"official_phase_samples": 11,
"complete_ladders": 1,
"strict_valid": false,
"official_fixed_2000_tps": null,
"measured_fixed_2000_tps": 39.58600405759037,
"valid_comparator": false,
"official_speed_eligible": false,
"promotion_eligible": false,
"qualifying_ladders": 0,
"FP16_qualification": false,
"baseline_kind": "FP16_TP4_MTP2"
},
"final_counters": {
"vllm:prompt_tokens_total": 6622,
"vllm:generation_tokens_total": 38212,
"vllm:request_decode_time_seconds_count": 14,
"vllm:request_prefill_time_seconds_count": 14,
"vllm:num_requests_running": 0,
"vllm:num_requests_waiting": 0
},
"runtime_differences": [
"Native MTP requires inherited ordinary target decoder/residual/norm/collectives instead of the unsupported TP4 posted boundary. The r0 pre-HTTP initialization failure remains retained.",
"Actual800-block attentionKV uses inherited fallback; historical non-MTP uses784 native route. No isolated MTP gain is established.",
"Same pinned official checkpoint, FP16 activation/attentionKV, greedy profile/context131072 and fresh private rootSSD caches.",
"Original ladder exits2 on strict failure. Only the remaining fixed2K/10K were continued with original proxy/settings on unchanged loaded model; idle gap and actual parent exits retained."
],
"limits": [
"StrictFAIL remainssolecorrectnessverdict. No repeatstrict, passfishing, officialspeedeligibility orFP16qualification.",
"MTPaccepted counts precede finalstoptrimming; committedtokens only speednumerator. E/draftbatches is descriptive, not measuredtokensperGPUstep.",
"Targetintegerpackcounts include retainedmultipleformats; targetplainFP16 sums are not totaluniquephysicalmemory. Draftsharedembedding/head overlaptarget; KVviews mayalias; allocator/cardVRAM/runtimecapacity are distinct.",
"Onecohort; ordinarytargetcollective/norm and800attentionfallback materially differ fromTP4offreference. No isolatedMTPgain orcausalclaim."
]
},
"MTP_2K_draft_batches": 734.0,
"MTP_2K_proposed": 1468.0,
"MTP_2K_accepted": 1266.0,
"MTP_2K_acceptance_percent": 86.23978201634877,
"MTP_2K_committed": 2000.0,
"detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
"detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
"detail_source_pointer": "FP16_W8_TP_MTP_matrix.cells[1]",
"loaded_metadata_sources_by_rank": {
"0": {
"path": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-0-pid-103.json",
"sha256": "928ef5f45f3514b3c1299bb216e29ccb1e7e68a1e3313e3265d40333654df8b5"
},
"1": {
"path": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-1-pid-104.json",
"sha256": "b42567a2c1f8817818b7794e37a5be515d7806e6504792b0933be1208f469df5"
},
"2": {
"path": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-2-pid-105.json",
"sha256": "a8646e8e4ef3ebba23f68645457131c031de8179741969d2292527f81de54c93"
},
"3": {
"path": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-3-pid-106.json",
"sha256": "67ad2947d99cb0464fdcbc70550ea2c6d111db32980b2f6edd6e2ed563e62d0a"
}
},
"memory_source": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-3-pid-106.json",
"memory_source_sha256": "67ad2947d99cb0464fdcbc70550ea2c6d111db32980b2f6edd6e2ed563e62d0a",
"retest_strict_tokens": 9359,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 7194.0,
"vllm:spec_decode_num_drafts_total": 3597.0,
"vllm:spec_decode_num_accepted_tokens_total": 5761.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 9359.0,
"vllm:request_generation_tokens_sum": 9359.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 5761.0
},
"accepted_per_position": {
"0": 3109.0,
"1": 2652.0
},
"accepted_fraction": 0.800806227411732,
"committed_emitted_tokens": 9359.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "2276ca23df0bed94b682de122f7bd0c2b00260cb47bab2037e2f43969ea70a4d",
"/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "e556fe48dbe5c75807870feb3d7fa9d231f189e83256bd5b0205ac4e3a6c6bb8"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 1108.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_drafts_total": 891.0,
"vllm:spec_decode_num_draft_tokens_total": 1782.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1108.0
},
"accepted_per_position": {
"0": 658.0,
"1": 450.0
},
"accepted_fraction": 0.621773288439955,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "beff10249012b5c21c4b4f08a002d9d1b1942567e62040c29fc81c3c7f18886f",
"/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "9edb860863c37ec9015da0b8a76e971b6c6b6591fbc394f34c7a05d88e4b116f"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 7690.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_drafts_total": 3845.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6155.0,
"vllm:spec_decode_num_accepted_tokens_total": 6155.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:request_generation_tokens_count": 1.0
},
"accepted_per_position": {
"0": 3324.0,
"1": 2831.0
},
"accepted_fraction": 0.8003901170351105,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "d7b34f54ae7309059cd34e1d1e1c711375863431290ef5ec2011db877332c33a",
"/home/ai/qwen38-release-retests-20261026/v620-fp16-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "b7472565372751ecbb75b3d7944f94907595198288e1e00dce240ad47304310b"
}
}
},
"retest_MTP_2K_accepted": 1108.0,
"retest_MTP_2K_proposed": 1782.0,
"retest_MTP_2K_draft_batches": 891.0,
"retest_MTP_2K_acceptance_percent": 62.177328843995504,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 29542203392,
"1": 29554749440,
"2": 29552521216,
"3": 29499723776
},
"retest_memory_observation_count": 46,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-fp16-tp4-mtp2/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 6.11,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 6.11,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 93600,
"retest_memory_log_report": {
"profile": "v620-fp16-tp4-mtp2",
"image": "sha256:470486980ed01b78334c8e0b3ae36c9396d0ac9d5dba1183d581a857fe95674d",
"container": "ecc08d8c02c333ce0a0cbdc6b46ac5a15fc230cb1189ca255b88a68f07f03fd9",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 20.64,
"retest_logged_model_load_seconds": 68.668489,
"retest_logged_available_KV_GiB": 6.11,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 6.11,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 93600,
"retest_logged_context_capacity_x": 2.72,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-fp16-tp4-mtp2/service-final.log",
"source_log_sha256": "7b07a9c59e06465fab4a524d823a544a019ffd6d1da77b716d1da525392aaa3c",
"sealed_index_sha256": "77ccd32707a78a1145f1917f797db402a47ad46148bc20fb5ff6f1af0a7821aa",
"source_lines": {
"model_load": {
"line": 893,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 23:45:18 [gpu_model_runner.py:4579] Model loading took 20.64 GiB memory and 68.668489 seconds"
},
"automatic_KV_budget": {
"line": 986,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-27 00:02:07 [gpu_worker.py:453] Available KV cache memory: 6.11 GiB"
},
"grouped_KV_tokens": {
"line": 991,
"text": "(EngineCore_DP0 pid=85) INFO 10-27 00:02:07 [kv_cache_utils.py:1314] GPU KV cache size: 93,600 tokens"
},
"context_capacity": {
"line": 992,
"text": "(EngineCore_DP0 pid=85) INFO 10-27 00:02:07 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 2.72x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-fp16-tp4-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 27.525005340576172,
"sampled_peak_VRAM_GiB": 27.55583953857422
} |
| V620Current Qwen3.8 | FP16Qwen/Qwen3.8-27B | TP2MTP2 | NOT RUNNot reportedCapacity blocked; no measured ladder | — | — | — / —10K: — / —Strict: — / — | 25.05 / 28.55FP16 parameter extents / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:0a8ae2ddb11ddae09242459cbfdbb56b028bdd1c1a9e02bdd621a76c8a3d9e08Runtime image config: sha256:a98060dd75e701abc267843ccd7d8513c7baf7b6abb168c6c289bd2dbad5d399 | Capacity-limited diagnostic; full benchmark unsupported.Short-request diagnostic · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256 mkdir -p launch-tools-v620-serve-r5 tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp2-mtp2 GPU_RENDER_NODES=renderD128,renderD129 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-fp16-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Preflight cleanup compatibilityAfter its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged. Retained lab Docker command · not portabledocker run -d \
--name serve-v620-fp16-tp2-mtp2-20261028 \
--network host \
--ipc host \
--cpuset-cpus 0-7,16-23 \
--cpuset-mems 0 \
--security-opt label=disable \
--shm-size 67108864 \
--device /dev/kfd:/dev/kfd:rwm \
--device /dev/dri/card5:/dev/dri/card5:rwm \
--device /dev/dri/card6:/dev/dri/card6:rwm \
--device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
--device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
--user root \
--workdir /tmp \
-e VLLM_DISABLE_COMPILE_CACHE=1 \
-e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
-e ROCM_VERSION=7.14.0 \
-e TORCH_VERSION=2.12.0+rocm7.14.0 \
-e VLLM_TARGET_DEVICE=rocm \
-e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
-e PYTORCH_ROCM_ARCH=gfx1030 \
-e GPU_ARCHS=gfx1030 \
-e AMDGPU_TARGETS=gfx1030 \
-e CMAKE_HIP_ARCHITECTURES=gfx1030 \
-e PYTHONDONTWRITEBYTECODE=1 \
-e PIP_NO_CACHE_DIR=1 \
-e PIP_DISABLE_PIP_VERSION_CHECK=1 \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp2/cache/.triton,dst=/root/.triton \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
--mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp2/cache/vllm,dst=/root/.cache/vllm \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-fp16-tp2-mtp2/cache/torch_extensions,dst=/root/.cache/torch_extensions \
--entrypoint /usr/bin/env \
sha256:a98060dd75e701abc267843ccd7d8513c7baf7b6abb168c6c289bd2dbad5d399 \
V756F94_BOUNDARY_MODE=posted \
DO_NOT_TRACK=1 \
GPU_DEVICE_ORDINAL=0,1 \
GPU_MAX_HW_QUEUES=1 \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
OMP_NUM_THREADS=4 \
TEMP=/tmp/torchinductor_root \
TMPDIR=/tmp/torchinductor_root \
TMP=/tmp/torchinductor_root \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
TRITON_CACHE_DIR=/root/.triton \
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
VLLM_GPU_MEMORY_UTILIZATION=0.95 \
VLLM_LOGGING_LEVEL=INFO \
VLLM_PORT=9002 \
VLLM_RDNA_FORCE_FP16=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
NCCL_P2P_DISABLE=0 \
NCCL_ALGO=Tree \
NCCL_PROTO=LL \
NCCL_MIN_NCHANNELS=4 \
NCCL_MAX_NCHANNELS=4 \
NCCL_NTHREADS=256 \
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
HIP_VISIBLE_DEVICES=0,1 \
CUDA_VISIBLE_DEVICES=0,1 \
ROCR_VISIBLE_DEVICES=0,1 \
VLLM_CACHE_ROOT=/root/.cache/vllm \
PYTORCH_ROCM_ARCH=gfx1030 \
PYTHONHASHSEED=0 \
/opt/venv/bin/vllm \
serve \
/opt/local-models/qwen38-27b-hf \
--served-model-name \
Qwen/Qwen3.8-27B \
--enable-auto-tool-choice \
--tool-call-parser \
qwen3_coder \
--dtype \
half \
--host \
0.0.0.0 \
--port \
9002 \
--tensor-parallel-size \
2 \
--max-model-len \
131072 \
--gpu-memory-utilization \
0.95 \
--trust-remote-code \
--generation-config \
vllm \
-O=3 \
--reasoning-parser \
qwen3 \
--async-scheduling \
--hf-overrides \
'{"architectures":["Qwen3_5ForCausalLM"]}' \
--language-model-only \
--max-num-seqs \
2 \
--max-num-batched-tokens \
2048 \
--mamba-ssm-cache-dtype \
float32 \
--no-enable-prefix-caching \
--no-enforce-eager \
--attention-backend \
TRITON_ATTN \
--disable-custom-all-reduce \
--seed \
0 \
--worker-cls \
vllm.vnext_qwen38_worker.Worker \
--speculative-config \
'{"method":"mtp","num_speculative_tokens":2}'Historical host paths and device addresses are retained as evidence. Use the serving command above.Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Reproduce the capacity failure · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-fp16-tp2-mtp2 GPU_RENDER_NODES=renderD128,renderD129 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-r2/portable/launch.py --config v620-fp16-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | Capacity limitation reproducedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: capacity-blocked-reviewed The unchanged FP16 TP2 MTP2 profile served a short request but could not complete the prescribed 2,000-token startup requests. KV saturation and zero generation were logged before the original 30-minute request timeout. No strict or speed sample exists. Reviewed capacity outcomeExact-command evidence and failures Earlier exported-image retest · separate evidenceStartup timeout; KV cache full; no measured ladderVerification · Raw retestEarlier setup attempt retainedAdmission observer repair and replayObserved startup limit and closureNO LADDER Retained failureTimeoutError('timed out') | Evidence and limitsCapacity-blocked under unchanged context131072, FP16 KV and utilization0.95 after bounded reserve-accounting repair. HTTP and 0.40GiB KV reached, but all three startup requests failed to complete; cache100% and preemptions observed. No strict verdict, TPS or measured latency. Exact cohort stopped and evidence retained; no further retry selected. {
"source": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/closure.json#$",
"source_revision": "bd4bcaf2a85f5f2e7620a1383eb258d2429bb28c",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"strict_tokens": null,
"attention_KV_dtype": "FP16",
"sampled_VRAM_startup_max_bytes_by_PCI": "{\"0000:16:00.0\": 30658633728, \"0000:19:00.0\": 30656995328}",
"KV_capacity": "{\"interpretation\": \"Exact captured kv_cache_utils.py1297-1300 logs floor(num_blocks/group_count)*min_block_size; max concurrency uses a different per-group max-memory calculation799-818. Thus 2400 is a grouped capacity indicator, not a separately proven exact single-request ceiling. Earlier473+1926=2399 observation is corroborative, not a complete proof of allocator exhaustion arithmetic.\", \"reported_GiB\": 0.4, \"reported_grouped_tokens\": 2400}",
"source_evidence_commit": "68f943b",
"comparison_limit": "Scoped post-draft-sharing reserve cleanup before memory profiling; actual FP16 target/draft tensor identities unchanged. Original 0.95 utilization, 131072 context, FP16 attention KV and 800-block fallback retained. No throughput, strict outcome or completed startup inferred. No number inferred. Originalprofile0.95preserved; no context/KV precision reduction. r0preHTTPfailure andr1failedstartup retained; no further speculative FP16tuning for this cell.",
"loaded_metadata_sources_by_rank": {
"0": {
"path": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-0-pid-103.json",
"sha256": "1574ea88a3fe9200c54eb968d1a210900afa83e8e7dc4fcbeba0b58e129e95ed"
},
"1": {
"path": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-1-pid-104.json",
"sha256": "4a31d3561be3db61a8df5134ea6669ef019e85afa15f4076db010f149aae5d06"
}
},
"memory_accounting_limit": "Retained FP16 parameter extents are not all model weights: TP4 projection tensors use lossless integer buffers. All packed-buffer extents include alternate representations. Draft tensors can share target embeddings/head. None of these logical sums is a unique physical allocation or measured memory traffic.",
"memory_source": "/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/fp16-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/loaded-model-audits/rank-1-pid-104.json",
"memory_source_sha256": "4a31d3561be3db61a8df5134ea6669ef019e85afa15f4076db010f149aae5d06",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30659104768,
"1": 30657032192
},
"retest_memory_observation_count": 46,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-fp16-tp2-mtp2/runtime-observations.jsonl",
"retest_outcome_source": "retest-evidence/v620-fp16-tp2-mtp2/outcome.json",
"retest_logged_available_KV_GiB": 0.36,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 0.36,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 2400,
"retest_memory_log_report": {
"profile": "v620-fp16-tp2-mtp2",
"image": "sha256:a98060dd75e701abc267843ccd7d8513c7baf7b6abb168c6c289bd2dbad5d399",
"container": "7da02e7c6d52440d7406b6ae5c7c29a90753fa6e5d16e0fe3656656a0ab9b989",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 25.76,
"retest_logged_model_load_seconds": 15.149719,
"retest_logged_available_KV_GiB": 0.36,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 0.36,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 2400,
"retest_logged_context_capacity_x": 0.08,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-fp16-tp2-mtp2/service-final.log",
"source_log_sha256": "3b8d72488fed8ca3839b7289454b1ea79619e1f6c0abe3566f286e7a5231fe0a",
"sealed_index_sha256": "ec69fada416a475bb7eb71d8a77a656e3a1b45000a1dec03b45a57191a86dc0b",
"source_lines": {
"model_load": {
"line": 117,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-27 02:20:55 [gpu_model_runner.py:4604] Model loading took 25.76 GiB memory and 15.149719 seconds"
},
"automatic_KV_budget": {
"line": 167,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-27 02:36:10 [gpu_worker.py:453] Available KV cache memory: 0.36 GiB"
},
"grouped_KV_tokens": {
"line": 170,
"text": "(EngineCore_DP0 pid=85) INFO 10-27 02:36:10 [kv_cache_utils.py:1314] GPU KV cache size: 2,400 tokens"
},
"context_capacity": {
"line": 171,
"text": "(EngineCore_DP0 pid=85) INFO 10-27 02:36:10 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 0.08x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-fp16-tp2-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.553516387939453,
"sampled_peak_VRAM_GiB": 28.553077697753906
} |
| V620Current Qwen3.8 | 8-bit RTNQwen/Qwen3.8-27B | TP4MTP off | PASS7563 tokens | 29.787 | 28.871 | 1.105 / 68.26310K: 1.107 / 347.487Strict: — / — | 5.80 / 23.40packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:74b0db52bb4f4aacf390ec2d18c6d8ff9d6ed403ca3c942fb55bc0bda09cfa3fRuntime image config: sha256:4f2d919ceed2b8b21cfd4d13661d83d197fbdddcc82e5a799dc4689c8c407794 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256 mkdir -p launch-tools-v620-serve-r5 tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp4-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w8-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Preflight cleanup compatibilityAfter its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged. Retained lab Docker command · not portabledocker run -d \
--name serve-v620-w8-tp4-mtp0-20261028 \
--network host \
--ipc host \
--cpuset-cpus 0-7,16-23 \
--cpuset-mems 0 \
--security-opt label=disable \
--shm-size 67108864 \
--device /dev/kfd:/dev/kfd:rwm \
--device /dev/dri/card5:/dev/dri/card5:rwm \
--device /dev/dri/card6:/dev/dri/card6:rwm \
--device /dev/dri/card7:/dev/dri/card7:rwm \
--device /dev/dri/card8:/dev/dri/card8:rwm \
--device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
--device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
--device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
--device /dev/dri/renderD136:/dev/dri/renderD136:rwm \
--user root \
--workdir /tmp \
-e VNEXT_RTN_W8_FUSED_CTA=1 \
-e VLLM_DISABLE_COMPILE_CACHE=1 \
-e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
-e ROCM_VERSION=7.14.0 \
-e TORCH_VERSION=2.12.0+rocm7.14.0 \
-e VLLM_TARGET_DEVICE=rocm \
-e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
-e PYTORCH_ROCM_ARCH=gfx1030 \
-e GPU_ARCHS=gfx1030 \
-e AMDGPU_TARGETS=gfx1030 \
-e CMAKE_HIP_ARCHITECTURES=gfx1030 \
-e PYTHONDONTWRITEBYTECODE=1 \
-e PIP_NO_CACHE_DIR=1 \
-e PIP_DISABLE_PIP_VERSION_CHECK=1 \
-e VNEXT_GDN_HEAD32=1 \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp0/cache/torch_extensions,dst=/root/.cache/torch_extensions \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp0/cache/.triton,dst=/root/.triton \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
--mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp0/cache/vllm,dst=/root/.cache/vllm \
--entrypoint /usr/bin/env \
sha256:4f2d919ceed2b8b21cfd4d13661d83d197fbdddcc82e5a799dc4689c8c407794 \
V756F94_BOUNDARY_MODE=posted \
DO_NOT_TRACK=1 \
GPU_DEVICE_ORDINAL=0,1,2,3 \
GPU_MAX_HW_QUEUES=1 \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
OMP_NUM_THREADS=4 \
TEMP=/tmp/torchinductor_root \
TMPDIR=/tmp/torchinductor_root \
TMP=/tmp/torchinductor_root \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
TRITON_CACHE_DIR=/root/.triton \
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
VLLM_GPU_MEMORY_UTILIZATION=0.95 \
VLLM_LOGGING_LEVEL=INFO \
VLLM_PORT=9002 \
VLLM_RDNA_FORCE_FP16=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
NCCL_P2P_DISABLE=0 \
NCCL_ALGO=Tree \
NCCL_PROTO=LL \
NCCL_MIN_NCHANNELS=4 \
NCCL_MAX_NCHANNELS=4 \
NCCL_NTHREADS=256 \
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
HIP_VISIBLE_DEVICES=0,1,2,3 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
ROCR_VISIBLE_DEVICES=0,1,2,3 \
VLLM_CACHE_ROOT=/root/.cache/vllm \
PYTORCH_ROCM_ARCH=gfx1030 \
PYTHONHASHSEED=0 \
/opt/venv/bin/python \
-m \
vllm.vnext_rtn_entry \
serve \
/opt/local-models/qwen38-27b-hf \
--served-model-name \
Qwen/Qwen3.8-27B \
--enable-auto-tool-choice \
--tool-call-parser \
qwen3_coder \
--dtype \
half \
--host \
0.0.0.0 \
--port \
9002 \
--tensor-parallel-size \
4 \
--max-model-len \
131072 \
--gpu-memory-utilization \
0.95 \
--trust-remote-code \
--generation-config \
vllm \
-O=3 \
--reasoning-parser \
qwen3 \
--async-scheduling \
--hf-overrides \
'{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":8,"group_size":128}}' \
--language-model-only \
--max-num-seqs \
2 \
--max-num-batched-tokens \
2048 \
--mamba-ssm-cache-dtype \
float32 \
--no-enable-prefix-caching \
--no-enforce-eager \
--attention-backend \
TRITON_ATTN \
--disable-custom-all-reduce \
--seed \
0 \
--worker-cls \
vllm.vnext_rtn_worker.Worker \
--quantization \
vnext_rtnHistorical host paths and device addresses are retained as evidence. Use the serving command above.Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-r4.tar.gz.sha256 mkdir -p launch-tools-v620-r4 tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp4-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-v620-r4/portable/launch.py --config v620-w8-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-v620-r4.tar.gz · SHA-256 launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. Portable observer compatibilityThe evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged. | All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 30.220 · eligible10K 29.262 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 18422,
"content_chars": 16775,
"content_has_think_close": false,
"elapsed_seconds": 651.1220705858432,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T20:38:30Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.017328426940366626,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 16775,
"qwen_answer_text_sha256": "39f0a17de9fb54affb4a939aedfb42dae34a7ed41e547fe64896061e5023ed9b",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 16775,
"qwen_visible_text_sha256": "39f0a17de9fb54affb4a939aedfb42dae34a7ed41e547fe64896061e5023ed9b",
"raw_has_think_close": false,
"reasoning_chars": 58239,
"text_chars": 75015,
"text_sha256": "65708cbbfc593557091707304990a3d2abc03c001ff0e9e842c59998f12be5b7",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 649.9961330560036,
"vllm_decode_tps": 28.341707070452312,
"vllm_generation_tokens_delta": 18422.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.1034011258743703,
"vllm_prefill_tps": 428.67456712551353,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 28.292697839937933
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 67.31212564720772,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T20:39:37Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008899811888113618,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 5641,
"text_chars": 5641,
"text_sha256": "58897a6606b49de4b7c6d917b21cdd4091c9547ccfe4d8a805378ae57fafc003",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 66.1812575191725,
"vllm_decode_tps": 30.220036230357305,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.1045551309362054,
"vllm_prefill_tps": 428.2267011870126,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 29.712328659508987
},
"10K": {
"completion_tokens": 10000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 342.86440511001274,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T20:45:20Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.018551325891166925,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 34008,
"text_chars": 34008,
"text_sha256": "8ec2c8505bb25ab3e4c7075862729060467b5a59839beada37ac51d9975fc583",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 341.7400923699606,
"vllm_decode_tps": 29.262004146631444,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.1042267300654203,
"vllm_prefill_tps": 428.354057297614,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 29.16604888393522
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS7,745 natural tokens2K 30.46210K 29.500 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T17:48:33.669083559Z",
"FinishedAt": "2026-10-26T18:39:14.260880813Z"
},
"devices_after": [
{
"card": 5,
"render": 133,
"pci": "0000:16:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:19:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 7,
"render": 135,
"pci": "0000:1c:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 8,
"render": 136,
"pci": "0000:1f:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
}
],
"finished": "2026-10-26T18:39:14.398926+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 30.461981881258904,
"retest_10K_TPS": 29.50011363293815,
"retest_strict_tokens": 7745,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 7745,
"elapsed_seconds": 261.14458458300214,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 260.0247855489142,
"vllm_decode_tps": 29.785622103870786,
"vllm_generation_tokens_delta": 7745.0,
"vllm_prefill_seconds_delta": 1.1050167970824987,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 66.77428013109602,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 65.65560992702376,
"vllm_decode_tps": 30.461981881258904,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 1.1085248430026695,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 340.1001898489194,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 338.98174510197714,
"vllm_decode_tps": 29.50011363293815,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 1.1070090770954266,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 1.1085248430026695,
"retest_2K_total_seconds": 66.77428013109602,
"retest_10K_prefill_seconds": 1.1070090770954266,
"retest_10K_total_seconds": 340.1001898489194,
"retest_strict_prefill_seconds": 1.1050167970824987,
"retest_strict_total_seconds": 261.14458458300214,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": "Complete record: chart-details/row-16-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 25104838656,
"1": 25121308672,
"2": 25121206272,
"3": 25071071232
},
"retest_memory_observation_count": 49,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-w8-tp4-mtp0/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 13.55,
"retest_logged_model_load_seconds": 69.213373,
"retest_logged_available_KV_GiB": 13.67,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 13.67,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 223440,
"retest_logged_context_capacity_x": 6.68,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "v620-w8-tp4-mtp0",
"image": "sha256:4f2d919ceed2b8b21cfd4d13661d83d197fbdddcc82e5a799dc4689c8c407794",
"container": "86d58c7ae420738f87e752aacaf340b66c6c2aaf59dc74b10e8688076ccb84d4",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 13.55,
"retest_logged_model_load_seconds": 69.213373,
"retest_logged_available_KV_GiB": 13.67,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 13.67,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 223440,
"retest_logged_context_capacity_x": 6.68,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w8-tp4-mtp0/service-final.log",
"source_log_sha256": "726feb8d07ee3d1196f5ae64c801549b862215683eec091a52bf1c101766150e",
"sealed_index_sha256": "24a7a4434d6a3c8345a672f0f4aba71a5b4211d8aaa88c3563efe7c930366fab",
"source_lines": {
"model_load": {
"line": 120,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 17:51:30 [gpu_model_runner.py:4579] Model loading took 13.55 GiB memory and 69.213373 seconds"
},
"automatic_KV_budget": {
"line": 139,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 17:55:28 [gpu_worker.py:453] Available KV cache memory: 13.67 GiB"
},
"grouped_KV_tokens": {
"line": 145,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 17:55:29 [kv_cache_utils.py:1314] GPU KV cache size: 223,440 tokens"
},
"context_capacity": {
"line": 146,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 17:55:29 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.68x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w8-tp4-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 23.396041870117188
}Complete retest loaded admission | Evidence and limitsstrict-valid W8 ordered-FP32 baseline {
"strict_tokens": 7563,
"source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.full_model_results.RTN8_fused_CTA",
"source_revision": "8668c2fc8e377c2990edd57988a85a03b87750bd",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"comparison_limit": "One admitted sample, with quantized projection/TP routes, Head32/compiler mode and root-SSD caches explicitly different from the preserved FP16 reference. No FP16 qualification or repeatability claim.",
"latency_sources": "[{\"path\": \"/home/joe/PhoneSync/Documents/gfx1030/.work/quantized-baselines/rtn8-fused-service-r0/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/summary.json\", \"sha256\": \"9564dacafa3c6f496df67ca9700658dc9f4e956124f24a361206b69ecc1dd12e\"}, {\"path\": \"/home/joe/PhoneSync/Documents/gfx1030/.work/quantized-baselines/rtn8-fused-service-r0/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/summary.json\", \"sha256\": \"56da2ba97b816e39f869fb067046b95ddc4e39c86cca2603988c4c7c9b7d3560\"}]",
"sampled_peak_VRAM_bytes_by_PCI": {
"0000:16:00.0": 25104637952,
"0000:19:00.0": 25121427456,
"0000:1c:00.0": 25121427456,
"0000:1f:00.0": 25071353856
},
"memory_accounting_limit": "Packed target matrices exclude FP16 embeddings/head/norm/recurrent weights. Sampled physical VRAM is not a guaranteed continuous peak.",
"memory_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#full_model_results.RTN8_fused_CTA.memory",
"memory_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
"retest_strict_tokens": 7745,
"retest_MTP_accounting_by_tier": {},
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 25104838656,
"1": 25121308672,
"2": 25121206272,
"3": 25071071232
},
"retest_memory_observation_count": 49,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-w8-tp4-mtp0/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 13.67,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 13.67,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 223440,
"retest_memory_log_report": {
"profile": "v620-w8-tp4-mtp0",
"image": "sha256:4f2d919ceed2b8b21cfd4d13661d83d197fbdddcc82e5a799dc4689c8c407794",
"container": "86d58c7ae420738f87e752aacaf340b66c6c2aaf59dc74b10e8688076ccb84d4",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 13.55,
"retest_logged_model_load_seconds": 69.213373,
"retest_logged_available_KV_GiB": 13.67,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 13.67,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 223440,
"retest_logged_context_capacity_x": 6.68,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w8-tp4-mtp0/service-final.log",
"source_log_sha256": "726feb8d07ee3d1196f5ae64c801549b862215683eec091a52bf1c101766150e",
"sealed_index_sha256": "24a7a4434d6a3c8345a672f0f4aba71a5b4211d8aaa88c3563efe7c930366fab",
"source_lines": {
"model_load": {
"line": 120,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 17:51:30 [gpu_model_runner.py:4579] Model loading took 13.55 GiB memory and 69.213373 seconds"
},
"automatic_KV_budget": {
"line": 139,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 17:55:28 [gpu_worker.py:453] Available KV cache memory: 13.67 GiB"
},
"grouped_KV_tokens": {
"line": 145,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 17:55:29 [kv_cache_utils.py:1314] GPU KV cache size: 223,440 tokens"
},
"context_capacity": {
"line": 146,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 17:55:29 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.68x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w8-tp4-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 23.396041870117188,
"sampled_peak_VRAM_GiB": 23.39615249633789
} |
| V620Current Qwen3.8 | 8-bit RTNQwen/Qwen3.8-27B | TP2MTP off | PASS8188 tokens | 20.931 | 19.987 | 0.773 / 96.33810K: 0.774 / 501.120Strict: 0.774 / 406.285 | 11.60 / 17.69packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:254536b65a31bbd16a2a864818b58581b4a6e0480d431c990dc7a51be38e016cRuntime image config: sha256:95626d8b1e5b583a5348b5e7c9a0cd91840877eb5e3407e916f53298f80cf9f6 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256 mkdir -p launch-tools-v620-serve-r5 tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp2-mtp0 GPU_RENDER_NODES=renderD128,renderD129 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w8-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Preflight cleanup compatibilityAfter its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged. Retained lab Docker command · not portabledocker run -d \
--name serve-v620-w8-tp2-mtp0-20261028 \
--network host \
--ipc host \
--cpuset-cpus 0-7,16-23 \
--cpuset-mems 0 \
--security-opt label=disable \
--shm-size 67108864 \
--device /dev/kfd:/dev/kfd:rwm \
--device /dev/dri/card5:/dev/dri/card5:rwm \
--device /dev/dri/card6:/dev/dri/card6:rwm \
--device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
--device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
--user root \
--workdir /tmp \
-e VNEXT_RTN_W8_FUSED_CTA=1 \
-e VLLM_DISABLE_COMPILE_CACHE=1 \
-e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
-e ROCM_VERSION=7.14.0 \
-e TORCH_VERSION=2.12.0+rocm7.14.0 \
-e VLLM_TARGET_DEVICE=rocm \
-e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
-e PYTORCH_ROCM_ARCH=gfx1030 \
-e GPU_ARCHS=gfx1030 \
-e AMDGPU_TARGETS=gfx1030 \
-e CMAKE_HIP_ARCHITECTURES=gfx1030 \
-e PYTHONDONTWRITEBYTECODE=1 \
-e PIP_NO_CACHE_DIR=1 \
-e PIP_DISABLE_PIP_VERSION_CHECK=1 \
-e VNEXT_GDN_HEAD32=1 \
--mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp0/cache/vllm,dst=/root/.cache/vllm \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp0/cache/torch_extensions,dst=/root/.cache/torch_extensions \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp0/cache/.triton,dst=/root/.triton \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
--entrypoint /usr/bin/env \
sha256:95626d8b1e5b583a5348b5e7c9a0cd91840877eb5e3407e916f53298f80cf9f6 \
V756F94_BOUNDARY_MODE=posted \
DO_NOT_TRACK=1 \
GPU_DEVICE_ORDINAL=0,1 \
GPU_MAX_HW_QUEUES=1 \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
OMP_NUM_THREADS=4 \
TEMP=/tmp/torchinductor_root \
TMPDIR=/tmp/torchinductor_root \
TMP=/tmp/torchinductor_root \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
TRITON_CACHE_DIR=/root/.triton \
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
VLLM_GPU_MEMORY_UTILIZATION=0.95 \
VLLM_LOGGING_LEVEL=INFO \
VLLM_PORT=9002 \
VLLM_RDNA_FORCE_FP16=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
NCCL_P2P_DISABLE=0 \
NCCL_ALGO=Tree \
NCCL_PROTO=LL \
NCCL_MIN_NCHANNELS=4 \
NCCL_MAX_NCHANNELS=4 \
NCCL_NTHREADS=256 \
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
HIP_VISIBLE_DEVICES=0,1 \
CUDA_VISIBLE_DEVICES=0,1 \
ROCR_VISIBLE_DEVICES=0,1 \
VLLM_CACHE_ROOT=/root/.cache/vllm \
PYTORCH_ROCM_ARCH=gfx1030 \
PYTHONHASHSEED=0 \
/opt/venv/bin/python \
-m \
vllm.vnext_rtn_entry \
serve \
/opt/local-models/qwen38-27b-hf \
--served-model-name \
Qwen/Qwen3.8-27B \
--enable-auto-tool-choice \
--tool-call-parser \
qwen3_coder \
--dtype \
half \
--host \
0.0.0.0 \
--port \
9002 \
--tensor-parallel-size \
2 \
--max-model-len \
131072 \
--gpu-memory-utilization \
0.95 \
--trust-remote-code \
--generation-config \
vllm \
-O=3 \
--reasoning-parser \
qwen3 \
--async-scheduling \
--hf-overrides \
'{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":8,"group_size":128}}' \
--language-model-only \
--max-num-seqs \
2 \
--max-num-batched-tokens \
2048 \
--mamba-ssm-cache-dtype \
float32 \
--no-enable-prefix-caching \
--no-enforce-eager \
--attention-backend \
TRITON_ATTN \
--disable-custom-all-reduce \
--seed \
0 \
--worker-cls \
vllm.vnext_rtn_worker.Worker \
--quantization \
vnext_rtnHistorical host paths and device addresses are retained as evidence. Use the serving command above.Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-r4.tar.gz.sha256 mkdir -p launch-tools-v620-r4 tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp2-mtp0 GPU_RENDER_NODES=renderD128,renderD129 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-v620-r4/portable/launch.py --config v620-w8-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-v620-r4.tar.gz · SHA-256 launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. Portable observer compatibilityThe evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged. | All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict FAIL2K 20.633 · diagnostic / ineligible10K 19.706 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 16516,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 868.9049477491062,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T22:14:21Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.018124318914487958,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 31279,
"text_chars": 31279,
"text_sha256": "67a2ad5a629db0dc2d1d7f6cac288c0ffd40df34d177b02d981357ccb3f1696d",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 868.1123461599927,
"vllm_decode_tps": 19.025187319425715,
"vllm_generation_tokens_delta": 16516.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.7737751340027899,
"vllm_prefill_tps": 611.2887054836458,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 19.00783283923588
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 97.72851502406411,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T22:16:01Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.018401601118966937,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 8011,
"text_chars": 8011,
"text_sha256": "c36609287d3c6bbbd2f917533763502b96d9181ac5a028ad6ceaf0ab8978787f",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 96.9333495539613,
"vllm_decode_tps": 20.632733823838727,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.7728608581237495,
"vllm_prefill_tps": 612.0118453770418,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 20.464856132394228
},
"10K": {
"completion_tokens": 10000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 508.2517163429875,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T22:24:30Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.020981378853321075,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 21006,
"text_chars": 21006,
"text_sha256": "1bd4ca5ff1900fcd998b29a09350f8aa5cdbc7fb321d5b4e835c123254180511",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 507.4579654929694,
"vllm_decode_tps": 19.70606568424936,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.7743516750633717,
"vllm_prefill_tps": 610.8335724350186,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 19.675290173052012
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS9,474 natural tokens2K 20.89810K 19.958 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T18:41:25.776124549Z",
"FinishedAt": "2026-10-26T19:39:36.891683362Z"
},
"devices_after": [
{
"card": 5,
"render": 133,
"pci": "0000:16:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:19:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
}
],
"finished": "2026-10-26T19:39:37.029335+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 20.897708334429417,
"retest_10K_TPS": 19.95841850159981,
"retest_strict_tokens": 9474,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 9474,
"elapsed_seconds": 473.81853826798033,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 473.03272304392885,
"vllm_decode_tps": 20.028212718637192,
"vllm_generation_tokens_delta": 9474.0,
"vllm_prefill_seconds_delta": 0.7731824229704216,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 96.49031426000874,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 95.70427378895693,
"vllm_decode_tps": 20.897708334429417,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 0.771934352000244,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 501.828497888986,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 501.0417032390833,
"vllm_decode_tps": 19.95841850159981,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 0.774901254917495,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 0.771934352000244,
"retest_2K_total_seconds": 96.49031426000874,
"retest_10K_prefill_seconds": 0.774901254917495,
"retest_10K_total_seconds": 501.828497888986,
"retest_strict_prefill_seconds": 0.7731824229704216,
"retest_strict_total_seconds": 473.81853826798033,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": "Complete record: chart-details/row-17-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 28108984320,
"1": 28109066240
},
"retest_memory_observation_count": 56,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-w8-tp2-mtp0/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 25.74,
"retest_logged_model_load_seconds": 120.96945,
"retest_logged_available_KV_GiB": 0.89,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 0.89,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 7056,
"retest_logged_context_capacity_x": 0.22,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "v620-w8-tp2-mtp0",
"image": "sha256:95626d8b1e5b583a5348b5e7c9a0cd91840877eb5e3407e916f53298f80cf9f6",
"container": "c3d8022dd7ce5fa8e2e24216effa6f2440fdfff5db6759f71cebf69280de0bc9",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 25.74,
"retest_logged_model_load_seconds": 120.96945,
"retest_logged_available_KV_GiB": 0.89,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 0.89,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 7056,
"retest_logged_context_capacity_x": 0.22,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w8-tp2-mtp0/service-final.log",
"source_log_sha256": "76f146dedd1427413c72a2fddf5bb88dc8157ba19bbd406d8e4f3a661b4da0eb",
"sealed_index_sha256": "b39dd2de284da65749c1d6bee3a8772a4ccc714135b8ce6dc83ff8ec1f7d42e1",
"source_lines": {
"model_load": {
"line": 92,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 18:45:06 [gpu_model_runner.py:4598] Model loading took 25.74 GiB memory and 120.969450 seconds"
},
"automatic_KV_budget": {
"line": 105,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 18:48:53 [gpu_worker.py:453] Available KV cache memory: 0.89 GiB"
},
"grouped_KV_tokens": {
"line": 107,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 18:48:53 [kv_cache_utils.py:1314] GPU KV cache size: 7,056 tokens"
},
"context_capacity": {
"line": 108,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 18:48:53 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 0.22x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w8-tp2-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 26.178607940673828
}Complete retest loaded admission | Evidence and limitsCompleted, independently reviewed ordered-FP32 W8 TP2 MTP-off matrix cell; one eligible sample, no promotion or FP16 qualification. {
"strict_tokens": 8188,
"source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.FP16_W8_TP_MTP_matrix.cells[3].result",
"source_revision": "1970f7c0fcbce8645012e1aa24493e3abb4fc243",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"attention_KV_dtype": "FP16",
"memory_accounting_limit": "Packed weight bytes exclude retained FP16 embeddings/head/norm/recurrent weights. Logical KV views may alias; allocator metrics, runtime capacity and sampled VRAM are distinct.",
"sampled_peak_VRAM_bytes_by_PCI": "{\"0000:16:00.0\": 18992091136, \"0000:19:00.0\": 18992078848}",
"KV_capacity": "{\"available_GiB\": 0.89, \"concurrency_at131072\": 0.22, \"tokens\": 7056}",
"attention_KV_logical_view_bytes_per_rank": 950534144,
"attention_KV_logical_view_shape": "[37, 2, 784, 2, 256]",
"post_KV_allocator_allocated_bytes_per_rank": 16209667072,
"post_KV_allocator_reserved_bytes_per_rank": 17171480576,
"source_evidence_commit": "85aa7a3",
"comparison_limit": "Selected U ordered-FP32 runtime extended only for TP2/MTP shapes; old U arithmetic preserved, M>32 retains original reconstruction. Native candidate45281e2c and source1970f7c are admitted. | TP2 reshards and ordinary decoder/fallback differ from historical TP4 U; no isolated TP or kernel-gain attribution. | Fresh private rootSSD caches, FP16 activation/attentionKV, context131072 and memoryutil0.95 retained. | Runtime reports7056 GPUKVtokens/0.22x at configured131072; full-context residency is not demonstrated. | One unchangedcohort/oneofficial2Ksample, no repeatability orFP16promotion. Strict alone correctness;engineeringreplay not an additionalgate. | NewTP2shards and ordinarydecoder/fallback differ fromTP4Ureference; no isolatedTPorUgain claim. | Packed12,460,636,160B/rank excludes retainedFP16embedding/head/norm/recurrentweights. LogicalKVviews950,534,144B/rank mayalias; allocator and observedVRAM are distinct. | Runtime0.89GiBavailableKV/7056tokens/0.22x atconfigured131072; no residentfullcontextcapacity demonstrated. | Generated source linkage and mapped DSO identities support route admission; no per-projection dynamic kernel trace was captured.",
"recorded_configuration_details": {
"source": "1970f7c0fcbce8645012e1aa24493e3abb4fc243",
"image": "sha256:95626d8b1e5b583a5348b5e7c9a0cd91840877eb5e3407e916f53298f80cf9f6",
"strict_valid": true,
"strict_tokens": 8188,
"strict_diagnostic_TPS": 20.1923964270491,
"official_fixed2K_TPS": 20.930590077305382,
"diagnostic_fixed10K_TPS": 19.986683546664402,
"metrics": {
"strict": {
"completion_tokens": 8188,
"finish_reason": "stop",
"elapsed_seconds": 406.28512426803354,
"vllm_prefill_seconds_delta": 0.7738519569975324,
"vllm_decode_seconds_delta": 405.49917042197194,
"vllm_decode_tps": 20.1923964270491,
"vllm_generation_tokens_delta": 8188.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "ce8f2a7ca5918c1cbdcc0df5a649d11b9df57dfd8a6319a8ba7852a040770d55",
"server_e2e_seconds": 406.28012442588806
},
"fixed2K": {
"completion_tokens": 2000,
"finish_reason": "length",
"elapsed_seconds": 96.33843383798376,
"vllm_prefill_seconds_delta": 0.7734992449986748,
"vllm_decode_seconds_delta": 95.55392335396027,
"vllm_decode_tps": 20.930590077305382,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "e608af506987d81b2a428e2d52738da1d72794e39cc6ea0df9504708305cf522",
"server_e2e_seconds": 96.3344099521637
},
"fixed10K": {
"completion_tokens": 10000,
"finish_reason": "length",
"elapsed_seconds": 501.12039932300104,
"vllm_prefill_seconds_delta": 0.7739897519932128,
"vllm_decode_seconds_delta": 500.3331331409863,
"vllm_decode_tps": 19.986683546664402,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "2d126341314c81e523f5197c8113887a1e7e43437821c325905f0ca620bd4002",
"server_e2e_seconds": 501.1148111820221
}
},
"weight_dtype": "symmetric RTN W8 in GPTQ-v2 layout; group128, ties-to-even, zero128",
"activation_dtype": "FP16",
"attention_KV_dtype": "FP16",
"TP": 2,
"MTP_tokens": 0,
"quantized_modules_per_rank": 256,
"packed_component_bytes_per_rank": {
"qweight": 12175278080,
"qzeros": 95119360,
"scales": 190238720,
"g_idx": 0
},
"packed_weight_bytes_per_rank": 12460636160,
"sampled_peak_VRAM_bytes": {
"0000:16:00.0": 18992091136,
"0000:19:00.0": 18992078848
},
"KV_capacity": {
"available_GiB": 0.89,
"tokens": 7056,
"concurrency_at131072": 0.22
},
"configured_context": 131072,
"complete_ladders": 1,
"sample_count": 1,
"repeatability_established": false,
"FP16_release_qualification": false,
"archive": {
"path": "/home/ai/vnext-v620-qwen38-rtn8-tp2-service-r0-20261023/raw-evidence.tar.gz",
"bytes": 983825,
"sha256": "c55b028de5f3d82086171c113b2b999cb779437ae2268f85761e0c0d0e124f7e",
"files": 238,
"index_sha256": "bcf623f7dda6961239239f8a905e5bbe745f060c694213f0509f126d2e5b8ffa"
},
"local_primary_record": ".work/fp16-w8-tp-mtp-matrix/w8-tp2-service-r0/sealed-mirror/retained/evidence/measurement-closure.json",
"review": {
"path": ".work/fp16-w8-tp-mtp-matrix/reviews/W8-TP2-final-cell-review.json",
"sha256": "3311aa6971d18b9b8b150197d05918113dc1c8b7d5f1ad03c9319363391cc8fd"
},
"closure": {
"container_id": "f7c859ecdb830c729b8d58601c0e285645014069254c8dc49aba1ec0e0d98f0d",
"image_id": "sha256:95626d8b1e5b583a5348b5e7c9a0cd91840877eb5e3407e916f53298f80cf9f6",
"exit_code": 0,
"OOMKilled": false,
"restart_count": 0,
"bank_idle": true,
"devices": [
{
"pci": "0000:16:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
},
{
"pci": "0000:19:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
}
],
"started_at": "2026-10-23T11:54:59.718906406Z",
"finished_at": "2026-10-23T12:52:52.033932533Z",
"closed_at": "2026-10-23T12:52:52.186527+00:00",
"kernel_fault_matches": [],
"startup_requests_completed_and_drained": 3,
"official_phase_samples": 11,
"complete_ladders": 1,
"strict_valid": true,
"official_fixed_2000_tps": 20.930590077305382,
"qualifying_ladders": 0,
"FP16_qualification": false,
"baseline_kind": "RTN8"
},
"final_counters": {
"vllm:prompt_tokens_total": 6622,
"vllm:generation_tokens_total": 42188,
"vllm:request_decode_time_seconds_count": 14,
"vllm:request_prefill_time_seconds_count": 14,
"vllm:num_requests_running": 0,
"vllm:num_requests_waiting": 0
},
"runtime_differences": [
"Selected U ordered-FP32 runtime extended only for TP2/MTP shapes; old U arithmetic preserved, M>32 retains original reconstruction. Native candidate45281e2c and source1970f7c are admitted.",
"TP2 reshards and ordinary decoder/fallback differ from historical TP4 U; no isolated TP or kernel-gain attribution.",
"Fresh private rootSSD caches, FP16 activation/attentionKV, context131072 and memoryutil0.95 retained.",
"Runtime reports7056 GPUKVtokens/0.22x at configured131072; full-context residency is not demonstrated."
],
"limits": [
"One unchangedcohort/oneofficial2Ksample, no repeatability orFP16promotion. Strict alone correctness;engineeringreplay not an additionalgate.",
"NewTP2shards and ordinarydecoder/fallback differ fromTP4Ureference; no isolatedTPorUgain claim.",
"Packed12,460,636,160B/rank excludes retainedFP16embedding/head/norm/recurrentweights. LogicalKVviews950,534,144B/rank mayalias; allocator and observedVRAM are distinct.",
"Runtime0.89GiBavailableKV/7056tokens/0.22x atconfigured131072; no residentfullcontextcapacity demonstrated.",
"Generated source linkage and mapped DSO identities support route admission; no per-projection dynamic kernel trace was captured."
]
},
"detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
"detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
"detail_source_pointer": "FP16_W8_TP_MTP_matrix.cells[3]",
"retest_strict_tokens": 9474,
"retest_MTP_accounting_by_tier": {},
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 28108984320,
"1": 28109066240
},
"retest_memory_observation_count": 56,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-w8-tp2-mtp0/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 0.89,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 0.89,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 7056,
"retest_memory_log_report": {
"profile": "v620-w8-tp2-mtp0",
"image": "sha256:95626d8b1e5b583a5348b5e7c9a0cd91840877eb5e3407e916f53298f80cf9f6",
"container": "c3d8022dd7ce5fa8e2e24216effa6f2440fdfff5db6759f71cebf69280de0bc9",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 25.74,
"retest_logged_model_load_seconds": 120.96945,
"retest_logged_available_KV_GiB": 0.89,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 0.89,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 7056,
"retest_logged_context_capacity_x": 0.22,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w8-tp2-mtp0/service-final.log",
"source_log_sha256": "76f146dedd1427413c72a2fddf5bb88dc8157ba19bbd406d8e4f3a661b4da0eb",
"sealed_index_sha256": "b39dd2de284da65749c1d6bee3a8772a4ccc714135b8ce6dc83ff8ec1f7d42e1",
"source_lines": {
"model_load": {
"line": 92,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 18:45:06 [gpu_model_runner.py:4598] Model loading took 25.74 GiB memory and 120.969450 seconds"
},
"automatic_KV_budget": {
"line": 105,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 18:48:53 [gpu_worker.py:453] Available KV cache memory: 0.89 GiB"
},
"grouped_KV_tokens": {
"line": 107,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 18:48:53 [kv_cache_utils.py:1314] GPU KV cache size: 7,056 tokens"
},
"context_capacity": {
"line": 108,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 18:48:53 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 0.22x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w8-tp2-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 26.178607940673828,
"sampled_peak_VRAM_GiB": 17.687763214111328
} |
| V620Current Qwen3.8 | 8-bit RTNQwen/Qwen3.8-27B | TP4MTP22K accepted 1094 / 1810 (60.44%) | PASS8037 tokens | 32.930 | 25.672 | 1.133 / 61.88010K: 1.148 / 390.687Strict: 1.133 / 293.352 | 5.90 / 28.43packed target/draft projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:e3ec7a663ee96460173b72b3be0fb6e5f4c77ab7a32c93b4824d60dcead4bde1Runtime image config: sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256 mkdir -p launch-tools-v620-serve-r5 tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp4-mtp2 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w8-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Preflight cleanup compatibilityAfter its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged. Retained lab Docker command · not portabledocker run -d \
--name serve-v620-w8-tp4-mtp2-20261028 \
--network host \
--ipc host \
--cpuset-cpus 0-7,16-23 \
--cpuset-mems 0 \
--security-opt label=disable \
--shm-size 67108864 \
--device /dev/kfd:/dev/kfd:rwm \
--device /dev/dri/card5:/dev/dri/card5:rwm \
--device /dev/dri/card6:/dev/dri/card6:rwm \
--device /dev/dri/card7:/dev/dri/card7:rwm \
--device /dev/dri/card8:/dev/dri/card8:rwm \
--device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
--device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
--device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
--device /dev/dri/renderD136:/dev/dri/renderD136:rwm \
--user root \
--workdir /tmp \
-e VNEXT_RTN_W8_NATIVE_DISPATCH=1 \
-e VNEXT_RTN_W8_FUSED_CTA=1 \
-e VLLM_DISABLE_COMPILE_CACHE=1 \
-e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
-e ROCM_VERSION=7.14.0 \
-e TORCH_VERSION=2.12.0+rocm7.14.0 \
-e VLLM_TARGET_DEVICE=rocm \
-e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
-e PYTORCH_ROCM_ARCH=gfx1030 \
-e GPU_ARCHS=gfx1030 \
-e AMDGPU_TARGETS=gfx1030 \
-e CMAKE_HIP_ARCHITECTURES=gfx1030 \
-e PYTHONDONTWRITEBYTECODE=1 \
-e PIP_NO_CACHE_DIR=1 \
-e PIP_DISABLE_PIP_VERSION_CHECK=1 \
-e VNEXT_GDN_HEAD32=1 \
--mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp2/cache/vllm,dst=/root/.cache/vllm \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp2/cache/torch_extensions,dst=/root/.cache/torch_extensions \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp2/cache/.triton,dst=/root/.triton \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp4-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
--entrypoint /usr/bin/env \
sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567 \
V756F94_BOUNDARY_MODE=posted \
DO_NOT_TRACK=1 \
GPU_DEVICE_ORDINAL=0,1,2,3 \
GPU_MAX_HW_QUEUES=1 \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
OMP_NUM_THREADS=4 \
TEMP=/tmp/torchinductor_root \
TMPDIR=/tmp/torchinductor_root \
TMP=/tmp/torchinductor_root \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
TRITON_CACHE_DIR=/root/.triton \
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
VLLM_GPU_MEMORY_UTILIZATION=0.95 \
VLLM_LOGGING_LEVEL=INFO \
VLLM_PORT=9002 \
VLLM_RDNA_FORCE_FP16=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
NCCL_P2P_DISABLE=0 \
NCCL_ALGO=Tree \
NCCL_PROTO=LL \
NCCL_MIN_NCHANNELS=4 \
NCCL_MAX_NCHANNELS=4 \
NCCL_NTHREADS=256 \
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
HIP_VISIBLE_DEVICES=0,1,2,3 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
ROCR_VISIBLE_DEVICES=0,1,2,3 \
VLLM_CACHE_ROOT=/root/.cache/vllm \
PYTORCH_ROCM_ARCH=gfx1030 \
PYTHONHASHSEED=0 \
/opt/venv/bin/python \
-m \
vllm.vnext_rtn_entry \
serve \
/opt/local-models/qwen38-27b-hf \
--served-model-name \
Qwen/Qwen3.8-27B \
--enable-auto-tool-choice \
--tool-call-parser \
qwen3_coder \
--dtype \
half \
--host \
0.0.0.0 \
--port \
9002 \
--tensor-parallel-size \
4 \
--speculative-config \
'{"method":"mtp","num_speculative_tokens":2}' \
--max-model-len \
131072 \
--gpu-memory-utilization \
0.95 \
--trust-remote-code \
--generation-config \
vllm \
-O=3 \
--reasoning-parser \
qwen3 \
--async-scheduling \
--hf-overrides \
'{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":8,"group_size":128}}' \
--language-model-only \
--max-num-seqs \
2 \
--max-num-batched-tokens \
2048 \
--mamba-ssm-cache-dtype \
float32 \
--no-enable-prefix-caching \
--no-enforce-eager \
--attention-backend \
TRITON_ATTN \
--disable-custom-all-reduce \
--seed \
0 \
--worker-cls \
vllm.vnext_rtn_worker.Worker \
--quantization \
vnext_rtnHistorical host paths and device addresses are retained as evidence. Use the serving command above.Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-r4.tar.gz.sha256 mkdir -p launch-tools-v620-r4 tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp4-mtp2 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-v620-r4/portable/launch.py --config v620-w8-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-v620-r4.tar.gz · SHA-256 launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. Portable observer compatibilityThe evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged. | All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict FAIL2K 41.076 · diagnostic / ineligible10K 28.673 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 4246,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 114.82530898298137,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T23:31:49Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.013302841922268271,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 7640,
"text_chars": 7640,
"text_sha256": "d4c7e10745a0e1515bdc4ea22c6f5f88dfe322b09830ee5f0129ea9ac13e4379",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 113.66740340087563,
"vllm_decode_tps": 37.354596594640704,
"vllm_generation_tokens_delta": 4246.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.142697302158922,
"vllm_prefill_tps": 413.93289290729155,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 36.977910511255956
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 49.847530151950195,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T23:32:41Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.012120805215090513,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 4415,
"text_chars": 4415,
"text_sha256": "70fbcad763bc75d1617c2fd34f2e0871229bddeeea35839eac3f6b3207b66ad3",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 48.690047662938014,
"vllm_decode_tps": 41.07615613451872,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.1360376980155706,
"vllm_prefill_tps": 416.359422602116,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 40.12234896901413
},
"10K": {
"completion_tokens": 10000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 349.91630028304644,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T23:38:31Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.017869479954242706,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 15915,
"text_chars": 15915,
"text_sha256": "ee7c5fc867eca6b5f4b62a627614b132f2a803ab6b18a5268cb2396bbe0ca109",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 348.76385564310476,
"vllm_decode_tps": 28.67269597521926,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.1339959390461445,
"vllm_prefill_tps": 417.10907747858585,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 28.57826283574393
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestMTP counter replayPASS10,158 natural tokens2K 33.38410K 24.895 · diagnostic2K MTP accepted 1122 / 1758 (63.82%) Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T19:41:48.215231213Z",
"FinishedAt": "2026-10-26T20:30:51.421619496Z"
},
"devices_after": [
{
"card": 5,
"render": 133,
"pci": "0000:16:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:19:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 7,
"render": 135,
"pci": "0000:1c:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 8,
"render": 136,
"pci": "0000:1f:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
}
],
"finished": "2026-10-26T20:30:51.545320+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 33.384149922977,
"retest_10K_TPS": 24.895468336506017,
"retest_strict_tokens": 10158,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 10158,
"elapsed_seconds": 412.2303533559898,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 411.0824744999409,
"vllm_decode_tps": 24.710369889537727,
"vllm_generation_tokens_delta": 10158.0,
"vllm_prefill_seconds_delta": 1.1355813550762832,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 61.05597344099078,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 59.90866937197279,
"vllm_decode_tps": 33.384149922977,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 1.136036392999813,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 402.8300866619684,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 401.6795291750459,
"vllm_decode_tps": 24.895468336506017,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 1.1382164129754528,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 1.136036392999813,
"retest_2K_total_seconds": 61.05597344099078,
"retest_10K_prefill_seconds": 1.1382164129754528,
"retest_10K_total_seconds": 402.8300866619684,
"retest_strict_prefill_seconds": 1.1355813550762832,
"retest_strict_total_seconds": 412.2303533559898,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:generation_tokens_total": 10158.0,
"vllm:spec_decode_num_draft_tokens_total": 8090.0,
"vllm:spec_decode_num_accepted_tokens_total": 6113.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:request_generation_tokens_sum": 10158.0,
"vllm:spec_decode_num_drafts_total": 4045.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6113.0
},
"accepted_per_position": {
"0": 3356.0,
"1": 2757.0
},
"accepted_fraction": 0.7556242274412855,
"committed_emitted_tokens": 10158.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "b0c7fc7c40a28cc81666d74d656deded0faa1edf58d638d792732d22f635a0a3",
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "4266e9362ae653507a568738210298328ae118dae456893caf38eac6b5e259c2"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:generation_tokens_total": 2000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 1758.0,
"vllm:spec_decode_num_accepted_tokens_total": 1122.0,
"vllm:spec_decode_num_drafts_total": 879.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1122.0,
"vllm:request_generation_tokens_sum": 2000.0
},
"accepted_per_position": {
"0": 652.0,
"1": 470.0
},
"accepted_fraction": 0.6382252559726962,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "b96c35f259b8cd18c589eb921758bf9ce6b7e775968528fc12d8e68656e4d816",
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "46b1bbec57fb4711473fc1074c580eeb32642138f4dda7915b8a8b898d7b7b55"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 7984.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_total": 6007.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_drafts_total": 3992.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6007.0
},
"accepted_per_position": {
"0": 3303.0,
"1": 2704.0
},
"accepted_fraction": 0.7523797595190381,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "f1cd06ff0d404e30521e2e57fd27914f1d1f3c059744359565eaba6cf8eac279",
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "6393ab308a1f1c437c10bdb67e948cc15e6d647d1a94cc644bb7a5cc99b3840f"
}
}
},
"retest_MTP_2K_accepted": 1122.0,
"retest_MTP_2K_proposed": 1758.0,
"retest_MTP_2K_draft_batches": 879.0,
"retest_MTP_2K_acceptance_percent": 63.82252559726962,
"retest_loaded_admission": "Complete record: chart-details/row-18-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30542385152,
"1": 30557487104,
"2": 30557450240,
"3": 30507130880
},
"retest_memory_observation_count": 46,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-w8-tp4-mtp2/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 8.25,
"retest_logged_model_load_seconds": 69.672332,
"retest_logged_available_KV_GiB": 18.53,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 18.53,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 284800,
"retest_logged_context_capacity_x": 8.25,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "v620-w8-tp4-mtp2",
"image": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
"container": "5457bd41fa75f10e8266c62e7d825106f9d861c3a122df35c891d462aff0ef09",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 8.25,
"retest_logged_model_load_seconds": 69.672332,
"retest_logged_available_KV_GiB": 18.53,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 18.53,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 284800,
"retest_logged_context_capacity_x": 8.25,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w8-tp4-mtp2/service-final.log",
"source_log_sha256": "196a6879d46ceafafa4327fa2e08df4d40d490907a2174e6abe1f3d38e50358f",
"sealed_index_sha256": "02fc37701b599060b58b02a78ac7165d21c59bda1893085577c5b7228fd3431e",
"source_lines": {
"model_load": {
"line": 154,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 19:44:59 [gpu_model_runner.py:4598] Model loading took 8.25 GiB memory and 69.672332 seconds"
},
"automatic_KV_budget": {
"line": 225,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 20:02:31 [gpu_worker.py:453] Available KV cache memory: 18.53 GiB"
},
"grouped_KV_tokens": {
"line": 229,
"text": "(EngineCore_DP0 pid=85) INFO 10-26 20:02:31 [kv_cache_utils.py:1314] GPU KV cache size: 284,800 tokens"
},
"context_capacity": {
"line": 230,
"text": "(EngineCore_DP0 pid=85) INFO 10-26 20:02:31 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 8.25x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w8-tp4-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.458877563476562
}Complete retest loaded admission | Evidence and limitsCompleted, independently reviewed W8 TP4 native MTP2 full ladder; strict PASS 8037. One eligible sample, no promotion or FP16 qualification. {
"strict_tokens": 8037,
"source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.FP16_W8_TP_MTP_matrix.cells[4].result",
"source_revision": "e4e131d2aa3424f1fb968a979a1e34bf5c510a16",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"attention_KV_dtype": "FP16",
"MTP_model_class": "vllm.model_executor.models.qwen3_5_mtp.Qwen3_5MTP",
"MTP_precision": "Five draft projections use the same W8/group128 RTN scheme; remaining draft parameters FP16, embedding/head shared with target. Same official checkpoint revision.",
"memory_accounting_limit": "Packed totals here include separately itemized target and draft projections, excluding retained FP16 parameters/shared embeddings/head. The 16 target plus one draft logical KV views are not a disjoint physical allocation sum. Sampled VRAM, allocator counters and runtime capacity are separate.",
"load_and_KV_allocator_bytes_per_rank": "[{\"KV_allocated\": 28207806976, \"KV_reserved\": 28718399488, \"KV_shapes\": [[1425, 2, 800, 1, 256]], \"load_allocated\": 8315061248, \"load_reserved\": 8856272896, \"rank\": 0}, {\"KV_allocated\": 28207913984, \"KV_reserved\": 28718399488, \"KV_shapes\": [[1425, 2, 800, 1, 256]], \"load_allocated\": 8315168256, \"load_reserved\": 8856272896, \"rank\": 1}, {\"KV_allocated\": 28207902720, \"KV_reserved\": 28718399488, \"KV_shapes\": [[1425, 2, 800, 1, 256]], \"load_allocated\": 8315156992, \"load_reserved\": 8856272896, \"rank\": 2}, {\"KV_allocated\": 28209225216, \"KV_reserved\": 28718399488, \"KV_shapes\": [[1425, 2, 800, 1, 256]], \"load_allocated\": 8316479488, \"load_reserved\": 8856272896, \"rank\": 3}]",
"sampled_peak_VRAM_bytes_by_PCI": "{\"0000:16:00.0\": 30507261952, \"0000:19:00.0\": 30521655296, \"0000:1c:00.0\": 30521917440, \"0000:1f:00.0\": 30471643136}",
"KV_capacity": "{\"available_GiB\": 18.49, \"draft_views\": 1, \"reported_concurrency_at131072\": 8.24, \"reported_grouped_tokens\": 284800, \"target_views\": 16, \"view_bytes\": 1167360000, \"view_shape\": [1425, 2, 800, 1, 256]}",
"attention_KV_logical_view_shape": "[1425, 2, 800, 1, 256]",
"MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 6366.0,
"vllm:spec_decode_num_accepted_tokens_total": 4853.0,
"vllm:request_generation_tokens_sum": 8037.0,
"vllm:generation_tokens_total": 8037.0,
"vllm:spec_decode_num_drafts_total": 3183.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 4853.0
},
"accepted_per_position": {
"0": 2665.0,
"1": 2188.0
},
"accepted_fraction": 0.7623311341501728,
"committed_emitted_tokens": 8037.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-before.txt": "f25ea1303c7b1b799dc39b451ba96372cf5f5ce3f3e95dcce6cb298949c411ed",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-after.txt": "6b4fc8665875fcfcb7c367f42504561e046ac6fbe9e7a66f99ec7e5e02bfe9c3"
}
},
"fixed2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 1094.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_draft_tokens_total": 1810.0,
"vllm:spec_decode_num_drafts_total": 905.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1094.0
},
"accepted_per_position": {
"0": 661.0,
"1": 433.0
},
"accepted_fraction": 0.6044198895027625,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-before.txt": "1d4560681bb3859f5d6973269d741310351b402726225fe1a8ddc261774ec795",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-after.txt": "aa43719aabaf5b07691d948f97c452bc201e4bc888f252f8c02c5bdcf8b020a9"
}
},
"fixed10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 6117.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6117.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_drafts_total": 3884.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 7768.0
},
"accepted_per_position": {
"0": 3319.0,
"1": 2798.0
},
"accepted_fraction": 0.7874613800205973,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-before.txt": "1ed44508b3ea244ca341d97f5dd473da7972d10374032b9f9af3342b91df3982",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-after.txt": "5073090e5bc7f524f85a7bff408b5058b951839f7491f1d4b6873525e0a5aa84"
}
}
},
"MTP_accounting_limit": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"source_evidence_commit": "0c0751c8b79fc57442317af1898946cc4f4cfec8",
"comparison_limit": "Opaque native selector repairs the free-s47 compiler failure; it forwards to the unchanged ordered-FP32 U operator for M<=32 and original safe reconstruction for larger batches. New dispatch flag1 and mapped DSOdf0df1a9; native U45281e2c and safef52f2e8b unchanged. | Native MTP2 target and draft, ordinary decoder/collectives and 800 attention fallback differ from historical non-MTP U profile. No isolated MTP or compiler-speed attribution. | Fresh private rootSSD caches, FP16 activations/attentionKV, memory utilization0.95, context131072, greedy concurrency1 and original requests/scorer retained. | One sample per tier; no repeatability or final qualification. | Speed numerator uses committed outputs, not draft proposals; acceptance counts precede stop trimming. | Static256 opaque routes plus loaded/mapped libraries are not dynamic per-kernel trace; MTP counters show actual speculative work. | New native opaque selector bypasses compiler symbol defect without changing U/safe numerical operators; MTP uses800 attention fallback and fresh private rootSSD caches. Historical W8/non-MTP cache/KV/backend differences prevent isolated causal attribution. | Logical packed extents and KV views are not summed into physical memory; remaining FP16 tensor bytes not fully itemized. | Earlier initialization NameError retained; inactive failed-tail unused because strict passed.",
"recorded_configuration_details": {
"source": "e4e131d2aa3424f1fb968a979a1e34bf5c510a16",
"image": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
"service_manifest_sha256": "c174581c0c18d46646363cc42d8e658ce1aa756746c803e4920aedce0ee737b6",
"model_repository": "Qwen/Qwen3.8-27B",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"strict_valid": true,
"strict_tokens": 8037,
"strict_diagnostic_TPS": 27.504710310538147,
"official_fixed2K_TPS": 32.930074165374116,
"diagnostic_fixed10K_TPS": 25.672148198446543,
"metrics": {
"strict": {
"completion_tokens": 8037,
"finish_reason": "stop",
"elapsed_seconds": 293.35235242999624,
"vllm_prefill_seconds_delta": 1.1333177799824625,
"vllm_decode_seconds_delta": 292.2044954940211,
"vllm_decode_tps": 27.504710310538147,
"vllm_generation_tokens_delta": 8037.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "d7a1b04f9328629cb64dc6acf53a90c6048065daecaf10ae16036c7e82fa778e",
"server_e2e_seconds": 293.3473253250122,
"MTP": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 6366.0,
"vllm:spec_decode_num_accepted_tokens_total": 4853.0,
"vllm:request_generation_tokens_sum": 8037.0,
"vllm:generation_tokens_total": 8037.0,
"vllm:spec_decode_num_drafts_total": 3183.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 4853.0
},
"accepted_per_position": {
"0": 2665.0,
"1": 2188.0
},
"accepted_fraction": 0.7623311341501728,
"committed_emitted_tokens": 8037.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-before.txt": "f25ea1303c7b1b799dc39b451ba96372cf5f5ce3f3e95dcce6cb298949c411ed",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-after.txt": "6b4fc8665875fcfcb7c367f42504561e046ac6fbe9e7a66f99ec7e5e02bfe9c3"
}
}
},
"fixed2K": {
"completion_tokens": 2000,
"finish_reason": "length",
"elapsed_seconds": 61.88037476199679,
"vllm_prefill_seconds_delta": 1.1334461470250972,
"vllm_decode_seconds_delta": 60.73475540796062,
"vllm_decode_tps": 32.930074165374116,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "96e2767d5adb74a3c90c5198ea21f68211501710b8811fd1515d243c4eab27a5",
"server_e2e_seconds": 61.87655425071716,
"MTP": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 1094.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_draft_tokens_total": 1810.0,
"vllm:spec_decode_num_drafts_total": 905.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1094.0
},
"accepted_per_position": {
"0": 661.0,
"1": 433.0
},
"accepted_fraction": 0.6044198895027625,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-before.txt": "1d4560681bb3859f5d6973269d741310351b402726225fe1a8ddc261774ec795",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-after.txt": "aa43719aabaf5b07691d948f97c452bc201e4bc888f252f8c02c5bdcf8b020a9"
}
}
},
"fixed10K": {
"completion_tokens": 10000,
"finish_reason": "length",
"elapsed_seconds": 390.68672217201674,
"vllm_prefill_seconds_delta": 1.1481118430383503,
"vllm_decode_seconds_delta": 389.5271997769596,
"vllm_decode_tps": 25.672148198446543,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "b63985089a787c260e30876d287e402a34a6373bf71bc3556f271b2799b61c22",
"server_e2e_seconds": 390.6815514564514,
"MTP": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 6117.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6117.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_drafts_total": 3884.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 7768.0
},
"accepted_per_position": {
"0": 3319.0,
"1": 2798.0
},
"accepted_fraction": 0.7874613800205973,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-before.txt": "1ed44508b3ea244ca341d97f5dd473da7972d10374032b9f9af3342b91df3982",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-after.txt": "5073090e5bc7f524f85a7bff408b5058b951839f7491f1d4b6873525e0a5aa84"
}
}
}
},
"weight_dtype": "symmetric RTN W8 in GPTQ-v2 layout; group128, ties-to-even, zero128, FP16 scales",
"activation_dtype": "FP16",
"attention_KV_dtype": "FP16",
"TP": 4,
"MTP_tokens": 2,
"MTP_model_class": "vllm.model_executor.models.qwen3_5_mtp.Qwen3_5MTP",
"MTP_precision": "Five draft projections use the same W8/group128 RTN scheme; remaining draft parameters FP16, embedding/head shared with target. Same official checkpoint revision.",
"quantized_target_modules_per_rank": 256,
"quantized_draft_modules_per_rank": 5,
"packed_component_bytes_per_rank": {
"target": {
"qweight": 6087639040,
"qzeros": 47559680,
"scales": 95119360,
"g_idx": 0
},
"draft": {
"qweight": 106168320,
"qzeros": 829440,
"scales": 1658880,
"g_idx": 0
}
},
"packed_target_bytes_per_rank": 6230318080,
"packed_draft_bytes_per_rank": 108656640,
"packed_weight_bytes_per_rank": 6338974720,
"sampled_peak_VRAM_bytes": {
"0000:16:00.0": 30507261952,
"0000:19:00.0": 30521655296,
"0000:1c:00.0": 30521917440,
"0000:1f:00.0": 30471643136
},
"loaded_allocator_and_KV_per_rank": [
{
"rank": 0,
"load_allocated": 8315061248,
"load_reserved": 8856272896,
"KV_allocated": 28207806976,
"KV_reserved": 28718399488,
"KV_shapes": [
[
1425,
2,
800,
1,
256
]
]
},
{
"rank": 1,
"load_allocated": 8315168256,
"load_reserved": 8856272896,
"KV_allocated": 28207913984,
"KV_reserved": 28718399488,
"KV_shapes": [
[
1425,
2,
800,
1,
256
]
]
},
{
"rank": 2,
"load_allocated": 8315156992,
"load_reserved": 8856272896,
"KV_allocated": 28207902720,
"KV_reserved": 28718399488,
"KV_shapes": [
[
1425,
2,
800,
1,
256
]
]
},
{
"rank": 3,
"load_allocated": 8316479488,
"load_reserved": 8856272896,
"KV_allocated": 28209225216,
"KV_reserved": 28718399488,
"KV_shapes": [
[
1425,
2,
800,
1,
256
]
]
}
],
"KV_capacity": {
"available_GiB": 18.49,
"reported_grouped_tokens": 284800,
"reported_concurrency_at131072": 8.24,
"target_views": 16,
"draft_views": 1,
"view_shape": [
1425,
2,
800,
1,
256
],
"view_bytes": 1167360000
},
"configured_context": 131072,
"complete_ladders": 1,
"sample_count": 1,
"repeatability_established": false,
"FP16_release_qualification": false,
"archive": {
"path": "/home/ai/vnext-v620-qwen38-rtn8-tp4-mtp2-service-r1-20261023/raw-evidence.tar.gz",
"bytes": 1169597,
"sha256": "7d1d299f8a809f68f5edf5c4a676fa226f28adb7a6639c80afcfec33169e357c",
"files": 251,
"index_sha256": "ade3ca824e03c977660ec0352a4ef2afa3ae2077295f2749bac38fd0203b2f84"
},
"local_primary_record": ".work/fp16-w8-tp-mtp-matrix/w8-tp4-mtp2-service-r1/sealed-mirror/retained/evidence/measurement-closure.json",
"review": {
"path": ".work/fp16-w8-tp-mtp-matrix/reviews/W8-TP4-MTP2-r1-final-cell-review.json",
"sha256": "fc8198cb6bad9e8c493b850a9742f062a4116bcef3d86daa8f4afdcd44541582"
},
"closure": {
"container_id": "5173931e9d3991c31c191cf6c40ad967ab14b9e8cb2ed4e7ac81df72cb9091c1",
"image_id": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
"exit_code": 0,
"OOMKilled": false,
"restart_count": 0,
"bank_idle": true,
"devices": [
{
"pci": "0000:16:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
},
{
"pci": "0000:19:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
},
{
"pci": "0000:1c:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
},
{
"pci": "0000:1f:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
}
],
"started_at": "2026-10-23T15:08:09.043168117Z",
"finished_at": "2026-10-23T15:54:05.354754912Z",
"closed_at": "2026-10-23T15:54:06.522428+00:00",
"kernel_fault_matches": [],
"startup_requests_completed_and_drained": 3,
"official_phase_samples": 11,
"complete_ladders": 1,
"strict_valid": true,
"official_fixed_2000_tps": 32.930074165374116,
"qualifying_ladders": 0,
"FP16_qualification": false,
"baseline_kind": "RTN8"
},
"final_counters": {
"requests": 14,
"prompt_tokens": 6622,
"committed_generation_tokens": 42037
},
"runtime_differences": [
"Opaque native selector repairs the free-s47 compiler failure; it forwards to the unchanged ordered-FP32 U operator for M<=32 and original safe reconstruction for larger batches. New dispatch flag1 and mapped DSOdf0df1a9; native U45281e2c and safef52f2e8b unchanged.",
"Native MTP2 target and draft, ordinary decoder/collectives and 800 attention fallback differ from historical non-MTP U profile. No isolated MTP or compiler-speed attribution.",
"Fresh private rootSSD caches, FP16 activations/attentionKV, memory utilization0.95, context131072, greedy concurrency1 and original requests/scorer retained."
],
"limits": [
"One sample per tier; no repeatability or final qualification.",
"Speed numerator uses committed outputs, not draft proposals; acceptance counts precede stop trimming.",
"Static256 opaque routes plus loaded/mapped libraries are not dynamic per-kernel trace; MTP counters show actual speculative work.",
"New native opaque selector bypasses compiler symbol defect without changing U/safe numerical operators; MTP uses800 attention fallback and fresh private rootSSD caches. Historical W8/non-MTP cache/KV/backend differences prevent isolated causal attribution.",
"Logical packed extents and KV views are not summed into physical memory; remaining FP16 tensor bytes not fully itemized.",
"Earlier initialization NameError retained; inactive failed-tail unused because strict passed."
]
},
"MTP_2K_draft_batches": 905.0,
"MTP_2K_proposed": 1810.0,
"MTP_2K_accepted": 1094.0,
"MTP_2K_acceptance_percent": 60.44198895027625,
"MTP_2K_committed": 2000.0,
"detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
"detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
"detail_source_pointer": "FP16_W8_TP_MTP_matrix.cells[4]",
"retest_strict_tokens": 10158,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:generation_tokens_total": 10158.0,
"vllm:spec_decode_num_draft_tokens_total": 8090.0,
"vllm:spec_decode_num_accepted_tokens_total": 6113.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:request_generation_tokens_sum": 10158.0,
"vllm:spec_decode_num_drafts_total": 4045.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6113.0
},
"accepted_per_position": {
"0": 3356.0,
"1": 2757.0
},
"accepted_fraction": 0.7556242274412855,
"committed_emitted_tokens": 10158.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "b0c7fc7c40a28cc81666d74d656deded0faa1edf58d638d792732d22f635a0a3",
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "4266e9362ae653507a568738210298328ae118dae456893caf38eac6b5e259c2"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:generation_tokens_total": 2000.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 1758.0,
"vllm:spec_decode_num_accepted_tokens_total": 1122.0,
"vllm:spec_decode_num_drafts_total": 879.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1122.0,
"vllm:request_generation_tokens_sum": 2000.0
},
"accepted_per_position": {
"0": 652.0,
"1": 470.0
},
"accepted_fraction": 0.6382252559726962,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "b96c35f259b8cd18c589eb921758bf9ce6b7e775968528fc12d8e68656e4d816",
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "46b1bbec57fb4711473fc1074c580eeb32642138f4dda7915b8a8b898d7b7b55"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 7984.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_total": 6007.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_drafts_total": 3992.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6007.0
},
"accepted_per_position": {
"0": 3303.0,
"1": 2704.0
},
"accepted_fraction": 0.7523797595190381,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "f1cd06ff0d404e30521e2e57fd27914f1d1f3c059744359565eaba6cf8eac279",
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "6393ab308a1f1c437c10bdb67e948cc15e6d647d1a94cc644bb7a5cc99b3840f"
}
}
},
"retest_MTP_2K_accepted": 1122.0,
"retest_MTP_2K_proposed": 1758.0,
"retest_MTP_2K_draft_batches": 879.0,
"retest_MTP_2K_acceptance_percent": 63.82252559726962,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30542385152,
"1": 30557487104,
"2": 30557450240,
"3": 30507130880
},
"retest_memory_observation_count": 46,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-w8-tp4-mtp2/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 18.53,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 18.53,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 284800,
"retest_memory_log_report": {
"profile": "v620-w8-tp4-mtp2",
"image": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
"container": "5457bd41fa75f10e8266c62e7d825106f9d861c3a122df35c891d462aff0ef09",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 8.25,
"retest_logged_model_load_seconds": 69.672332,
"retest_logged_available_KV_GiB": 18.53,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 18.53,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 284800,
"retest_logged_context_capacity_x": 8.25,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w8-tp4-mtp2/service-final.log",
"source_log_sha256": "196a6879d46ceafafa4327fa2e08df4d40d490907a2174e6abe1f3d38e50358f",
"sealed_index_sha256": "02fc37701b599060b58b02a78ac7165d21c59bda1893085577c5b7228fd3431e",
"source_lines": {
"model_load": {
"line": 154,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 19:44:59 [gpu_model_runner.py:4598] Model loading took 8.25 GiB memory and 69.672332 seconds"
},
"automatic_KV_budget": {
"line": 225,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 20:02:31 [gpu_worker.py:453] Available KV cache memory: 18.53 GiB"
},
"grouped_KV_tokens": {
"line": 229,
"text": "(EngineCore_DP0 pid=85) INFO 10-26 20:02:31 [kv_cache_utils.py:1314] GPU KV cache size: 284,800 tokens"
},
"context_capacity": {
"line": 230,
"text": "(EngineCore_DP0 pid=85) INFO 10-26 20:02:31 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 8.25x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w8-tp4-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.458877563476562,
"sampled_peak_VRAM_GiB": 28.425750732421875
} |
| V620Current Qwen3.8 | 8-bit RTNQwen/Qwen3.8-27B | TP2MTP22K accepted 1170 / 1658 (70.57%) | PASS4872 tokens | 25.998 | 21.719 | 0.802 / 77.74610K: 0.799 / 461.243Strict: 0.802 / 205.310 | 11.81 / 26.68packed target/draft projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:e3ec7a663ee96460173b72b3be0fb6e5f4c77ab7a32c93b4824d60dcead4bde1Runtime image config: sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256 mkdir -p launch-tools-v620-serve-r5 tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp2-mtp2 GPU_RENDER_NODES=renderD128,renderD129 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w8-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Preflight cleanup compatibilityAfter its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged. Retained lab Docker command · not portabledocker run -d \
--name serve-v620-w8-tp2-mtp2-20261028 \
--network host \
--ipc host \
--cpuset-cpus 0-7,16-23 \
--cpuset-mems 0 \
--security-opt label=disable \
--shm-size 67108864 \
--device /dev/kfd:/dev/kfd:rwm \
--device /dev/dri/card5:/dev/dri/card5:rwm \
--device /dev/dri/card6:/dev/dri/card6:rwm \
--device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
--device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
--user root \
--workdir /tmp \
-e VNEXT_RTN_W8_NATIVE_DISPATCH=1 \
-e VNEXT_RTN_W8_FUSED_CTA=1 \
-e VLLM_DISABLE_COMPILE_CACHE=1 \
-e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
-e ROCM_VERSION=7.14.0 \
-e TORCH_VERSION=2.12.0+rocm7.14.0 \
-e VLLM_TARGET_DEVICE=rocm \
-e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
-e PYTORCH_ROCM_ARCH=gfx1030 \
-e GPU_ARCHS=gfx1030 \
-e AMDGPU_TARGETS=gfx1030 \
-e CMAKE_HIP_ARCHITECTURES=gfx1030 \
-e PYTHONDONTWRITEBYTECODE=1 \
-e PIP_NO_CACHE_DIR=1 \
-e PIP_DISABLE_PIP_VERSION_CHECK=1 \
-e VNEXT_GDN_HEAD32=1 \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp2/cache/.triton,dst=/root/.triton \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
--mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp2/cache/vllm,dst=/root/.cache/vllm \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w8-tp2-mtp2/cache/torch_extensions,dst=/root/.cache/torch_extensions \
--entrypoint /usr/bin/env \
sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567 \
V756F94_BOUNDARY_MODE=posted \
DO_NOT_TRACK=1 \
GPU_DEVICE_ORDINAL=0,1 \
GPU_MAX_HW_QUEUES=1 \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
OMP_NUM_THREADS=4 \
TEMP=/tmp/torchinductor_root \
TMPDIR=/tmp/torchinductor_root \
TMP=/tmp/torchinductor_root \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
TRITON_CACHE_DIR=/root/.triton \
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
VLLM_GPU_MEMORY_UTILIZATION=0.95 \
VLLM_LOGGING_LEVEL=INFO \
VLLM_PORT=9002 \
VLLM_RDNA_FORCE_FP16=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
NCCL_P2P_DISABLE=0 \
NCCL_ALGO=Tree \
NCCL_PROTO=LL \
NCCL_MIN_NCHANNELS=4 \
NCCL_MAX_NCHANNELS=4 \
NCCL_NTHREADS=256 \
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
HIP_VISIBLE_DEVICES=0,1 \
CUDA_VISIBLE_DEVICES=0,1 \
ROCR_VISIBLE_DEVICES=0,1 \
VLLM_CACHE_ROOT=/root/.cache/vllm \
PYTORCH_ROCM_ARCH=gfx1030 \
PYTHONHASHSEED=0 \
/opt/venv/bin/python \
-m \
vllm.vnext_rtn_entry \
serve \
/opt/local-models/qwen38-27b-hf \
--served-model-name \
Qwen/Qwen3.8-27B \
--enable-auto-tool-choice \
--tool-call-parser \
qwen3_coder \
--dtype \
half \
--host \
0.0.0.0 \
--port \
9002 \
--tensor-parallel-size \
2 \
--speculative-config \
'{"method":"mtp","num_speculative_tokens":2}' \
--max-model-len \
131072 \
--gpu-memory-utilization \
0.95 \
--trust-remote-code \
--generation-config \
vllm \
-O=3 \
--reasoning-parser \
qwen3 \
--async-scheduling \
--hf-overrides \
'{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":8,"group_size":128}}' \
--language-model-only \
--max-num-seqs \
2 \
--max-num-batched-tokens \
2048 \
--mamba-ssm-cache-dtype \
float32 \
--no-enable-prefix-caching \
--no-enforce-eager \
--attention-backend \
TRITON_ATTN \
--disable-custom-all-reduce \
--seed \
0 \
--worker-cls \
vllm.vnext_rtn_worker.Worker \
--quantization \
vnext_rtnHistorical host paths and device addresses are retained as evidence. Use the serving command above.Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-r4.tar.gz.sha256 mkdir -p launch-tools-v620-r4 tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w8-tp2-mtp2 GPU_RENDER_NODES=renderD128,renderD129 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-v620-r4/portable/launch.py --config v620-w8-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-v620-r4.tar.gz · SHA-256 launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. Portable observer compatibilityThe evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged. | All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 24.072 · eligible10K 20.016 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 7027,
"content_chars": 6647,
"content_has_think_close": false,
"elapsed_seconds": 315.2030647529755,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T00:56:43Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.011896442156285048,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 6647,
"qwen_answer_text_sha256": "26a2ea768dce161526e7577cced0a08108a8d9b313c768dedbaca547688d8e5b",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 6647,
"qwen_visible_text_sha256": "26a2ea768dce161526e7577cced0a08108a8d9b313c768dedbaca547688d8e5b",
"raw_has_think_close": false,
"reasoning_chars": 27053,
"text_chars": 33701,
"text_sha256": "6fa8e4faec13968960681e2d41dd83052329313155b407a52246c75b6b809df4",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 314.38775441702455,
"vllm_decode_tps": 22.35137947096669,
"vllm_generation_tokens_delta": 7027.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.7998496741056442,
"vllm_prefill_tps": 591.361121111773,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 22.293564961073766
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 83.8934364290908,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T00:58:07Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.009258593898266554,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 8929,
"text_chars": 8929,
"text_sha256": "e1c5e4a62e7f1e164f56ca453844bfff8198bd48bce270b29a39a7e864e2b3c4",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 83.08291288209148,
"vllm_decode_tps": 24.07233847034629,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.7974292829167098,
"vllm_prefill_tps": 593.1560454739459,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 23.839767270595225
},
"10K": {
"completion_tokens": 10000,
"content_chars": 22460,
"content_has_think_close": false,
"elapsed_seconds": 500.42501594801433,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T01:06:28Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.01606009411625564,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 22460,
"qwen_answer_text_sha256": "31fd810e67d65a343e3a49f3e86e854fc1beadaed3b0984180ad70a959a64dc4",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 22460,
"qwen_visible_text_sha256": "31fd810e67d65a343e3a49f3e86e854fc1beadaed3b0984180ad70a959a64dc4",
"raw_has_think_close": false,
"reasoning_chars": 27053,
"text_chars": 49514,
"text_sha256": "56f5602a23640359af1fc4bb9386a4268a92fa54668d3693b5a01b1f3a8098af",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 499.6082297200337,
"vllm_decode_tps": 20.01568309954325,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8002281670924276,
"vllm_prefill_tps": 591.081418339237,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 19.98301380089046
}
}Earlier exported-image retest · separate evidenceRetest completed; strict failed; fixed speeds diagnostic onlyVerification · Raw retestMTP counter replayFAIL14,095 natural tokens2K 28.828*10K 22.897* · diagnostic2K MTP accepted 1252 / 1494 (83.80%) Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T20:33:03.042335455Z",
"FinishedAt": "2026-10-26T21:33:17.506571757Z"
},
"devices_after": [
{
"card": 5,
"render": 133,
"pci": "0000:16:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:19:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
}
],
"finished": "2026-10-26T21:33:17.649653+00:00"
},
"retest_strict": "FAIL",
"retest_official_2K_eligible": false,
"retest_2K_TPS": 28.827830303864896,
"retest_10K_TPS": 22.897475815790525,
"retest_strict_tokens": 14095,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 14095,
"elapsed_seconds": 697.7960248619784,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 696.983301272965,
"vllm_decode_tps": 20.222866135037954,
"vllm_generation_tokens_delta": 14095.0,
"vllm_prefill_seconds_delta": 0.801551835029386,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 70.18575884401798,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 69.37740297894925,
"vllm_decode_tps": 28.827830303864896,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 0.797065292019397,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 437.54100987606216,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 436.72936180606484,
"vllm_decode_tps": 22.897475815790525,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 0.7997282180003822,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 0.797065292019397,
"retest_2K_total_seconds": 70.18575884401798,
"retest_10K_prefill_seconds": 0.7997282180003822,
"retest_10K_total_seconds": 437.54100987606216,
"retest_strict_prefill_seconds": 0.801551835029386,
"retest_strict_total_seconds": 697.7960248619784,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 9314.0,
"vllm:spec_decode_num_drafts_total": 4780.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 14095.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 9314.0,
"vllm:request_generation_tokens_sum": 14095.0,
"vllm:spec_decode_num_draft_tokens_total": 9560.0
},
"accepted_per_position": {
"0": 4699.0,
"1": 4615.0
},
"accepted_fraction": 0.9742677824267783,
"committed_emitted_tokens": 14095.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "d499da222db5f4c902370bd706b3037ebd84b7cd1c3ca91296c6f0641cfb1dcf",
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "4c68d0bfce8174357d766b4c218392da6257632f05d570487984b9c97f56b6dc"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_draft_tokens_total": 1494.0,
"vllm:spec_decode_num_accepted_tokens_total": 1252.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1252.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:spec_decode_num_drafts_total": 747.0,
"vllm:request_generation_tokens_count": 1.0
},
"accepted_per_position": {
"0": 667.0,
"1": 585.0
},
"accepted_fraction": 0.8380187416331994,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "f7462730d6f10a811663c5107c3584869ce2f10c7a1ace0726aeb09a783c0275",
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "f9eefe90a5ed4ea8030f25ec580474c5d1f9ee6238d18cc2d27bab36615688af"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_draft_tokens_total": 6830.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6586.0,
"vllm:spec_decode_num_drafts_total": 3415.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_total": 6586.0,
"vllm:generation_tokens_total": 10000.0
},
"accepted_per_position": {
"0": 3335.0,
"1": 3251.0
},
"accepted_fraction": 0.9642752562225476,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "3d9f3d802948b3286135fb227b09e5ae44fa5ee621e625f95dc6f2df71691291",
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "e6abdb87a069a4d72731346e46434237176a71f431b126902abadb04ba82bd7b"
}
}
},
"retest_MTP_2K_accepted": 1252.0,
"retest_MTP_2K_proposed": 1494.0,
"retest_MTP_2K_draft_batches": 747.0,
"retest_MTP_2K_acceptance_percent": 83.80187416331995,
"retest_loaded_admission": "Complete record: chart-details/row-19-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 1, \"fixed\": 1, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
"retest_diagnostic_counter_audit": "retest-evidence/v620-w8-tp2-mtp2/local-diagnostic-fixed-audit.json",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 28642738176,
"1": 28642762752
},
"retest_memory_observation_count": 58,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-w8-tp2-mtp2/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 16.56,
"retest_logged_model_load_seconds": 123.006295,
"retest_logged_available_KV_GiB": 9.6,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 9.6,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 73600,
"retest_logged_context_capacity_x": 2.14,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "v620-w8-tp2-mtp2",
"image": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
"container": "21a9828f12b1d34b82e2d0a87d86fb0d50ba8af66ac161c5ef4070a0e8424ed6",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 16.56,
"retest_logged_model_load_seconds": 123.006295,
"retest_logged_available_KV_GiB": 9.6,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 9.6,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 73600,
"retest_logged_context_capacity_x": 2.14,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w8-tp2-mtp2/service-final.log",
"source_log_sha256": "55ab409a441ae421f1d1b05f492f45e5125d690e7dcbd2140e32c403b07075c4",
"sealed_index_sha256": "80034e3ab771353bf7a27f0c11c3a059a53363a207eb153ed442827132adb6d0",
"source_lines": {
"model_load": {
"line": 118,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 20:37:10 [gpu_model_runner.py:4598] Model loading took 16.56 GiB memory and 123.006295 seconds"
},
"automatic_KV_budget": {
"line": 172,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 20:57:42 [gpu_worker.py:453] Available KV cache memory: 9.6 GiB"
},
"grouped_KV_tokens": {
"line": 176,
"text": "(EngineCore_DP0 pid=85) INFO 10-26 20:57:42 [kv_cache_utils.py:1314] GPU KV cache size: 73,600 tokens"
},
"context_capacity": {
"line": 177,
"text": "(EngineCore_DP0 pid=85) INFO 10-26 20:57:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 2.14x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w8-tp2-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 26.67565155029297
}Complete retest loaded admission | Evidence and limitsCompleted, independently reviewed W8 TP2 native MTP2 full ladder; strict PASS 4872. One eligible sample, no promotion or FP16 qualification. {
"strict_tokens": 4872,
"source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.FP16_W8_TP_MTP_matrix.cells[5].result",
"source_revision": "e4e131d2aa3424f1fb968a979a1e34bf5c510a16",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"attention_KV_dtype": "FP16",
"MTP_model_class": "vllm.model_executor.models.qwen3_5_mtp.Qwen3_5MTP",
"MTP_precision": "Five draft projections use the same W8/group128 RTN scheme; remaining draft parameters FP16, embedding/head shared with target. Same official checkpoint revision.",
"memory_accounting_limit": "Packed totals here include separately itemized target and draft projections, excluding retained FP16 parameters/shared embeddings/head. The 16 target plus one draft logical KV views are not a disjoint physical allocation sum. Sampled VRAM, allocator counters and runtime capacity are separate.",
"load_and_KV_allocator_bytes_per_rank": "[{\"KV_allocated\": 25826995200, \"KV_reserved\": 26908557312, \"KV_shapes\": [[370, 2, 800, 2, 256]], \"load_allocated\": 15485343232, \"load_reserved\": 17783848960, \"rank\": 0}, {\"KV_allocated\": 25826995200, \"KV_reserved\": 26908557312, \"KV_shapes\": [[370, 2, 800, 2, 256]], \"load_allocated\": 15485343232, \"load_reserved\": 17783848960, \"rank\": 1}]",
"sampled_peak_VRAM_bytes_by_PCI": "{\"0000:16:00.0\": 28642746368, \"0000:19:00.0\": 28642762752}",
"KV_capacity": "{\"available_GiB\": 9.6, \"draft_views\": 1, \"reported_concurrency_at131072\": 2.14, \"reported_grouped_tokens\": 73600, \"target_views\": 16, \"view_bytes\": 606208000, \"view_shape\": [370, 2, 800, 2, 256]}",
"attention_KV_logical_view_shape": "[370, 2, 800, 2, 256]",
"MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 3910.0,
"vllm:spec_decode_num_drafts_total": 1955.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:request_generation_tokens_sum": 4872.0,
"vllm:generation_tokens_total": 4872.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 2917.0,
"vllm:spec_decode_num_accepted_tokens_total": 2917.0
},
"accepted_per_position": {
"0": 1616.0,
"1": 1301.0
},
"accepted_fraction": 0.7460358056265984,
"committed_emitted_tokens": 4872.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-before.txt": "dc381957e058f7374c01740b893fe43f123756b0c1873e362382cffbab159d63",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-after.txt": "913477fb591e79e28b0263d22984402724da7441a2c1975e165ae031e8996497"
}
},
"fixed2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1170.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_draft_tokens_total": 1658.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_drafts_total": 829.0,
"vllm:spec_decode_num_accepted_tokens_total": 1170.0,
"vllm:generation_tokens_total": 2000.0
},
"accepted_per_position": {
"0": 663.0,
"1": 507.0
},
"accepted_fraction": 0.7056694813027744,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-before.txt": "8b04539b329762ccbad207878cfcc4169eb25bbc24df72877b7b711f86a03b74",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-after.txt": "7581278c5eb0aa1bf5eb578e68dc608ceb85ceb1717f0e591cb134ef5a25bc47"
}
},
"fixed10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_drafts_total": 3667.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:spec_decode_num_draft_tokens_total": 7334.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6334.0,
"vllm:spec_decode_num_accepted_tokens_total": 6334.0,
"vllm:request_generation_tokens_count": 1.0
},
"accepted_per_position": {
"0": 3325.0,
"1": 3009.0
},
"accepted_fraction": 0.8636487592037088,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-before.txt": "cb8714611dbc685931a96f11cf30558a7c0c561a159cfbe9db8d4ae031a792c8",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-after.txt": "a849896389c8c7c5d65d4f8b32a0ef56b02204f365bfbe3803d241f5c8eff407"
}
}
},
"MTP_accounting_limit": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"source_evidence_commit": "1fdf5463420716858c50f6804a9105df16bacc07",
"comparison_limit": "Opaque native selector repairs the free-s47 compiler failure; it forwards to the unchanged ordered-FP32 U operator for M<=32 and original safe reconstruction for larger batches. New dispatch flag1 and mapped DSOdf0df1a9; native U45281e2c and safef52f2e8b unchanged. | Native MTP2 target and draft, ordinary decoder/collectives and 800 attention fallback differ from historical non-MTP U profile. Same eaf8 image as measured TP4 MTP2; TP2 reshards and KV budget differ. No isolated MTP or compiler-speed attribution. | Fresh private rootSSD caches, FP16 activations/attentionKV, memory utilization0.95, context131072, greedy concurrency1 and original requests/scorer retained. | Existing MTP-scoped post-sharing cache cleanup runs for TP2; the non-MTP TP2 path does not take that branch. Observed allocator reserve and KV budget differ; this is not an isolated MTP gain. | One sample per tier; no repeatability or final qualification. | Committed output only supplies numerator; speculative acceptance precedes stop trimming. | Static256 opaque calls and library mappings do not establish dynamic per-kernel trace; MTP counters show actual speculation. | Same eaf8 image/e4e131d source as TP4 MTP; TP reshards and KV budget differ, native MTP800 attention fallback/fresh private rootSSD caches materially differ from historical nonMTP. | Logical packs/views are not unique whole-model physical memory; remaining FP16 exclusions not fully itemized. | Inactive failed-tail unused because strict passed. | Reported grouped KV tokens and concurrency are runtime capacity indicators, not an independently tested exact request ceiling or demonstrated full-context residency. | Original launch observer ended143 without its SSH child exit receipt. Separate exact-container/owned-waiter verification confirms stopped/drained; no claim that all parents exited0.",
"recorded_configuration_details": {
"source": "e4e131d2aa3424f1fb968a979a1e34bf5c510a16",
"image": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
"service_manifest_sha256": "c174581c0c18d46646363cc42d8e658ce1aa756746c803e4920aedce0ee737b6",
"model_repository": "Qwen/Qwen3.8-27B",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"strict_valid": true,
"strict_tokens": 4872,
"strict_diagnostic_TPS": 23.824685399012747,
"official_fixed2K_TPS": 25.99758548499443,
"diagnostic_fixed10K_TPS": 21.71884469539952,
"metrics": {
"strict": {
"completion_tokens": 4872,
"finish_reason": "stop",
"elapsed_seconds": 205.30964287399547,
"vllm_prefill_seconds_delta": 0.8021039759623818,
"vllm_decode_seconds_delta": 204.49378106801305,
"vllm_decode_tps": 23.824685399012747,
"vllm_generation_tokens_delta": 4872.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "f4c9bd7fda89ab399c5a21af2f247e4dccb8abac2eaa82b175150daacf1b474b",
"server_e2e_seconds": 205.30365681648254,
"MTP": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 3910.0,
"vllm:spec_decode_num_drafts_total": 1955.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:request_generation_tokens_sum": 4872.0,
"vllm:generation_tokens_total": 4872.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 2917.0,
"vllm:spec_decode_num_accepted_tokens_total": 2917.0
},
"accepted_per_position": {
"0": 1616.0,
"1": 1301.0
},
"accepted_fraction": 0.7460358056265984,
"committed_emitted_tokens": 4872.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-before.txt": "dc381957e058f7374c01740b893fe43f123756b0c1873e362382cffbab159d63",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_128_strict_proxy/metrics-after.txt": "913477fb591e79e28b0263d22984402724da7441a2c1975e165ae031e8996497"
}
}
},
"fixed2K": {
"completion_tokens": 2000,
"finish_reason": "length",
"elapsed_seconds": 77.74615426099626,
"vllm_prefill_seconds_delta": 0.802028578997124,
"vllm_decode_seconds_delta": 76.93022112204926,
"vllm_decode_tps": 25.99758548499443,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "a06c4f4ef2f7bd568ef38d66f317106245a09492ee0e9e8359460231252df5e9",
"server_e2e_seconds": 77.74068355560303,
"MTP": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1170.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_draft_tokens_total": 1658.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_drafts_total": 829.0,
"vllm:spec_decode_num_accepted_tokens_total": 1170.0,
"vllm:generation_tokens_total": 2000.0
},
"accepted_per_position": {
"0": 663.0,
"1": 507.0
},
"accepted_fraction": 0.7056694813027744,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-before.txt": "8b04539b329762ccbad207878cfcc4169eb25bbc24df72877b7b711f86a03b74",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_2000_proxy/metrics-after.txt": "7581278c5eb0aa1bf5eb578e68dc608ceb85ceb1717f0e591cb134ef5a25bc47"
}
}
},
"fixed10K": {
"completion_tokens": 10000,
"finish_reason": "length",
"elapsed_seconds": 461.2426872580545,
"vllm_prefill_seconds_delta": 0.7994827929651365,
"vllm_decode_seconds_delta": 460.42964716802817,
"vllm_decode_tps": 21.71884469539952,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prompt_tokens_delta": 473.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"summary_sha256": "5e067ccaed002e188f86f7f65df7542574524851c2ca1a5e9905cb5af524a7ed",
"server_e2e_seconds": 461.23744559288025,
"MTP": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_drafts_total": 3667.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:spec_decode_num_draft_tokens_total": 7334.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6334.0,
"vllm:spec_decode_num_accepted_tokens_total": 6334.0,
"vllm:request_generation_tokens_count": 1.0
},
"accepted_per_position": {
"0": 3325.0,
"1": 3009.0
},
"accepted_fraction": 0.8636487592037088,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-before.txt": "cb8714611dbc685931a96f11cf30558a7c0c561a159cfbe9db8d4ae031a792c8",
"/home/joe/PhoneSync/Documents/gfx1030/.work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/ladder1/c1_10000_proxy/metrics-after.txt": "a849896389c8c7c5d65d4f8b32a0ef56b02204f365bfbe3803d241f5c8eff407"
}
}
}
},
"weight_dtype": "symmetric RTN W8 in GPTQ-v2 layout; group128, ties-to-even, zero128, FP16 scales",
"activation_dtype": "FP16",
"attention_KV_dtype": "FP16",
"TP": 2,
"MTP_tokens": 2,
"MTP_model_class": "vllm.model_executor.models.qwen3_5_mtp.Qwen3_5MTP",
"MTP_precision": "Five draft projections use the same W8/group128 RTN scheme; remaining draft parameters FP16, embedding/head shared with target. Same official checkpoint revision.",
"quantized_target_modules_per_rank": 256,
"quantized_draft_modules_per_rank": 5,
"packed_component_bytes_per_rank": {
"target": {
"qweight": 12175278080,
"qzeros": 95119360,
"scales": 190238720,
"g_idx": 0
},
"draft": {
"qweight": 212336640,
"qzeros": 1658880,
"scales": 3317760,
"g_idx": 0
}
},
"packed_target_bytes_per_rank": 12460636160,
"packed_draft_bytes_per_rank": 217313280,
"packed_weight_bytes_per_rank": 12677949440,
"sampled_peak_VRAM_bytes": {
"0000:16:00.0": 28642746368,
"0000:19:00.0": 28642762752
},
"loaded_allocator_and_KV_per_rank": [
{
"rank": 0,
"load_allocated": 15485343232,
"load_reserved": 17783848960,
"KV_allocated": 25826995200,
"KV_reserved": 26908557312,
"KV_shapes": [
[
370,
2,
800,
2,
256
]
]
},
{
"rank": 1,
"load_allocated": 15485343232,
"load_reserved": 17783848960,
"KV_allocated": 25826995200,
"KV_reserved": 26908557312,
"KV_shapes": [
[
370,
2,
800,
2,
256
]
]
}
],
"KV_capacity": {
"available_GiB": 9.6,
"reported_grouped_tokens": 73600,
"reported_concurrency_at131072": 2.14,
"target_views": 16,
"draft_views": 1,
"view_shape": [
370,
2,
800,
2,
256
],
"view_bytes": 606208000
},
"configured_context": 131072,
"complete_ladders": 1,
"sample_count": 1,
"repeatability_established": false,
"FP16_release_qualification": false,
"archive": {
"path": "/home/ai/vnext-v620-qwen38-rtn8-tp2-mtp2-service-r1-20261023/raw-evidence.tar.gz",
"bytes": 781297,
"sha256": "aee594a5aa2bf1da84a258539e121fb6a5f04b26c370ba01238952389c31a115",
"files": 243,
"index_sha256": "f7bef99af0d881dcf489b48bd83031337999e514ae604d07250bddd16dd9c031"
},
"local_primary_record": ".work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/sealed-mirror/retained/evidence/measurement-closure.json",
"review": {
"path": ".work/fp16-w8-tp-mtp-matrix/reviews/W8-TP2-MTP2-r1-final-cell-review.json",
"sha256": "028f1e91be7e589908ab9308842bbbd7b72721272a4ce3772c7ad6a93fdaba68"
},
"closure": {
"container_id": "d24d898cadfa99b4a20d3004c4ed56b4100702daa0afe37d2eb1adef9092d4a8",
"image_id": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
"exit_code": 0,
"OOMKilled": false,
"restart_count": 0,
"bank_idle": true,
"devices": [
{
"pci": "0000:16:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
},
{
"pci": "0000:19:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
}
],
"started_at": "2026-10-23T15:54:41.228163364Z",
"finished_at": "2026-10-23T16:44:32.609572419Z",
"closed_at": "2026-10-23T16:44:32.779906+00:00",
"kernel_fault_matches": [],
"startup_requests_completed_and_drained": 3,
"official_phase_samples": 11,
"complete_ladders": 1,
"strict_valid": true,
"official_fixed_2000_tps": 25.99758548499443,
"qualifying_ladders": 0,
"FP16_qualification": false,
"baseline_kind": "RTN8"
},
"final_counters": {
"requests": 14,
"prompt_tokens": 6622,
"committed_generation_tokens": 38872
},
"runtime_differences": [
"Opaque native selector repairs the free-s47 compiler failure; it forwards to the unchanged ordered-FP32 U operator for M<=32 and original safe reconstruction for larger batches. New dispatch flag1 and mapped DSOdf0df1a9; native U45281e2c and safef52f2e8b unchanged.",
"Native MTP2 target and draft, ordinary decoder/collectives and 800 attention fallback differ from historical non-MTP U profile. Same eaf8 image as measured TP4 MTP2; TP2 reshards and KV budget differ. No isolated MTP or compiler-speed attribution.",
"Fresh private rootSSD caches, FP16 activations/attentionKV, memory utilization0.95, context131072, greedy concurrency1 and original requests/scorer retained.",
"Existing MTP-scoped post-sharing cache cleanup runs for TP2; the non-MTP TP2 path does not take that branch. Observed allocator reserve and KV budget differ; this is not an isolated MTP gain."
],
"limits": [
"One sample per tier; no repeatability or final qualification.",
"Committed output only supplies numerator; speculative acceptance precedes stop trimming.",
"Static256 opaque calls and library mappings do not establish dynamic per-kernel trace; MTP counters show actual speculation.",
"Same eaf8 image/e4e131d source as TP4 MTP; TP reshards and KV budget differ, native MTP800 attention fallback/fresh private rootSSD caches materially differ from historical nonMTP.",
"Logical packs/views are not unique whole-model physical memory; remaining FP16 exclusions not fully itemized.",
"Inactive failed-tail unused because strict passed.",
"Reported grouped KV tokens and concurrency are runtime capacity indicators, not an independently tested exact request ceiling or demonstrated full-context residency.",
"Original launch observer ended143 without its SSH child exit receipt. Separate exact-container/owned-waiter verification confirms stopped/drained; no claim that all parents exited0."
],
"launch_observer_exception": {
"path": ".work/fp16-w8-tp-mtp-matrix/w8-tp2-mtp2-service-r1/launch-observer-terminal.json",
"sha256": "e537cc8661b1b1021723ef0b22b5d8cc23b4e8ecf855b227bc68646c84ba9608",
"tool_terminal_exit_code": 143,
"original_SSH_parent_exit": null,
"actual_container_exit_code": 0,
"exact_owned_waiter_absent": true,
"interpretation": "The original launch-observer tool terminated143 without writing its SSH child parent receipt. Do not report its exit as0. Actual inference and closure are independently retained; exact container is stopped and its exact remote docker-wait observer is absent."
},
"parameter_dtypes_per_rank": [
{
"rank": 0,
"parameter_dtype_counts": {
"torch.float16": 611,
"torch.int32": 768
},
"draft_parameter_dtype_counts": {
"torch.int32": 15,
"torch.float16": 14
}
},
{
"rank": 1,
"parameter_dtype_counts": {
"torch.float16": 611,
"torch.int32": 768
},
"draft_parameter_dtype_counts": {
"torch.int32": 15,
"torch.float16": 14
}
}
]
},
"MTP_2K_draft_batches": 829.0,
"MTP_2K_proposed": 1658.0,
"MTP_2K_accepted": 1170.0,
"MTP_2K_acceptance_percent": 70.56694813027744,
"MTP_2K_committed": 2000.0,
"detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
"detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
"detail_source_pointer": "FP16_W8_TP_MTP_matrix.cells[5]",
"retest_strict_tokens": 14095,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 9314.0,
"vllm:spec_decode_num_drafts_total": 4780.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 14095.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 9314.0,
"vllm:request_generation_tokens_sum": 14095.0,
"vllm:spec_decode_num_draft_tokens_total": 9560.0
},
"accepted_per_position": {
"0": 4699.0,
"1": 4615.0
},
"accepted_fraction": 0.9742677824267783,
"committed_emitted_tokens": 14095.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "d499da222db5f4c902370bd706b3037ebd84b7cd1c3ca91296c6f0641cfb1dcf",
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "4c68d0bfce8174357d766b4c218392da6257632f05d570487984b9c97f56b6dc"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_draft_tokens_total": 1494.0,
"vllm:spec_decode_num_accepted_tokens_total": 1252.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1252.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:spec_decode_num_drafts_total": 747.0,
"vllm:request_generation_tokens_count": 1.0
},
"accepted_per_position": {
"0": 667.0,
"1": 585.0
},
"accepted_fraction": 0.8380187416331994,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "f7462730d6f10a811663c5107c3584869ce2f10c7a1ace0726aeb09a783c0275",
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "f9eefe90a5ed4ea8030f25ec580474c5d1f9ee6238d18cc2d27bab36615688af"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_draft_tokens_total": 6830.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6586.0,
"vllm:spec_decode_num_drafts_total": 3415.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_total": 6586.0,
"vllm:generation_tokens_total": 10000.0
},
"accepted_per_position": {
"0": 3335.0,
"1": 3251.0
},
"accepted_fraction": 0.9642752562225476,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "3d9f3d802948b3286135fb227b09e5ae44fa5ee621e625f95dc6f2df71691291",
"/home/ai/qwen38-release-retests-20261026/v620-w8-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "e6abdb87a069a4d72731346e46434237176a71f431b126902abadb04ba82bd7b"
}
}
},
"retest_MTP_2K_accepted": 1252.0,
"retest_MTP_2K_proposed": 1494.0,
"retest_MTP_2K_draft_batches": 747.0,
"retest_MTP_2K_acceptance_percent": 83.80187416331995,
"retest_diagnostic_counter_audit": "retest-evidence/v620-w8-tp2-mtp2/local-diagnostic-fixed-audit.json",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 28642738176,
"1": 28642762752
},
"retest_memory_observation_count": 58,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-w8-tp2-mtp2/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 9.6,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 9.6,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 73600,
"retest_memory_log_report": {
"profile": "v620-w8-tp2-mtp2",
"image": "sha256:eaf8fffc828de2bb017493048c35a81a22b96964ca4852af5732cf17d6f38567",
"container": "21a9828f12b1d34b82e2d0a87d86fb0d50ba8af66ac161c5ef4070a0e8424ed6",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 16.56,
"retest_logged_model_load_seconds": 123.006295,
"retest_logged_available_KV_GiB": 9.6,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 9.6,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 73600,
"retest_logged_context_capacity_x": 2.14,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w8-tp2-mtp2/service-final.log",
"source_log_sha256": "55ab409a441ae421f1d1b05f492f45e5125d690e7dcbd2140e32c403b07075c4",
"sealed_index_sha256": "80034e3ab771353bf7a27f0c11c3a059a53363a207eb153ed442827132adb6d0",
"source_lines": {
"model_load": {
"line": 118,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 20:37:10 [gpu_model_runner.py:4598] Model loading took 16.56 GiB memory and 123.006295 seconds"
},
"automatic_KV_budget": {
"line": 172,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 20:57:42 [gpu_worker.py:453] Available KV cache memory: 9.6 GiB"
},
"grouped_KV_tokens": {
"line": 176,
"text": "(EngineCore_DP0 pid=85) INFO 10-26 20:57:42 [kv_cache_utils.py:1314] GPU KV cache size: 73,600 tokens"
},
"context_capacity": {
"line": 177,
"text": "(EngineCore_DP0 pid=85) INFO 10-26 20:57:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 2.14x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w8-tp2-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 26.67565155029297,
"sampled_peak_VRAM_GiB": 26.67565155029297
} |
| V620Current Qwen3.8 | 4-bit RTNQwen/Qwen3.8-27B | TP4MTP off | PASS15275 tokens | 40.235 | 38.552 | 1.093 / 50.81610K: 1.093 / 260.499Strict: — / — | 2.95 / 21.09packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:22607123e397e8cfd18f2ba40413e2cd7321d435507e65041f1336416114c596Runtime image config: sha256:b97918608d83591ad38f818bb4e944b5aa8d7d2fa9df2f354d47f1db190d9fcb | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256 mkdir -p launch-tools-v620-serve-r5 tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp4-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w4-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Preflight cleanup compatibilityAfter its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged. Retained lab Docker command · not portabledocker run -d \
--name serve-v620-w4-tp4-mtp0-20261028 \
--network host \
--ipc host \
--cpuset-cpus 0-7,16-23 \
--cpuset-mems 0 \
--security-opt label=disable \
--device /dev/kfd:/dev/kfd:rwm \
--device /dev/dri/card5:/dev/dri/card5:rwm \
--device /dev/dri/card6:/dev/dri/card6:rwm \
--device /dev/dri/card7:/dev/dri/card7:rwm \
--device /dev/dri/card8:/dev/dri/card8:rwm \
--device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
--device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
--device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
--device /dev/dri/renderD136:/dev/dri/renderD136:rwm \
--user root \
--workdir /tmp \
-e VLLM_DISABLE_COMPILE_CACHE=1 \
-e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
-e ROCM_VERSION=7.14.0 \
-e TORCH_VERSION=2.12.0+rocm7.14.0 \
-e VLLM_TARGET_DEVICE=rocm \
-e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
-e PYTORCH_ROCM_ARCH=gfx1030 \
-e GPU_ARCHS=gfx1030 \
-e AMDGPU_TARGETS=gfx1030 \
-e CMAKE_HIP_ARCHITECTURES=gfx1030 \
-e PYTHONDONTWRITEBYTECODE=1 \
-e PIP_NO_CACHE_DIR=1 \
-e PIP_DISABLE_PIP_VERSION_CHECK=1 \
-e VNEXT_GDN_HEAD32=1 \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0/cache/vllm,dst=/root/.cache/vllm \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0/cache/torch_extensions,dst=/root/.cache/torch_extensions \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0/cache/.triton,dst=/root/.triton \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
--mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
--entrypoint /usr/bin/env \
sha256:b97918608d83591ad38f818bb4e944b5aa8d7d2fa9df2f354d47f1db190d9fcb \
V756F94_BOUNDARY_MODE=posted \
DO_NOT_TRACK=1 \
GPU_DEVICE_ORDINAL=0,1,2,3 \
GPU_MAX_HW_QUEUES=1 \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
OMP_NUM_THREADS=4 \
TEMP=/tmp/torchinductor_root \
TMPDIR=/tmp/torchinductor_root \
TMP=/tmp/torchinductor_root \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
TRITON_CACHE_DIR=/root/.triton \
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
VLLM_GPU_MEMORY_UTILIZATION=0.95 \
VLLM_LOGGING_LEVEL=INFO \
VLLM_PORT=9002 \
VLLM_RDNA_FORCE_FP16=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
NCCL_P2P_DISABLE=0 \
NCCL_ALGO=Tree \
NCCL_PROTO=LL \
NCCL_MIN_NCHANNELS=4 \
NCCL_MAX_NCHANNELS=4 \
NCCL_NTHREADS=256 \
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
HIP_VISIBLE_DEVICES=0,1,2,3 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
ROCR_VISIBLE_DEVICES=0,1,2,3 \
VLLM_CACHE_ROOT=/root/.cache/vllm \
PYTORCH_ROCM_ARCH=gfx1030 \
PYTHONHASHSEED=0 \
/opt/venv/bin/python \
-m \
vllm.vnext_rtn_entry \
serve \
/opt/local-models/qwen38-27b-hf \
--served-model-name \
Qwen/Qwen3.8-27B \
--enable-auto-tool-choice \
--tool-call-parser \
qwen3_coder \
--dtype \
half \
--host \
0.0.0.0 \
--port \
9002 \
--tensor-parallel-size \
4 \
--max-model-len \
131072 \
--gpu-memory-utilization \
0.95 \
--trust-remote-code \
--generation-config \
vllm \
-O=3 \
--reasoning-parser \
qwen3 \
--async-scheduling \
--hf-overrides \
'{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":4,"group_size":128}}' \
--language-model-only \
--max-num-seqs \
2 \
--max-num-batched-tokens \
2048 \
--mamba-ssm-cache-dtype \
float32 \
--no-enable-prefix-caching \
--no-enforce-eager \
--attention-backend \
TRITON_ATTN \
--disable-custom-all-reduce \
--seed \
0 \
--worker-cls \
vllm.vnext_rtn_worker.Worker \
--quantization \
vnext_rtnHistorical host paths and device addresses are retained as evidence. Use the serving command above.Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-r4.tar.gz.sha256 mkdir -p launch-tools-v620-r4 tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp4-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-v620-r4/portable/launch.py --config v620-w4-tp4-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-v620-r4.tar.gz · SHA-256 launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. Portable observer compatibilityThe evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged. | All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 39.296 · eligible10K 37.676 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 5806,
"content_chars": 5723,
"content_has_think_close": false,
"elapsed_seconds": 151.7962831039913,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T13:19:26Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.011801122920587659,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 5723,
"qwen_answer_text_sha256": "8e59252e3ad8a4168398b9374ee274fc88656e2899d69096795556d8456cfa40",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 5723,
"qwen_visible_text_sha256": "8e59252e3ad8a4168398b9374ee274fc88656e2899d69096795556d8456cfa40",
"raw_has_think_close": false,
"reasoning_chars": 20749,
"text_chars": 26473,
"text_sha256": "c8e300e905259796a9b14c161b6d0a6c59a73258ff4f5c6a4e8572a569d17edf",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 150.68790842499584,
"vllm_decode_tps": 38.529966078133654,
"vllm_generation_tokens_delta": 5806.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.0972954761236906,
"vllm_prefill_tps": 431.05982872628005,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 38.24863087077353
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 52.00303460191935,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T13:20:18Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.01141878985799849,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 7704,
"text_chars": 7704,
"text_sha256": "e53a38a234f910abe1a3b42cc909a8dec4b0f5e024f3f3cb5552ae1133004552",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 50.895956503925845,
"vllm_decode_tps": 39.29585250737415,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.0962058920413256,
"vllm_prefill_tps": 431.48828466812193,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 38.45929406446952
},
"10K": {
"completion_tokens": 10000,
"content_chars": 17402,
"content_has_think_close": true,
"elapsed_seconds": 266.53216635319404,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T13:24:45Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.007715465035289526,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 10070,
"qwen_answer_text_sha256": "96dd1749d2bef179d10b9d1b6a0afc735325a7ae603c43af72851efd69b35bb2",
"qwen_gate_close_in_content": true,
"qwen_gate_close_in_raw": true,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 10070,
"qwen_visible_text_sha256": "96dd1749d2bef179d10b9d1b6a0afc735325a7ae603c43af72851efd69b35bb2",
"raw_has_think_close": true,
"reasoning_chars": 26874,
"text_chars": 44277,
"text_sha256": "a080354fab849841fae9032656d66dbbd478d408ad07b3176e2ba72fdf8cf58d",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 265.41904551698826,
"vllm_decode_tps": 37.67627142401108,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.0997616411186755,
"vllm_prefill_tps": 430.0931968484237,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 37.518923651220916
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS6,146 natural tokens2K 40.36110K 38.396 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T14:35:39.487703175Z",
"FinishedAt": "2026-10-26T15:14:01.113195407Z"
},
"devices_after": [
{
"card": 5,
"render": 133,
"pci": "0000:16:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:19:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 7,
"render": 135,
"pci": "0000:1c:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 8,
"render": 136,
"pci": "0000:1f:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
}
],
"finished": "2026-10-26T15:14:01.539984+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 40.36122244861641,
"retest_10K_TPS": 38.395805792032654,
"retest_strict_tokens": 6146,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 6146,
"elapsed_seconds": 156.78889757196885,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 155.68011727603152,
"vllm_decode_tps": 39.47838752653764,
"vllm_generation_tokens_delta": 6146.0,
"vllm_prefill_seconds_delta": 1.095762872020714,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 50.6597384349443,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 49.5525129979942,
"vllm_decode_tps": 40.36122244861641,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 1.0948759620077908,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 261.55461658199783,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 260.4451135669369,
"vllm_decode_tps": 38.395805792032654,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 1.094910062965937,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 1.0948759620077908,
"retest_2K_total_seconds": 50.6597384349443,
"retest_10K_prefill_seconds": 1.094910062965937,
"retest_10K_total_seconds": 261.55461658199783,
"retest_strict_prefill_seconds": 1.095762872020714,
"retest_strict_total_seconds": 156.78889757196885,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": {
"all_four_quantized_module_receipts_passed": true,
"generated_quantized_call_support_complete": true,
"graphs": [
{
"rank": 0,
"pid": 83,
"source_sha256": "b8f11a3f101d5df627672bb7bf0cdaf702e53502badd899ed5b9faffb523dcaa",
"generated_gemm_calls": 256
},
{
"rank": 1,
"pid": 84,
"source_sha256": "5bac1879e929112251a66ce5b2b42bddb1c9446e72115e2d982d4d8d7c6e749c",
"generated_gemm_calls": 256
},
{
"rank": 2,
"pid": 85,
"source_sha256": "7f3599a6704c7950176ff6c909c3a8a894837947a21e37b6357c45ecee9ae4de",
"generated_gemm_calls": 256
},
{
"rank": 3,
"pid": 86,
"source_sha256": "b5631bb04aff8712641b794dd9379862ed15578210663671fae4dd54f19b6a5c",
"generated_gemm_calls": 256
}
],
"attention_KV_observations": [
{
"rank": 3,
"pid": 86,
"attention_layers": [
3,
7,
11,
15,
19,
23,
27,
31,
35,
39,
43,
47,
51,
55,
59,
63
],
"logical_view_bytes": 14758969344,
"Mamba_SSM_config_dtype": "float32",
"limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
},
{
"rank": 1,
"pid": 84,
"attention_layers": [
3,
7,
11,
15,
19,
23,
27,
31,
35,
39,
43,
47,
51,
55,
59,
63
],
"logical_view_bytes": 14758969344,
"Mamba_SSM_config_dtype": "float32",
"limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
},
{
"rank": 0,
"pid": 83,
"attention_layers": [
3,
7,
11,
15,
19,
23,
27,
31,
35,
39,
43,
47,
51,
55,
59,
63
],
"logical_view_bytes": 14758969344,
"Mamba_SSM_config_dtype": "float32",
"limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
},
{
"rank": 2,
"pid": 85,
"attention_layers": [
3,
7,
11,
15,
19,
23,
27,
31,
35,
39,
43,
47,
51,
55,
59,
63
],
"logical_view_bytes": 14758969344,
"Mamba_SSM_config_dtype": "float32",
"limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
}
],
"all_four_KV_receipts_passed": true,
"limits": "Static generated call support plus bound loaded-module receipts/native mappings and completed startup. Not a GPU completion trace or correctness verdict."
},
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_logged_model_load_delta_GiB": 13.47,
"retest_logged_model_load_seconds": 59.703993,
"retest_logged_available_KV_GiB": 13.76,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 13.76,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 225008,
"retest_logged_context_capacity_x": 6.72,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "v620-w4-tp4-mtp0",
"image": "sha256:b97918608d83591ad38f818bb4e944b5aa8d7d2fa9df2f354d47f1db190d9fcb",
"container": "c1f1d4225224fe57f881a4a2eea02bb60ad836c7c0e96adf895bcd78d6ec7694",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 13.47,
"retest_logged_model_load_seconds": 59.703993,
"retest_logged_available_KV_GiB": 13.76,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 13.76,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 225008,
"retest_logged_context_capacity_x": 6.72,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w4-tp4-mtp0/service-final.log",
"source_log_sha256": "0e0bfd459de2a7ef66c6f6225696c4722266fd04a763121e3bb1ebc028697ca3",
"sealed_index_sha256": "30b88085c5128904cba9bfbad035f6f909fd5b6ae8920387bc8fd26a2807844b",
"source_lines": {
"model_load": {
"line": 120,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 14:38:21 [gpu_model_runner.py:4579] Model loading took 13.47 GiB memory and 59.703993 seconds"
},
"automatic_KV_budget": {
"line": 141,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 14:41:09 [gpu_worker.py:453] Available KV cache memory: 13.76 GiB"
},
"grouped_KV_tokens": {
"line": 144,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 14:41:10 [kv_cache_utils.py:1314] GPU KV cache size: 225,008 tokens"
},
"context_capacity": {
"line": 145,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 14:41:10 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.72x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w4-tp4-mtp0/memory-log-report.json"
} | Evidence and limitsselected historical RTN4 reference {
"strict_tokens": 15275,
"source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.full_model_results.RTN4",
"source_revision": "999813806eae1e07fffb3baccd359595fbcc2358",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"comparison_limit": "One quantized sample; no repeated-sample spread. Quantized TP route, Head32/compiler mode and rootSSD/cache warmth differ from preserved FP16 control. No FP16 qualification.",
"latency_sources": "[{\"path\": \"/home/joe/PhoneSync/Documents/gfx1030/.work/quantized-baselines/rtn4-service-r1/sealed-mirror/retained/evidence/ladder2/c1_2000_proxy/summary.json\", \"sha256\": \"2c0f21a7379f624b75090fba65cb533335f92c6790cc8a85ce3084f140349814\"}, {\"path\": \"/home/joe/PhoneSync/Documents/gfx1030/.work/quantized-baselines/rtn4-service-r1/sealed-mirror/retained/evidence/ladder2/c1_10000_proxy/summary.json\", \"sha256\": \"a743ecb3b71d4aad74a8ee25eae600b9fd3577a2b9bedd6104f33146a99f46f1\"}]",
"sampled_peak_VRAM_bytes_by_PCI": {
"0000:16:00.0": 22627618816,
"0000:19:00.0": 22644396032,
"0000:1c:00.0": 22644400128,
"0000:1f:00.0": 22594056192
},
"memory_accounting_limit": "Packed target matrices exclude FP16 embeddings/head/norm/recurrent weights. Sampled physical VRAM is not a guaranteed continuous peak.",
"memory_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#full_model_results.RTN4.memory",
"memory_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
"retest_strict_tokens": 6146,
"retest_MTP_accounting_by_tier": {},
"retest_logged_available_KV_GiB": 13.76,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 13.76,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 225008,
"retest_memory_log_report": {
"profile": "v620-w4-tp4-mtp0",
"image": "sha256:b97918608d83591ad38f818bb4e944b5aa8d7d2fa9df2f354d47f1db190d9fcb",
"container": "c1f1d4225224fe57f881a4a2eea02bb60ad836c7c0e96adf895bcd78d6ec7694",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 13.47,
"retest_logged_model_load_seconds": 59.703993,
"retest_logged_available_KV_GiB": 13.76,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 13.76,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 225008,
"retest_logged_context_capacity_x": 6.72,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w4-tp4-mtp0/service-final.log",
"source_log_sha256": "0e0bfd459de2a7ef66c6f6225696c4722266fd04a763121e3bb1ebc028697ca3",
"sealed_index_sha256": "30b88085c5128904cba9bfbad035f6f909fd5b6ae8920387bc8fd26a2807844b",
"source_lines": {
"model_load": {
"line": 120,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 14:38:21 [gpu_model_runner.py:4579] Model loading took 13.47 GiB memory and 59.703993 seconds"
},
"automatic_KV_budget": {
"line": 141,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 14:41:09 [gpu_worker.py:453] Available KV cache memory: 13.76 GiB"
},
"grouped_KV_tokens": {
"line": 144,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 14:41:10 [kv_cache_utils.py:1314] GPU KV cache size: 225,008 tokens"
},
"context_capacity": {
"line": 145,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 14:41:10 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.72x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w4-tp4-mtp0/memory-log-report.json",
"sampled_peak_VRAM_GiB": 21.08924102783203
} |
| V620Current Qwen3.8 | 4-bit RTNQwen/Qwen3.8-27B | TP2MTP off | PASS6876 tokens | 32.807 | 30.823 | 0.774 / 61.75210K: 0.827 / 325.279Strict: 0.755 / 217.468 | 5.89 / 12.71packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:b7beecf0afb95cd5501aa395497b11d9eaff91f7e967af31335e825e6715f553Runtime image config: sha256:a16a6f2d47e7ac6ffc55d7a3d9298cad7e03601d8a30886421e905c92aeae386 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256 mkdir -p launch-tools-v620-serve-r5 tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp2-mtp0 GPU_RENDER_NODES=renderD128,renderD129 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w4-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Preflight cleanup compatibilityAfter its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged. Retained lab Docker command · not portabledocker run -d \
--name serve-v620-w4-tp2-mtp0-20261028 \
--network host \
--ipc host \
--cpuset-cpus 0-7,16-23 \
--cpuset-mems 0 \
--security-opt label=disable \
--shm-size 67108864 \
--device /dev/kfd:/dev/kfd:rwm \
--device /dev/dri/card5:/dev/dri/card5:rwm \
--device /dev/dri/card6:/dev/dri/card6:rwm \
--device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
--device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
--user root \
--workdir /tmp \
-e VLLM_DISABLE_COMPILE_CACHE=1 \
-e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
-e ROCM_VERSION=7.14.0 \
-e TORCH_VERSION=2.12.0+rocm7.14.0 \
-e VLLM_TARGET_DEVICE=rocm \
-e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
-e PYTORCH_ROCM_ARCH=gfx1030 \
-e GPU_ARCHS=gfx1030 \
-e AMDGPU_TARGETS=gfx1030 \
-e CMAKE_HIP_ARCHITECTURES=gfx1030 \
-e PYTHONDONTWRITEBYTECODE=1 \
-e PIP_NO_CACHE_DIR=1 \
-e PIP_DISABLE_PIP_VERSION_CHECK=1 \
-e VNEXT_GDN_HEAD32=1 \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp0/cache/vllm,dst=/root/.cache/vllm \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp0/cache/torch_extensions,dst=/root/.cache/torch_extensions \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp0/cache/.triton,dst=/root/.triton \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \
--mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
--entrypoint /usr/bin/env \
sha256:a16a6f2d47e7ac6ffc55d7a3d9298cad7e03601d8a30886421e905c92aeae386 \
V756F94_BOUNDARY_MODE=posted \
DO_NOT_TRACK=1 \
GPU_DEVICE_ORDINAL=0,1 \
GPU_MAX_HW_QUEUES=1 \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
OMP_NUM_THREADS=4 \
TEMP=/tmp/torchinductor_root \
TMPDIR=/tmp/torchinductor_root \
TMP=/tmp/torchinductor_root \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
TRITON_CACHE_DIR=/root/.triton \
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
VLLM_GPU_MEMORY_UTILIZATION=0.95 \
VLLM_LOGGING_LEVEL=INFO \
VLLM_PORT=9002 \
VLLM_RDNA_FORCE_FP16=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
NCCL_P2P_DISABLE=0 \
NCCL_ALGO=Tree \
NCCL_PROTO=LL \
NCCL_MIN_NCHANNELS=4 \
NCCL_MAX_NCHANNELS=4 \
NCCL_NTHREADS=256 \
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
HIP_VISIBLE_DEVICES=0,1 \
CUDA_VISIBLE_DEVICES=0,1 \
ROCR_VISIBLE_DEVICES=0,1 \
VLLM_CACHE_ROOT=/root/.cache/vllm \
PYTORCH_ROCM_ARCH=gfx1030 \
PYTHONHASHSEED=0 \
/opt/venv/bin/python \
-m \
vllm.vnext_rtn_entry \
serve \
/opt/local-models/qwen38-27b-hf \
--served-model-name \
Qwen/Qwen3.8-27B \
--enable-auto-tool-choice \
--tool-call-parser \
qwen3_coder \
--dtype \
half \
--host \
0.0.0.0 \
--port \
9002 \
--tensor-parallel-size \
2 \
--max-model-len \
131072 \
--gpu-memory-utilization \
0.95 \
--trust-remote-code \
--generation-config \
vllm \
-O=3 \
--reasoning-parser \
qwen3 \
--async-scheduling \
--hf-overrides \
'{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":4,"group_size":128}}' \
--language-model-only \
--max-num-seqs \
2 \
--max-num-batched-tokens \
2048 \
--mamba-ssm-cache-dtype \
float32 \
--no-enable-prefix-caching \
--no-enforce-eager \
--attention-backend \
TRITON_ATTN \
--disable-custom-all-reduce \
--seed \
0 \
--worker-cls \
vllm.vnext_rtn_worker.Worker \
--quantization \
vnext_rtnHistorical host paths and device addresses are retained as evidence. Use the serving command above.Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-r4.tar.gz.sha256 mkdir -p launch-tools-v620-r4 tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp2-mtp0 GPU_RENDER_NODES=renderD128,renderD129 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-v620-r4/portable/launch.py --config v620-w4-tp2-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-v620-r4.tar.gz · SHA-256 launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. Portable observer compatibilityThe evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged. | All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 32.905 · eligible10K 30.623 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 13863,
"content_chars": 13741,
"content_has_think_close": false,
"elapsed_seconds": 468.5006339759566,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T02:20:17Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.018792743096128106,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 13741,
"qwen_answer_text_sha256": "2e9c97ea581575ec316e08addf3ccc841d38debee190556442a487c75930119a",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 13741,
"qwen_visible_text_sha256": "2e9c97ea581575ec316e08addf3ccc841d38debee190556442a487c75930119a",
"raw_has_think_close": false,
"reasoning_chars": 45894,
"text_chars": 59636,
"text_sha256": "1d3ba196ea092142805106b38c0a5d382fc2c7dc33d86a2e796aa58da9891812",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 467.7273157769814,
"vllm_decode_tps": 29.63906432740837,
"vllm_generation_tokens_delta": 13863.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.7524218091275543,
"vllm_prefill_tps": 628.6367490443312,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 29.59014138860578
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 61.55541231902316,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T02:21:18Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.01002282602712512,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 8669,
"text_chars": 8669,
"text_sha256": "0c4f91d92e62192ace2c7cc81141fa7fd2ee22ef862689251a23d577f289357e",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 60.78103750408627,
"vllm_decode_tps": 32.90499935716861,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.7525132559239864,
"vllm_prefill_tps": 628.5603559491038,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 32.49105033420298
},
"10K": {
"completion_tokens": 10000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 327.3320147569757,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T02:26:46Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.014586291974410415,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 35150,
"text_chars": 35150,
"text_sha256": "9328393526b4b230ec4a5b020a16cb345bc0602ac1ca9c27d3c20bbd3698125e",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 326.5560789539013,
"vllm_decode_tps": 30.622611687506396,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.7571389379445463,
"vllm_prefill_tps": 624.7202148711086,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 30.550021229742523
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS8,472 natural tokens2K 33.22610K 30.907 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T16:13:32.669837852Z",
"FinishedAt": "2026-10-26T16:59:16.661768663Z"
},
"devices_after": [
{
"card": 5,
"render": 133,
"pci": "0000:16:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:19:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
}
],
"finished": "2026-10-26T16:59:16.794941+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 33.226071157256506,
"retest_10K_TPS": 30.90653983479376,
"retest_strict_tokens": 8472,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 8472,
"elapsed_seconds": 271.0291323700221,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 270.2647703959374,
"vllm_decode_tps": 31.347037897645837,
"vllm_generation_tokens_delta": 8472.0,
"vllm_prefill_seconds_delta": 0.7526620751013979,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 60.957591224927455,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 60.193695202004164,
"vllm_decode_tps": 33.226071157256506,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 0.7528414420085028,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 324.3207754510222,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 323.556116390042,
"vllm_decode_tps": 30.90653983479376,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 0.7524013130459934,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 0.7528414420085028,
"retest_2K_total_seconds": 60.957591224927455,
"retest_10K_prefill_seconds": 0.7524013130459934,
"retest_10K_total_seconds": 324.3207754510222,
"retest_strict_prefill_seconds": 0.7526620751013979,
"retest_strict_total_seconds": 271.0291323700221,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": {
"all_two_quantized_module_receipts_passed": true,
"generated_quantized_call_support_complete": true,
"graphs": [
{
"rank": 0,
"pid": 83,
"source_sha256": "40be7ddb392affdf12e3436e1a59fb43d565da92e234b2d5df1f2c4fb7068600",
"generated_gemm_calls": 256
},
{
"rank": 1,
"pid": 84,
"source_sha256": "6ac2e022f4974aab00e5590d8c15af8a94ed08c16b0acd84a38877ae853b9e87",
"generated_gemm_calls": 256
}
],
"attention_KV_observations": [
{
"rank": 1,
"pid": 84,
"attention_layers": [
3,
7,
11,
15,
19,
23,
27,
31,
35,
39,
43,
47,
51,
55,
59,
63
],
"logical_view_bytes": 1104674816,
"Mamba_SSM_config_dtype": "float32",
"limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
},
{
"rank": 0,
"pid": 83,
"attention_layers": [
3,
7,
11,
15,
19,
23,
27,
31,
35,
39,
43,
47,
51,
55,
59,
63
],
"logical_view_bytes": 1104674816,
"Mamba_SSM_config_dtype": "float32",
"limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
}
],
"all_two_KV_receipts_passed": true,
"limits": "Static generated call support plus bound loaded-module receipts/native mappings and completed startup. Not a GPU completion trace or correctness verdict."
},
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 13639860224,
"1": 13642035200
},
"retest_memory_observation_count": 33,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-w4-tp2-mtp0/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 25.59,
"retest_logged_model_load_seconds": 114.199732,
"retest_logged_available_KV_GiB": 1.05,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 1.05,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 7840,
"retest_logged_context_capacity_x": 0.25,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "v620-w4-tp2-mtp0",
"image": "sha256:a16a6f2d47e7ac6ffc55d7a3d9298cad7e03601d8a30886421e905c92aeae386",
"container": "63d508581fc91c4a06d201f732ab17b0e41917d68c06a98af7a60049e74f0225",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 25.59,
"retest_logged_model_load_seconds": 114.199732,
"retest_logged_available_KV_GiB": 1.05,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 1.05,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 7840,
"retest_logged_context_capacity_x": 0.25,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w4-tp2-mtp0/service-final.log",
"source_log_sha256": "1de293b6b62bc00ad5bad8ca7ae9b8602161b0137af4ffd056b3e51f09b244b3",
"sealed_index_sha256": "d5b582c7896849a6be95b24fdc53dcdd4cecd5c8e248a63f69ae296aee7afd9d",
"source_lines": {
"model_load": {
"line": 92,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 16:17:07 [gpu_model_runner.py:4579] Model loading took 25.59 GiB memory and 114.199732 seconds"
},
"automatic_KV_budget": {
"line": 104,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 16:20:42 [gpu_worker.py:453] Available KV cache memory: 1.05 GiB"
},
"grouped_KV_tokens": {
"line": 107,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 16:20:42 [kv_cache_utils.py:1314] GPU KV cache size: 7,840 tokens"
},
"context_capacity": {
"line": 108,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 16:20:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 0.25x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w4-tp2-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 12.705135345458984
} | Evidence and limitscompleted matrix cell {
"strict_tokens": 6876,
"source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.local_RTN4_TP_MTP_matrix.ordered_cells[TP=2,MTP_tokens=0].result",
"source_revision": "7a6d45978c2ccecdf704899625b813cab2aa7979",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"recorded_configuration_details": {
"strict_valid": true,
"strict_tokens": 6876,
"strict_diagnostic_TPS": 31.730622009286282,
"official_fixed2K_TPS": 32.806665219045115,
"diagnostic_fixed10K_TPS": 30.822545818435916,
"source": "7a6d45978c2ccecdf704899625b813cab2aa7979",
"image": "sha256:a16a6f2d47e7ac6ffc55d7a3d9298cad7e03601d8a30886421e905c92aeae386",
"sample_count_per_tier": 1,
"metrics": {
"strict": {
"completion_tokens": 6876,
"elapsed_seconds": 217.46813429001486,
"vllm_prefill_seconds_delta": 0.7545584980107378,
"vllm_decode_seconds_delta": 216.69918723899173,
"vllm_decode_tps": 31.730622009286282,
"vllm_prompt_tokens_delta": 473.0,
"vllm_generation_tokens_delta": 6876.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0
},
"fixed2K": {
"completion_tokens": 2000,
"elapsed_seconds": 61.752023694018135,
"vllm_prefill_seconds_delta": 0.7744208740186878,
"vllm_decode_seconds_delta": 60.96322154800873,
"vllm_decode_tps": 32.806665219045115,
"vllm_prompt_tokens_delta": 473.0,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0
},
"fixed10K": {
"completion_tokens": 10000,
"elapsed_seconds": 325.2792088240094,
"vllm_prefill_seconds_delta": 0.8272731059987564,
"vllm_decode_seconds_delta": 324.437833879987,
"vllm_decode_tps": 30.822545818435916,
"vllm_prompt_tokens_delta": 473.0,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0
}
},
"closure": {
"container_id": "69d228271089fe5543cb2c4fcaf89af086fc93ce8b5e3d09571e659e3b6e9d87",
"image_id": "sha256:a16a6f2d47e7ac6ffc55d7a3d9298cad7e03601d8a30886421e905c92aeae386",
"exit_code": 0,
"OOMKilled": false,
"restart_count": 0,
"bank_idle": true,
"devices": [
{
"pci": "0000:16:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
},
{
"pci": "0000:19:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
}
],
"started_at": "2026-10-22T18:33:31.270581575Z",
"finished_at": "2026-10-22T19:16:28.568820286Z",
"closed_at": "2026-10-22T19:16:28.715957+00:00",
"kernel_fault_matches": [],
"startup_requests_completed_and_drained": 3,
"official_phase_samples": 11,
"complete_ladders": 1,
"strict_valid": true,
"official_fixed_2000_tps": 32.806665219045115,
"qualifying_ladders": 0,
"FP16_qualification": false,
"baseline_kind": "RTN4"
},
"archive": {
"path": "/home/ai/vnext-v620-qwen38-rtn4-tp2-service-r0-20261022/raw-evidence.tar.gz",
"bytes": 734829,
"sha256": "782012dac7c921179e01dc090de11770841eac6eb853d3964012cc12115c269b",
"files": 236,
"index_sha256": "285267ead1cb3460651e70de1e025fadc91e9f3dda778f524f434eb997cb3979"
},
"independent_review": {
"path": ".work/helpers/projection-analyst/.work/native-mtp-review/TP2-sealed/review.json",
"SHA256": "37f2bbad942bf6980f3827dae33d70447bbae553a64733d4b680b4bcfc82c484"
},
"FP16_qualification": false,
"limits": "One sample per tier; historical TP4 cache/request history differs, so observed TP difference is not isolated causal attribution or repeatability. Context configured131072; runtime-reported capacity limitation retained in deployment observation.",
"local_evidence": ".work/rtn4-tp-mtp-matrix/tp2-service-r0/sealed-mirror/retained/evidence"
},
"detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
"detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
"detail_source_pointer": "local_RTN4_TP_MTP_matrix.cells[0]",
"sampled_VRAM_startup_max_bytes_by_rank": [
13639901184,
13642076160
],
"memory_accounting_limit": "Retained own-device observation after admission; not a guaranteed continuous peak.",
"memory_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#local_RTN4_TP_MTP_matrix",
"memory_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
"retest_strict_tokens": 8472,
"retest_MTP_accounting_by_tier": {},
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 13639860224,
"1": 13642035200
},
"retest_memory_observation_count": 33,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-w4-tp2-mtp0/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 1.05,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 1.05,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 7840,
"retest_memory_log_report": {
"profile": "v620-w4-tp2-mtp0",
"image": "sha256:a16a6f2d47e7ac6ffc55d7a3d9298cad7e03601d8a30886421e905c92aeae386",
"container": "63d508581fc91c4a06d201f732ab17b0e41917d68c06a98af7a60049e74f0225",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 25.59,
"retest_logged_model_load_seconds": 114.199732,
"retest_logged_available_KV_GiB": 1.05,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 1.05,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 7840,
"retest_logged_context_capacity_x": 0.25,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w4-tp2-mtp0/service-final.log",
"source_log_sha256": "1de293b6b62bc00ad5bad8ca7ae9b8602161b0137af4ffd056b3e51f09b244b3",
"sealed_index_sha256": "d5b582c7896849a6be95b24fdc53dcdd4cecd5c8e248a63f69ae296aee7afd9d",
"source_lines": {
"model_load": {
"line": 92,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 16:17:07 [gpu_model_runner.py:4579] Model loading took 25.59 GiB memory and 114.199732 seconds"
},
"automatic_KV_budget": {
"line": 104,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-26 16:20:42 [gpu_worker.py:453] Available KV cache memory: 1.05 GiB"
},
"grouped_KV_tokens": {
"line": 107,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 16:20:42 [kv_cache_utils.py:1314] GPU KV cache size: 7,840 tokens"
},
"context_capacity": {
"line": 108,
"text": "(EngineCore_DP0 pid=65) INFO 10-26 16:20:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 0.25x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w4-tp2-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 12.705135345458984,
"sampled_peak_VRAM_GiB": 12.70517349243164
} |
| V620Current Qwen3.8 | 4-bit RTNQwen/Qwen3.8-27B | TP4MTP22K accepted 1091 / 1820 (59.95%) | PASS8141 tokens | 39.708 | 27.714 | 1.125 / 51.50510K: 1.124 / 361.961Strict: 1.126 / 260.606 | 3.00 / 28.68packed target/draft projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:322972df1a8ebf776e497ff3cfecad1ac2a2576d9fb72541c52e22212de6ffc7Runtime image config: sha256:762cae7dd32e7f3542a57754b547742471c371a40d287a115ac349616ec64681 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256 mkdir -p launch-tools-v620-serve-r5 tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp4-mtp2 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w4-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Preflight cleanup compatibilityAfter its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged. Retained lab Docker command · not portabledocker run -d \
--name serve-v620-w4-tp4-mtp2-20261028 \
--network host \
--ipc host \
--cpuset-cpus 0-7,16-23 \
--cpuset-mems 0 \
--security-opt label=disable \
--shm-size 67108864 \
--device /dev/kfd:/dev/kfd:rwm \
--device /dev/dri/card5:/dev/dri/card5:rwm \
--device /dev/dri/card6:/dev/dri/card6:rwm \
--device /dev/dri/card7:/dev/dri/card7:rwm \
--device /dev/dri/card8:/dev/dri/card8:rwm \
--device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
--device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
--device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
--device /dev/dri/renderD136:/dev/dri/renderD136:rwm \
--user root \
--workdir /tmp \
-e VLLM_DISABLE_COMPILE_CACHE=1 \
-e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
-e ROCM_VERSION=7.14.0 \
-e TORCH_VERSION=2.12.0+rocm7.14.0 \
-e VLLM_TARGET_DEVICE=rocm \
-e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
-e PYTORCH_ROCM_ARCH=gfx1030 \
-e GPU_ARCHS=gfx1030 \
-e AMDGPU_TARGETS=gfx1030 \
-e CMAKE_HIP_ARCHITECTURES=gfx1030 \
-e PYTHONDONTWRITEBYTECODE=1 \
-e PIP_NO_CACHE_DIR=1 \
-e PIP_DISABLE_PIP_VERSION_CHECK=1 \
-e VNEXT_GDN_HEAD32=1 \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
--mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp2/cache/vllm,dst=/root/.cache/vllm \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp2/cache/torch_extensions,dst=/root/.cache/torch_extensions \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp2/cache/.triton,dst=/root/.triton \
--entrypoint /usr/bin/env \
sha256:762cae7dd32e7f3542a57754b547742471c371a40d287a115ac349616ec64681 \
V756F94_BOUNDARY_MODE=posted \
DO_NOT_TRACK=1 \
GPU_DEVICE_ORDINAL=0,1,2,3 \
GPU_MAX_HW_QUEUES=1 \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
OMP_NUM_THREADS=4 \
TEMP=/tmp/torchinductor_root \
TMPDIR=/tmp/torchinductor_root \
TMP=/tmp/torchinductor_root \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
TRITON_CACHE_DIR=/root/.triton \
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
VLLM_GPU_MEMORY_UTILIZATION=0.95 \
VLLM_LOGGING_LEVEL=INFO \
VLLM_PORT=9002 \
VLLM_RDNA_FORCE_FP16=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
NCCL_P2P_DISABLE=0 \
NCCL_ALGO=Tree \
NCCL_PROTO=LL \
NCCL_MIN_NCHANNELS=4 \
NCCL_MAX_NCHANNELS=4 \
NCCL_NTHREADS=256 \
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
HIP_VISIBLE_DEVICES=0,1,2,3 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
ROCR_VISIBLE_DEVICES=0,1,2,3 \
VLLM_CACHE_ROOT=/root/.cache/vllm \
PYTORCH_ROCM_ARCH=gfx1030 \
PYTHONHASHSEED=0 \
/opt/venv/bin/python \
-m \
vllm.vnext_rtn_entry \
serve \
/opt/local-models/qwen38-27b-hf \
--served-model-name \
Qwen/Qwen3.8-27B \
--enable-auto-tool-choice \
--tool-call-parser \
qwen3_coder \
--dtype \
half \
--host \
0.0.0.0 \
--port \
9002 \
--tensor-parallel-size \
4 \
--speculative-config \
'{"method":"mtp","num_speculative_tokens":2}' \
--max-model-len \
131072 \
--gpu-memory-utilization \
0.95 \
--trust-remote-code \
--generation-config \
vllm \
-O=3 \
--reasoning-parser \
qwen3 \
--async-scheduling \
--hf-overrides \
'{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":4,"group_size":128}}' \
--language-model-only \
--max-num-seqs \
2 \
--max-num-batched-tokens \
2048 \
--mamba-ssm-cache-dtype \
float32 \
--no-enable-prefix-caching \
--no-enforce-eager \
--attention-backend \
TRITON_ATTN \
--disable-custom-all-reduce \
--seed \
0 \
--worker-cls \
vllm.vnext_rtn_worker.Worker \
--quantization \
vnext_rtnHistorical host paths and device addresses are retained as evidence. Use the serving command above.Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-r4.tar.gz.sha256 mkdir -p launch-tools-v620-r4 tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp4-mtp2 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-v620-r4/portable/launch.py --config v620-w4-tp4-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-v620-r4.tar.gz · SHA-256 launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. Portable observer compatibilityThe evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged. | All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict FAIL2K 42.203 · diagnostic / ineligible10K 29.563 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 9075,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 269.3403684841469,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T03:35:53Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.012874047039076686,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 15946,
"text_chars": 15946,
"text_sha256": "1c2824a1e900e1f1c7b125b67ac45cbbe30faba1360283e1208dd506b70850cb",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 268.0994095131755,
"vllm_decode_tps": 33.84938451180743,
"vllm_generation_tokens_delta": 9075.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.2195868210401386,
"vllm_prefill_tps": 387.8362670372221,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 33.693426837849394
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 48.53436310100369,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T03:36:44Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.006693734088912606,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 8781,
"text_chars": 8781,
"text_sha256": "5096b049167f360900c93aca8b79452d6b9cfaab26f2fd552a024bf16485afa0",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 47.39049668004736,
"vllm_decode_tps": 42.20255410072653,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.125919857993722,
"vllm_prefill_tps": 420.1009482529594,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 41.207916869906136
},
"10K": {
"completion_tokens": 10000,
"content_chars": 17340,
"content_has_think_close": true,
"elapsed_seconds": 339.4023382640444,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T03:42:23Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.018082647817209363,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 11257,
"qwen_answer_text_sha256": "1c25c1dd1cc8d000f9c74c3cf23b6a54811153dc389a271e367bea0a1035cc20",
"qwen_gate_close_in_content": true,
"qwen_gate_close_in_raw": true,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 11257,
"qwen_visible_text_sha256": "1c25c1dd1cc8d000f9c74c3cf23b6a54811153dc389a271e367bea0a1035cc20",
"raw_has_think_close": true,
"reasoning_chars": 30737,
"text_chars": 48078,
"text_sha256": "26508a10a01e51cdd70e874bc2518f39a0e0a445e04b6f15d265a29417caeef4",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 338.26377589209005,
"vllm_decode_tps": 29.562728003101675,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.1227409939747304,
"vllm_prefill_tps": 421.2903978196113,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 29.463556589348872
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestMTP counter replayPASS12,049 natural tokens2K 50.03810K 29.474 · diagnostic2K MTP accepted 1247 / 1506 (82.80%) Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T15:18:34.551273762Z",
"FinishedAt": "2026-10-26T16:11:20.878215369Z"
},
"devices_after": [
{
"card": 5,
"render": 133,
"pci": "0000:16:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:19:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 7,
"render": 135,
"pci": "0000:1c:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 8,
"render": 136,
"pci": "0000:1f:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
}
],
"finished": "2026-10-26T16:11:21.024146+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 50.03782778839164,
"retest_10K_TPS": 29.473873646089725,
"retest_strict_tokens": 12049,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 12049,
"elapsed_seconds": 627.4919198229909,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 626.3237443990074,
"vllm_decode_tps": 19.237654819492256,
"vllm_generation_tokens_delta": 12049.0,
"vllm_prefill_seconds_delta": 1.1552683959016576,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 41.104696169961244,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 39.969760647043586,
"vllm_decode_tps": 50.03782778839164,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 1.1210625360254198,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 340.4189020500053,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 339.2835336161079,
"vllm_decode_tps": 29.473873646089725,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 1.1211414389545098,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 1.1210625360254198,
"retest_2K_total_seconds": 41.104696169961244,
"retest_10K_prefill_seconds": 1.1211414389545098,
"retest_10K_total_seconds": 340.4189020500053,
"retest_strict_prefill_seconds": 1.1552683959016576,
"retest_strict_total_seconds": 627.4919198229909,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 7368.0,
"vllm:spec_decode_num_accepted_tokens_total": 7368.0,
"vllm:spec_decode_num_draft_tokens_total": 9360.0,
"vllm:generation_tokens_total": 12049.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:request_generation_tokens_sum": 12049.0,
"vllm:spec_decode_num_drafts_total": 4680.0
},
"accepted_per_position": {
"0": 3987.0,
"1": 3381.0
},
"accepted_fraction": 0.7871794871794872,
"committed_emitted_tokens": 12049.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "58b60fc09292dc78a04103d15cc41f6c3819f438f8ac6ee3f0c3138210f80a55",
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "4743b077a4308e60d98e2fed918ad8f39a0db6c484f585a10c15203b655627e8"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 1247.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_draft_tokens_total": 1506.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1247.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:spec_decode_num_drafts_total": 753.0,
"vllm:request_generation_tokens_count": 1.0
},
"accepted_per_position": {
"0": 657.0,
"1": 590.0
},
"accepted_fraction": 0.8280212483399735,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "6305a1606c4c60bcf66a5e25d6687e417b165acded425de5114a1a3794b48b8c",
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "9cfd84f27f62d6ed382652c0b7f2037bdc495760cb18bda854876e88093e84e5"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:spec_decode_num_drafts_total": 3887.0,
"vllm:spec_decode_num_accepted_tokens_total": 6112.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6112.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_draft_tokens_total": 7774.0
},
"accepted_per_position": {
"0": 3320.0,
"1": 2792.0
},
"accepted_fraction": 0.7862104450733213,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "908c5b8ee8fecb4e947e3a3fd4ea2d996c36f0608e307d3d7320d069de9912c7",
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "a2ef83da43f40f2d0a0a94a6cf25ed6a3b46325cdc00aaf620f9aaf5fc322cc3"
}
}
},
"retest_MTP_2K_accepted": 1247.0,
"retest_MTP_2K_proposed": 1506.0,
"retest_MTP_2K_draft_batches": 753.0,
"retest_MTP_2K_acceptance_percent": 82.80212483399735,
"retest_loaded_admission": {
"all_four_quantized_module_receipts_passed": true,
"all_four_MTP_draft_receipts_passed": true,
"draft_sharing_observations": [
{
"rank": 0,
"shared_embedding": true,
"shared_lm_head": true
},
{
"rank": 3,
"shared_embedding": true,
"shared_lm_head": true
},
{
"rank": 1,
"shared_embedding": true,
"shared_lm_head": true
},
{
"rank": 2,
"shared_embedding": true,
"shared_lm_head": true
}
],
"MTP_execution_evidence": "See separately retained standard speculative counter deltas; load/source observations alone are not invocation proof",
"generated_quantized_call_support_complete": true,
"graphs": [
{
"rank": 0,
"pid": 103,
"source_sha256": "acb7fa3377c9627dad430edf5a9e679e6d929f8b057350e0c6139f8cbb9bb484",
"generated_gemm_calls": 256
},
{
"rank": 1,
"pid": 104,
"source_sha256": "be1dd3311f27d998762cbe71f0e857aa5baf8fa893a059bfbc1c20c597a876c2",
"generated_gemm_calls": 256
},
{
"rank": 2,
"pid": 105,
"source_sha256": "bfff37de3c45381b038055a2dd7d544c6dbe07f5339fc55d6aa6d67d553c5419",
"generated_gemm_calls": 256
},
{
"rank": 3,
"pid": 106,
"source_sha256": "5520ef1773b673749e04ee7e6584940fe22dc49779fd00d3eb66cda74d80c541",
"generated_gemm_calls": 256
}
],
"attention_KV_observations": [
{
"rank": 1,
"pid": 104,
"attention_layers": [
3,
7,
11,
15,
19,
23,
27,
31,
35,
39,
43,
47,
51,
55,
59,
63
],
"logical_view_bytes": 21456486400,
"mtp_draft_attention_KV": [
{
"name": "model.layers.0.self_attn.attn",
"tensors": [
{
"shape": [
1637,
2,
800,
1,
256
],
"dtype": "torch.float16",
"device": "cuda:1",
"bytes": 1341030400
}
]
}
],
"Mamba_SSM_config_dtype": "float32",
"limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
},
{
"rank": 0,
"pid": 103,
"attention_layers": [
3,
7,
11,
15,
19,
23,
27,
31,
35,
39,
43,
47,
51,
55,
59,
63
],
"logical_view_bytes": 21456486400,
"mtp_draft_attention_KV": [
{
"name": "model.layers.0.self_attn.attn",
"tensors": [
{
"shape": [
1637,
2,
800,
1,
256
],
"dtype": "torch.float16",
"device": "cuda:0",
"bytes": 1341030400
}
]
}
],
"Mamba_SSM_config_dtype": "float32",
"limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
},
{
"rank": 2,
"pid": 105,
"attention_layers": [
3,
7,
11,
15,
19,
23,
27,
31,
35,
39,
43,
47,
51,
55,
59,
63
],
"logical_view_bytes": 21456486400,
"mtp_draft_attention_KV": [
{
"name": "model.layers.0.self_attn.attn",
"tensors": [
{
"shape": [
1637,
2,
800,
1,
256
],
"dtype": "torch.float16",
"device": "cuda:2",
"bytes": 1341030400
}
]
}
],
"Mamba_SSM_config_dtype": "float32",
"limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
},
{
"rank": 3,
"pid": 106,
"attention_layers": [
3,
7,
11,
15,
19,
23,
27,
31,
35,
39,
43,
47,
51,
55,
59,
63
],
"logical_view_bytes": 21456486400,
"mtp_draft_attention_KV": [
{
"name": "model.layers.0.self_attn.attn",
"tensors": [
{
"shape": [
1637,
2,
800,
1,
256
],
"dtype": "torch.float16",
"device": "cuda:3",
"bytes": 1341030400
}
]
}
],
"Mamba_SSM_config_dtype": "float32",
"limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
}
],
"all_four_KV_receipts_passed": true,
"limits": "Static generated call support plus bound loaded-module receipts/native mappings and completed startup. Not a GPU completion trace or correctness verdict."
},
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
"retest_logged_model_load_delta_GiB": 5.51,
"retest_logged_model_load_seconds": 60.300757,
"retest_logged_available_KV_GiB": 21.24,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 21.24,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 327200,
"retest_logged_context_capacity_x": 9.46,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "v620-w4-tp4-mtp2",
"image": "sha256:762cae7dd32e7f3542a57754b547742471c371a40d287a115ac349616ec64681",
"container": "cb24d05f08e6f9ae6d34cb9601d716a131da7448c1b3cb35342c550e9caedbdd",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 5.51,
"retest_logged_model_load_seconds": 60.300757,
"retest_logged_available_KV_GiB": 21.24,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 21.24,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 327200,
"retest_logged_context_capacity_x": 9.46,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w4-tp4-mtp2/service-final.log",
"source_log_sha256": "0807c48469b50f3cc209a1eaa5f6f67d3306b2fc2d612248ef1f5e64527c26a5",
"sealed_index_sha256": "fe7ba6674f7a15c60fd9db595447d9b2e0bb7728107d4990e709736e59b41617",
"source_lines": {
"model_load": {
"line": 154,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 15:21:33 [gpu_model_runner.py:4598] Model loading took 5.51 GiB memory and 60.300757 seconds"
},
"automatic_KV_budget": {
"line": 223,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 15:38:24 [gpu_worker.py:453] Available KV cache memory: 21.24 GiB"
},
"grouped_KV_tokens": {
"line": 228,
"text": "(EngineCore_DP0 pid=85) INFO 10-26 15:38:24 [kv_cache_utils.py:1314] GPU KV cache size: 327,200 tokens"
},
"context_capacity": {
"line": 229,
"text": "(EngineCore_DP0 pid=85) INFO 10-26 15:38:24 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 9.46x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w4-tp4-mtp2/memory-log-report.json"
} | Evidence and limitscompleted matrix cell {
"strict_tokens": 8141,
"source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.local_RTN4_TP_MTP_matrix.ordered_cells[TP=4,MTP_tokens=2].result",
"source_revision": "1e540c9b0c93a8a336dc7862d01d05f033971cca",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"recorded_configuration_details": {
"strict_valid": true,
"strict_tokens": 8141,
"strict_diagnostic_TPS": 31.37568371725553,
"official_fixed2K_TPS": 39.708142381881636,
"diagnostic_fixed10K_TPS": 27.71436282221888,
"source": "1e540c9b0c93a8a336dc7862d01d05f033971cca",
"image": "sha256:762cae7dd32e7f3542a57754b547742471c371a40d287a115ac349616ec64681",
"manifest": "8b48dab1371cf69d2a1ac70be2c6194297fcb1db13d28b180b767bba59e90dc6",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"TP": 4,
"MTP_tokens": 2,
"sample_count_per_tier": 1,
"metrics": {
"strict": {
"completion_tokens": 8141,
"elapsed_seconds": 260.6062123499869,
"vllm_prefill_seconds_delta": 1.1256862390146125,
"vllm_decode_seconds_delta": 259.4684493049863,
"vllm_decode_tps": 31.37568371725553,
"vllm_prompt_tokens_delta": 473.0,
"vllm_generation_tokens_delta": 8141.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"server_e2e_seconds": 260.6011629104614
},
"fixed2K": {
"completion_tokens": 2000,
"elapsed_seconds": 51.50504913899931,
"vllm_prefill_seconds_delta": 1.1254832400009036,
"vllm_decode_seconds_delta": 50.367503489978844,
"vllm_decode_tps": 39.708142381881636,
"vllm_prompt_tokens_delta": 473.0,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"server_e2e_seconds": 51.500343799591064
},
"fixed10K": {
"completion_tokens": 10000,
"elapsed_seconds": 361.96059344001696,
"vllm_prefill_seconds_delta": 1.1244875820120797,
"vllm_decode_seconds_delta": 360.82373836799525,
"vllm_decode_tps": 27.71436282221888,
"vllm_prompt_tokens_delta": 473.0,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"server_e2e_seconds": 361.9555811882019
}
},
"MTP_fixed2K_accounting": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 1091.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_draft_tokens_total": 1820.0,
"vllm:spec_decode_num_drafts_total": 910.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1091.0
},
"accepted_per_position": {
"0": 658.0,
"1": 433.0
},
"accepted_fraction": 0.5994505494505494,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/vnext-v620-qwen38-rtn4-tp4-mtp2-service-r4-20261022/ladder1/c1_2000_proxy/metrics-before.txt": "edb17f0e277b98783f4f4c340c6ffe8b0dcb544ebbe326b7b8c6f53725d91d2f",
"/home/ai/vnext-v620-qwen38-rtn4-tp4-mtp2-service-r4-20261022/ladder1/c1_2000_proxy/metrics-after.txt": "356fe17092373cc916f1ac374c0753eb42dfb7dff341ca2e59be20a99dfca3c6"
}
},
"loaded_packed_bytes_per_rank": {
"target": 3162718720,
"draft": 55157760,
"total": 3217876480
},
"maximum_sampled_VRAM_bytes_during_loaded_serving": {
"0000:16:00.0": 30775201792,
"0000:19:00.0": 30792024064,
"0000:1c:00.0": 30792175616,
"0000:1f:00.0": 30741901312
},
"VRAM_limit": "Periodic own-device observations; not a continuous peak. Includes runtime/KV and cannot be equated to packed weight storage.",
"actual_KV": {
"path": ".work/rtn4-tp-mtp-matrix/tp4-mtp2-service-r4/KV-observation",
"review_path": ".work/helpers/projection-analyst/.work/native-mtp-review/TP4-r4-KV-observation-review.json",
"review_SHA256": "85781eaf749586e0c67d931df073cc977768f8fa57e350068de5b6b3cc38804f",
"target_views_per_rank": 16,
"draft_views_per_rank": 1,
"dtype": "torch.float16",
"shape": [
1636,
2,
800,
1,
256
],
"logical_bytes_per_view": 1340211200,
"allocated_bytes_by_rank": [
27974586368,
27974586368,
27974586368,
27974717440
],
"reserved_bytes_by_rank": [
28730982400,
28730982400,
28730982400,
28730982400
],
"runtime_KV_GiB": 21.27,
"runtime_KV_tokens": 327200,
"runtime_max_concurrency_at_131072": 9.46,
"SSM_config_dtype": "float32",
"limits": "Logical views may alias and are not physical allocation; hybrid cache estimates are not full resident-context proof. Actual completed native MTP counters and strict-valid inference are retained separately in this result."
},
"precision": {
"target_parameter_dtype_counts": {
"torch.float16": 611,
"torch.int32": 768
},
"draft_parameter_dtype_counts": {
"torch.int32": 15,
"torch.float16": 14
},
"draft_class": "vllm.model_executor.models.qwen3_5_mtp.Qwen3_5MTP",
"shared_embedding": true,
"shared_lm_head": true,
"exclusions": "LM head/embedding, norms, convolution, recurrent parameters retain FP16; Mamba SSM cache retains original float32",
"attention_KV": "FP16",
"SSM_configuration": "float32",
"BF16_observed": false
},
"historical_TP4_off_comparison": {
"official_fixed2K_TPS": 40.235022545409606,
"fixed2K_delta_percent": -1.3095063210995517,
"diagnostic_fixed10K_TPS": 38.55185454344116,
"fixed10K_delta_percent": -28.111466619615744,
"limits": "One sample per cell. Historical code-cache seed/request history differs from fresh-cache matrix. MTP adds head and requires800-block existing Triton fallback for both target/draft; historical TP4off uses784 native attention. Cache budget also differs; no isolated causal or repeatability claim."
},
"closure": {
"container_id": "e80cfd8a3ef2b42799e6c77c83ca38b5f96ea60f0dbfec5435aee66e74a0e75e",
"image_id": "sha256:762cae7dd32e7f3542a57754b547742471c371a40d287a115ac349616ec64681",
"exit_code": 0,
"OOMKilled": false,
"restart_count": 0,
"bank_idle": true,
"devices": [
{
"pci": "0000:16:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
},
{
"pci": "0000:19:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
},
{
"pci": "0000:1c:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
},
{
"pci": "0000:1f:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
}
],
"started_at": "2026-10-22T21:30:17.277864186Z",
"finished_at": "2026-10-22T22:14:55.056663491Z",
"closed_at": "2026-10-22T22:14:55.216256+00:00",
"kernel_fault_matches": [],
"startup_requests_completed_and_drained": 3,
"official_phase_samples": 11,
"complete_ladders": 1,
"strict_valid": true,
"official_fixed_2000_tps": 39.708142381881636,
"qualifying_ladders": 0,
"FP16_qualification": false,
"baseline_kind": "RTN4"
},
"archive": {
"path": "/home/ai/vnext-v620-qwen38-rtn4-tp4-mtp2-service-r4-20261022/raw-evidence.tar.gz",
"bytes": 1148107,
"sha256": "88dc5e837386b24d30eadde8175b601fa14b4b5231172aa0a3f47920ea43e977",
"files": 248,
"index_sha256": "e0768f050d92ebd38b6dc6be432f8642e0046985728fca0dd6a119e41a8b62a4"
},
"evidence": ".work/rtn4-tp-mtp-matrix/tp4-mtp2-service-r4/sealed-mirror/retained/evidence",
"all_248_files_verified": true,
"independent_review": {
"path": ".work/helpers/projection-analyst/.work/native-mtp-review/TP4-MTP-sealed/review.json",
"SHA256": "0cf11e9c9122a2559eb8a9e01a929123af46651dafec7711b10476e7eb879c05",
"decision": "One strict-valid completed TP4 RTN4 native-MTP2 ladder; official2K observation, no repeatability or FP16 qualification.",
"unchanged_CPU_auditors_all_passed": true,
"binding_observation_correction": {
"only_binding_field_changed": "maps_sha256",
"explanation": "Complete process mapping text changes across execution; selected mapped library identities and worker starts are unchanged. Initial overly broad whole-binding-equality assertion failed on this field only; no evidence discrepancy."
}
},
"FP16_qualification": false,
"accounting": {
"startup": 3,
"warmups": 8,
"strict": 1,
"fixed": 2,
"requests": 14,
"prompt": 6622,
"generated": 42141,
"drained": true
},
"per_tier_MTP": {
"strict": {
"draft_batches": 3226.0,
"proposed": 6452.0,
"accepted": 4914.0,
"accepted_per_position": {
"0": 2688.0,
"1": 2226.0
},
"committed": 8141.0,
"request_tokens_agree": true,
"accepted_fraction": 0.7616243025418475,
"committed_per_reported_spec_batch": 2.523558586484811,
"audit_sha256": "8338b2b0a35c39d728de6e8a2631c8b8c7ca5990d002a0a8b82e03e48db53dda"
},
"official2K": {
"draft_batches": 910.0,
"proposed": 1820.0,
"accepted": 1091.0,
"accepted_per_position": {
"0": 658.0,
"1": 433.0
},
"committed": 2000.0,
"request_tokens_agree": true,
"accepted_fraction": 0.5994505494505494,
"committed_per_reported_spec_batch": 2.197802197802198,
"audit_sha256": "59da4cc5801af8b8f73ea453308b2ad6333e52031ad5f675411f4c014d683c76"
},
"diagnostic10K": {
"draft_batches": 4056.0,
"proposed": 8112.0,
"accepted": 5943.0,
"accepted_per_position": {
"0": 3293.0,
"1": 2650.0
},
"committed": 10000.0,
"request_tokens_agree": true,
"accepted_fraction": 0.7326183431952663,
"committed_per_reported_spec_batch": 2.465483234714004,
"audit_sha256": "679f2a8b18c045c5ef6ae506b33a716eb60f084145da9fafd293fb8761fee7bb"
}
},
"MTP_ratio_limits": "Committed tokens per reported speculative batch only; not all scheduler/GPU/decode steps. Acceptance precedes stop trimming. Proposed/rejected drafts never enter speed numerator."
},
"MTP_accounting_by_tier": {
"strict": {
"draft_batches": 3226.0,
"proposed": 6452.0,
"accepted": 4914.0,
"accepted_per_position": {
"0": 2688.0,
"1": 2226.0
},
"committed": 8141.0,
"request_tokens_agree": true,
"accepted_fraction": 0.7616243025418475,
"committed_per_reported_spec_batch": 2.523558586484811,
"audit_sha256": "8338b2b0a35c39d728de6e8a2631c8b8c7ca5990d002a0a8b82e03e48db53dda"
},
"official2K": {
"draft_batches": 910.0,
"proposed": 1820.0,
"accepted": 1091.0,
"accepted_per_position": {
"0": 658.0,
"1": 433.0
},
"committed": 2000.0,
"request_tokens_agree": true,
"accepted_fraction": 0.5994505494505494,
"committed_per_reported_spec_batch": 2.197802197802198,
"audit_sha256": "59da4cc5801af8b8f73ea453308b2ad6333e52031ad5f675411f4c014d683c76"
},
"diagnostic10K": {
"draft_batches": 4056.0,
"proposed": 8112.0,
"accepted": 5943.0,
"accepted_per_position": {
"0": 3293.0,
"1": 2650.0
},
"committed": 10000.0,
"request_tokens_agree": true,
"accepted_fraction": 0.7326183431952663,
"committed_per_reported_spec_batch": 2.465483234714004,
"audit_sha256": "679f2a8b18c045c5ef6ae506b33a716eb60f084145da9fafd293fb8761fee7bb"
}
},
"MTP_2K_draft_batches": 910.0,
"MTP_2K_proposed": 1820.0,
"MTP_2K_accepted": 1091.0,
"MTP_2K_acceptance_percent": 59.94505494505494,
"MTP_2K_committed": 2000.0,
"detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
"detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
"detail_source_pointer": "local_RTN4_TP_MTP_matrix.cells[1]",
"sampled_peak_VRAM_bytes_by_PCI": {
"0000:16:00.0": 30775201792,
"0000:19:00.0": 30792024064,
"0000:1c:00.0": 30792175616,
"0000:1f:00.0": 30741901312
},
"MTP_fixed2K_accounting": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 1091.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_draft_tokens_total": 1820.0,
"vllm:spec_decode_num_drafts_total": 910.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1091.0
},
"accepted_per_position": {
"0": 658.0,
"1": 433.0
},
"accepted_fraction": 0.5994505494505494,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/vnext-v620-qwen38-rtn4-tp4-mtp2-service-r4-20261022/ladder1/c1_2000_proxy/metrics-before.txt": "edb17f0e277b98783f4f4c340c6ffe8b0dcb544ebbe326b7b8c6f53725d91d2f",
"/home/ai/vnext-v620-qwen38-rtn4-tp4-mtp2-service-r4-20261022/ladder1/c1_2000_proxy/metrics-after.txt": "356fe17092373cc916f1ac374c0753eb42dfb7dff341ca2e59be20a99dfca3c6"
}
},
"memory_accounting_limit": "Periodic own-device observations; not a continuous peak. Includes runtime/KV and cannot be equated to packed weight storage.",
"memory_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#local_RTN4_TP_MTP_matrix",
"memory_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
"retest_strict_tokens": 12049,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 7368.0,
"vllm:spec_decode_num_accepted_tokens_total": 7368.0,
"vllm:spec_decode_num_draft_tokens_total": 9360.0,
"vllm:generation_tokens_total": 12049.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:request_generation_tokens_sum": 12049.0,
"vllm:spec_decode_num_drafts_total": 4680.0
},
"accepted_per_position": {
"0": 3987.0,
"1": 3381.0
},
"accepted_fraction": 0.7871794871794872,
"committed_emitted_tokens": 12049.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "58b60fc09292dc78a04103d15cc41f6c3819f438f8ac6ee3f0c3138210f80a55",
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "4743b077a4308e60d98e2fed918ad8f39a0db6c484f585a10c15203b655627e8"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 1247.0,
"vllm:request_generation_tokens_sum": 2000.0,
"vllm:spec_decode_num_draft_tokens_total": 1506.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1247.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:spec_decode_num_drafts_total": 753.0,
"vllm:request_generation_tokens_count": 1.0
},
"accepted_per_position": {
"0": 657.0,
"1": 590.0
},
"accepted_fraction": 0.8280212483399735,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "6305a1606c4c60bcf66a5e25d6687e417b165acded425de5114a1a3794b48b8c",
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "9cfd84f27f62d6ed382652c0b7f2037bdc495760cb18bda854876e88093e84e5"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:request_generation_tokens_count": 1.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:spec_decode_num_drafts_total": 3887.0,
"vllm:spec_decode_num_accepted_tokens_total": 6112.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6112.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_draft_tokens_total": 7774.0
},
"accepted_per_position": {
"0": 3320.0,
"1": 2792.0
},
"accepted_fraction": 0.7862104450733213,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "908c5b8ee8fecb4e947e3a3fd4ea2d996c36f0608e307d3d7320d069de9912c7",
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "a2ef83da43f40f2d0a0a94a6cf25ed6a3b46325cdc00aaf620f9aaf5fc322cc3"
}
}
},
"retest_MTP_2K_accepted": 1247.0,
"retest_MTP_2K_proposed": 1506.0,
"retest_MTP_2K_draft_batches": 753.0,
"retest_MTP_2K_acceptance_percent": 82.80212483399735,
"retest_logged_available_KV_GiB": 21.24,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 21.24,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 327200,
"retest_memory_log_report": {
"profile": "v620-w4-tp4-mtp2",
"image": "sha256:762cae7dd32e7f3542a57754b547742471c371a40d287a115ac349616ec64681",
"container": "cb24d05f08e6f9ae6d34cb9601d716a131da7448c1b3cb35342c550e9caedbdd",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 5.51,
"retest_logged_model_load_seconds": 60.300757,
"retest_logged_available_KV_GiB": 21.24,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 21.24,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 327200,
"retest_logged_context_capacity_x": 9.46,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w4-tp4-mtp2/service-final.log",
"source_log_sha256": "0807c48469b50f3cc209a1eaa5f6f67d3306b2fc2d612248ef1f5e64527c26a5",
"sealed_index_sha256": "fe7ba6674f7a15c60fd9db595447d9b2e0bb7728107d4990e709736e59b41617",
"source_lines": {
"model_load": {
"line": 154,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 15:21:33 [gpu_model_runner.py:4598] Model loading took 5.51 GiB memory and 60.300757 seconds"
},
"automatic_KV_budget": {
"line": 223,
"text": "(Worker pid=103) (Worker_TP0 pid=103) INFO 10-26 15:38:24 [gpu_worker.py:453] Available KV cache memory: 21.24 GiB"
},
"grouped_KV_tokens": {
"line": 228,
"text": "(EngineCore_DP0 pid=85) INFO 10-26 15:38:24 [kv_cache_utils.py:1314] GPU KV cache size: 327,200 tokens"
},
"context_capacity": {
"line": 229,
"text": "(EngineCore_DP0 pid=85) INFO 10-26 15:38:24 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 9.46x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w4-tp4-mtp2/memory-log-report.json",
"sampled_peak_VRAM_GiB": 28.677448272705078
} |
| V620Current Qwen3.8 | 4-bit RTNQwen/Qwen3.8-27B | TP2MTP22K accepted 1104 / 1790 (61.68%) | PASS7252 tokens | 36.427 | 25.228 | 0.774 / 55.69210K: 0.772 / 397.171Strict: 0.773 / 239.844 | 5.99 / 26.93packed target/draft projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:81115520e0ed7fc632670d872835ceefd1bea3a1ce7aada4ea7b6035eb04edbaRuntime image config: sha256:7f987d7773de6755f27160795f693fb1374eb42f54f2f322e917432c0886fd84 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256 mkdir -p launch-tools-v620-serve-r5 tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp2-mtp2 GPU_RENDER_NODES=renderD128,renderD129 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w4-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Preflight cleanup compatibilityAfter its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged. Retained lab Docker command · not portabledocker run -d \
--name serve-v620-w4-tp2-mtp2-20261028 \
--network host \
--ipc host \
--cpuset-cpus 0-7,16-23 \
--cpuset-mems 0 \
--security-opt label=disable \
--shm-size 67108864 \
--device /dev/kfd:/dev/kfd:rwm \
--device /dev/dri/card5:/dev/dri/card5:rwm \
--device /dev/dri/card6:/dev/dri/card6:rwm \
--device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
--device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
--user root \
--workdir /tmp \
-e VLLM_DISABLE_COMPILE_CACHE=1 \
-e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
-e ROCM_VERSION=7.14.0 \
-e TORCH_VERSION=2.12.0+rocm7.14.0 \
-e VLLM_TARGET_DEVICE=rocm \
-e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
-e PYTORCH_ROCM_ARCH=gfx1030 \
-e GPU_ARCHS=gfx1030 \
-e AMDGPU_TARGETS=gfx1030 \
-e CMAKE_HIP_ARCHITECTURES=gfx1030 \
-e PYTHONDONTWRITEBYTECODE=1 \
-e PIP_NO_CACHE_DIR=1 \
-e PIP_DISABLE_PIP_VERSION_CHECK=1 \
-e VNEXT_GDN_HEAD32=1 \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp2/cache/vllm,dst=/root/.cache/vllm \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp2/cache/torch_extensions,dst=/root/.cache/torch_extensions \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp2/cache/.triton,dst=/root/.triton \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp2-mtp2/cache/torchinductor,dst=/tmp/torchinductor_root \
--mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
--entrypoint /usr/bin/env \
sha256:7f987d7773de6755f27160795f693fb1374eb42f54f2f322e917432c0886fd84 \
V756F94_BOUNDARY_MODE=posted \
DO_NOT_TRACK=1 \
GPU_DEVICE_ORDINAL=0,1 \
GPU_MAX_HW_QUEUES=1 \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
OMP_NUM_THREADS=4 \
TEMP=/tmp/torchinductor_root \
TMPDIR=/tmp/torchinductor_root \
TMP=/tmp/torchinductor_root \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
TRITON_CACHE_DIR=/root/.triton \
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
VLLM_GPU_MEMORY_UTILIZATION=0.95 \
VLLM_LOGGING_LEVEL=INFO \
VLLM_PORT=9002 \
VLLM_RDNA_FORCE_FP16=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
NCCL_P2P_DISABLE=0 \
NCCL_ALGO=Tree \
NCCL_PROTO=LL \
NCCL_MIN_NCHANNELS=4 \
NCCL_MAX_NCHANNELS=4 \
NCCL_NTHREADS=256 \
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
HIP_VISIBLE_DEVICES=0,1 \
CUDA_VISIBLE_DEVICES=0,1 \
ROCR_VISIBLE_DEVICES=0,1 \
VLLM_CACHE_ROOT=/root/.cache/vllm \
PYTORCH_ROCM_ARCH=gfx1030 \
PYTHONHASHSEED=0 \
/opt/venv/bin/python \
-m \
vllm.vnext_rtn_entry \
serve \
/opt/local-models/qwen38-27b-hf \
--served-model-name \
Qwen/Qwen3.8-27B \
--enable-auto-tool-choice \
--tool-call-parser \
qwen3_coder \
--dtype \
half \
--host \
0.0.0.0 \
--port \
9002 \
--tensor-parallel-size \
2 \
--speculative-config \
'{"method":"mtp","num_speculative_tokens":2}' \
--max-model-len \
131072 \
--gpu-memory-utilization \
0.95 \
--trust-remote-code \
--generation-config \
vllm \
-O=3 \
--reasoning-parser \
qwen3 \
--async-scheduling \
--hf-overrides \
'{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":4,"group_size":128}}' \
--language-model-only \
--max-num-seqs \
2 \
--max-num-batched-tokens \
2048 \
--mamba-ssm-cache-dtype \
float32 \
--no-enable-prefix-caching \
--no-enforce-eager \
--attention-backend \
TRITON_ATTN \
--disable-custom-all-reduce \
--seed \
0 \
--worker-cls \
vllm.vnext_rtn_worker.Worker \
--quantization \
vnext_rtnHistorical host paths and device addresses are retained as evidence. Use the serving command above.Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-r4.tar.gz.sha256 mkdir -p launch-tools-v620-r4 tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp2-mtp2 GPU_RENDER_NODES=renderD128,renderD129 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-v620-r4/portable/launch.py --config v620-w4-tp2-mtp2 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-v620-r4.tar.gz · SHA-256 launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. Portable observer compatibilityThe evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged. | All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 37.875 · eligible10K 27.890 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 11944,
"content_chars": 6434,
"content_has_think_close": false,
"elapsed_seconds": 479.41922588297166,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T04:58:26Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.022947221994400024,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 6434,
"qwen_answer_text_sha256": "1e3200097878d38d8f28fe4b337f3fc70a55ddb199b09679ee5f03ae93f795f7",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 6434,
"qwen_visible_text_sha256": "1e3200097878d38d8f28fe4b337f3fc70a55ddb199b09679ee5f03ae93f795f7",
"raw_has_think_close": false,
"reasoning_chars": 48599,
"text_chars": 55034,
"text_sha256": "a26575ceb56cd1dc8e86e4d77caf57f02012b0a5b23e81532fe1e46cb3ca1f9d",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 478.62756001902744,
"vllm_decode_tps": 24.954685015474613,
"vllm_generation_tokens_delta": 11944.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.7719826169777662,
"vllm_prefill_tps": 612.7080967855819,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 24.91347729745737
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 53.60688337893225,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T04:59:20Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.01787939900532365,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 10224,
"text_chars": 10224,
"text_sha256": "2bd04ea6e797c0d8b618e371dc199f3ed0c4e852ee8a94d627773b5e456d0365",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 52.80462608905509,
"vllm_decode_tps": 37.87546940730149,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.770544630009681,
"vllm_prefill_tps": 613.8515299159989,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 37.30864161347625
},
"10K": {
"completion_tokens": 10000,
"content_chars": 26224,
"content_has_think_close": true,
"elapsed_seconds": 359.33965503191575,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T05:05:20Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.011819053906947374,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 19549,
"qwen_answer_text_sha256": "bd3e61386f31470e92406a3955ba35684361276a8c308bc35322506677832345",
"qwen_gate_close_in_content": true,
"qwen_gate_close_in_raw": true,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 19549,
"qwen_visible_text_sha256": "bd3e61386f31470e92406a3955ba35684361276a8c308bc35322506677832345",
"raw_has_think_close": true,
"reasoning_chars": 23179,
"text_chars": 49404,
"text_sha256": "86255e1edfe19b984dae99b7aed2fa59484e03d7a4787afe609f6e0bdc7137a1",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 358.54745610407554,
"vllm_decode_tps": 27.89031083544294,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.7714070088695735,
"vllm_prefill_tps": 613.1652870164329,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 27.828823955184745
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestMTP counter replayPASS8,588 natural tokens2K 35.84710K 27.756 · diagnostic2K MTP accepted 1089 / 1820 (59.84%) Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T17:01:27.927094303Z",
"FinishedAt": "2026-10-26T17:46:22.725664239Z"
},
"devices_after": [
{
"card": 5,
"render": 133,
"pci": "0000:16:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:19:00.0",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
}
],
"finished": "2026-10-26T17:46:22.864001+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 35.84675810076804,
"retest_10K_TPS": 27.756224855608366,
"retest_strict_tokens": 8588,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 8588,
"elapsed_seconds": 300.9188194599701,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 300.1355729380157,
"vllm_decode_tps": 28.613735839215575,
"vllm_generation_tokens_delta": 8588.0,
"vllm_prefill_seconds_delta": 0.7710430989973247,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 56.57390893495176,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 55.79305091907736,
"vllm_decode_tps": 35.84675810076804,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 0.7696576459566131,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 361.0622771610506,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 360.2795427700039,
"vllm_decode_tps": 27.756224855608366,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 0.7706899599870667,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 0.7696576459566131,
"retest_2K_total_seconds": 56.57390893495176,
"retest_10K_prefill_seconds": 0.7706899599870667,
"retest_10K_total_seconds": 361.0622771610506,
"retest_strict_prefill_seconds": 0.7710430989973247,
"retest_strict_total_seconds": 300.9188194599701,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:generation_tokens_total": 8588.0,
"vllm:spec_decode_num_draft_tokens_total": 6822.0,
"vllm:request_generation_tokens_sum": 8588.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 5177.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_drafts_total": 3411.0,
"vllm:spec_decode_num_accepted_tokens_total": 5177.0
},
"accepted_per_position": {
"0": 2845.0,
"1": 2332.0
},
"accepted_fraction": 0.7588683670477866,
"committed_emitted_tokens": 8588.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "7bb29d1e750c70d3e30f302f68bce4a3031519b03fbaa7f796ac3ef15f050166",
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "e4a731472fb2f1212b357ecc4d1bee15736d1a42fd7dc0c79956048e43d3c4a2"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 1089.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 1820.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1089.0,
"vllm:spec_decode_num_drafts_total": 910.0,
"vllm:request_generation_tokens_sum": 2000.0
},
"accepted_per_position": {
"0": 653.0,
"1": 436.0
},
"accepted_fraction": 0.5983516483516483,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "71672c9bc8139659a7c942a3d660e713cc869f85dbaa51d989cd9c7a54d0c06e",
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "5f853b7052353c1649940ce9512aab00fab221e37ee9e1b7accfe0196947d95b"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 7686.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6157.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_drafts_total": 3843.0,
"vllm:spec_decode_num_accepted_tokens_total": 6157.0,
"vllm:generation_tokens_total": 10000.0
},
"accepted_per_position": {
"0": 3321.0,
"1": 2836.0
},
"accepted_fraction": 0.8010668748373666,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "22a11ba0d334e8412b04fd0cc21e3c4ad023fd1dcce9a8129d9122ecc9d6d707",
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "08899f1240a16630dd09114338fbfe2ddb3bae95b4a9cf14f0af8266e0d86e9b"
}
}
},
"retest_MTP_2K_accepted": 1089.0,
"retest_MTP_2K_proposed": 1820.0,
"retest_MTP_2K_draft_batches": 910.0,
"retest_MTP_2K_acceptance_percent": 59.83516483516483,
"retest_loaded_admission": {
"all_two_quantized_module_receipts_passed": true,
"all_two_MTP_draft_receipts_passed": true,
"draft_sharing_observations": [
{
"rank": 1,
"shared_embedding": true,
"shared_lm_head": true
},
{
"rank": 0,
"shared_embedding": true,
"shared_lm_head": true
}
],
"MTP_execution_evidence": "See separately retained standard speculative counter deltas; load/source observations alone are not invocation proof",
"generated_quantized_call_support_complete": true,
"graphs": [
{
"rank": 0,
"pid": 102,
"source_sha256": "02e19c8d98c52bfc070e61d7ee0084e9719dfedcc2f7574218332c129643cc0e",
"generated_gemm_calls": 256
},
{
"rank": 1,
"pid": 103,
"source_sha256": "35d844a586d8d59fdeb96ad2d03968cc12dcaa3ce039f08b2bb812afecb08632",
"generated_gemm_calls": 256
}
],
"attention_KV_observations": [
{
"rank": 1,
"pid": 103,
"attention_layers": [
3,
7,
11,
15,
19,
23,
27,
31,
35,
39,
43,
47,
51,
55,
59,
63
],
"logical_view_bytes": 15335424000,
"mtp_draft_attention_KV": [
{
"name": "model.layers.0.self_attn.attn",
"tensors": [
{
"shape": [
585,
2,
800,
2,
256
],
"dtype": "torch.float16",
"device": "cuda:1",
"bytes": 958464000
}
]
}
],
"Mamba_SSM_config_dtype": "float32",
"limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
},
{
"rank": 0,
"pid": 102,
"attention_layers": [
3,
7,
11,
15,
19,
23,
27,
31,
35,
39,
43,
47,
51,
55,
59,
63
],
"logical_view_bytes": 15335424000,
"mtp_draft_attention_KV": [
{
"name": "model.layers.0.self_attn.attn",
"tensors": [
{
"shape": [
585,
2,
800,
2,
256
],
"dtype": "torch.float16",
"device": "cuda:0",
"bytes": 958464000
}
]
}
],
"Mamba_SSM_config_dtype": "float32",
"limits": "Sum of logical tensor view extents; views may alias shared backing storage. Not physical KV allocation bytes."
}
],
"all_two_KV_receipts_passed": true,
"limits": "Static generated call support plus bound loaded-module receipts/native mappings and completed startup. Not a GPU completion trace or correctness verdict."
},
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0, \"c1_128_strict_proxy-mtp\": 0, \"c1_2000_proxy-mtp\": 0, \"c1_10000_proxy-mtp\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30544822272,
"1": 30544805888
},
"retest_memory_observation_count": 45,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-w4-tp2-mtp2/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 10.97,
"retest_logged_model_load_seconds": 106.4305,
"retest_logged_available_KV_GiB": 15.19,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 15.19,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 116800,
"retest_logged_context_capacity_x": 3.38,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "v620-w4-tp2-mtp2",
"image": "sha256:7f987d7773de6755f27160795f693fb1374eb42f54f2f322e917432c0886fd84",
"container": "70e39d1b172d119241eb03dbd6619fcb0c16edb72e4d6c821d31d0d3628f146b",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 10.97,
"retest_logged_model_load_seconds": 106.4305,
"retest_logged_available_KV_GiB": 15.19,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 15.19,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 116800,
"retest_logged_context_capacity_x": 3.38,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w4-tp2-mtp2/service-final.log",
"source_log_sha256": "38a702105b6b0ffe6eca92cfc94fe05a20949d83e51f76d0326d098ca29c1e05",
"sealed_index_sha256": "89aaf41caccca05a57040a8728c276f5acab0ccd3fc718e6700e103b8e71b36e",
"source_lines": {
"model_load": {
"line": 118,
"text": "(Worker pid=102) (Worker_TP0 pid=102) INFO 10-26 17:05:15 [gpu_model_runner.py:4598] Model loading took 10.97 GiB memory and 106.430500 seconds"
},
"automatic_KV_budget": {
"line": 168,
"text": "(Worker pid=102) (Worker_TP0 pid=102) INFO 10-26 17:21:47 [gpu_worker.py:453] Available KV cache memory: 15.19 GiB"
},
"grouped_KV_tokens": {
"line": 172,
"text": "(EngineCore_DP0 pid=84) INFO 10-26 17:21:47 [kv_cache_utils.py:1314] GPU KV cache size: 116,800 tokens"
},
"context_capacity": {
"line": 173,
"text": "(EngineCore_DP0 pid=84) INFO 10-26 17:21:47 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 3.38x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w4-tp2-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.44708251953125
} | Evidence and limitscompleted matrix cell {
"strict_tokens": 7252,
"source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.local_RTN4_TP_MTP_matrix.ordered_cells[TP=2,MTP_tokens=2].result",
"source_revision": "9463306390eb4f2e48cc2b9a2c4f03b52ed2b279",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"recorded_configuration_details": {
"strict_valid": true,
"strict_tokens": 7252,
"strict_diagnostic_TPS": 30.335624585208077,
"official_fixed2K_TPS": 36.42672258849918,
"diagnostic_fixed10K_TPS": 25.227890996166266,
"source": "9463306390eb4f2e48cc2b9a2c4f03b52ed2b279",
"image": "sha256:7f987d7773de6755f27160795f693fb1374eb42f54f2f322e917432c0886fd84",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"TP": 2,
"MTP_tokens": 2,
"sample_count_per_tier": 1,
"metrics": {
"strict": {
"completion_tokens": 7252,
"elapsed_seconds": 239.84428551400197,
"vllm_prefill_seconds_delta": 0.7732716030150186,
"vllm_decode_seconds_delta": 239.0588655799802,
"vllm_decode_tps": 30.335624585208077,
"vllm_prompt_tokens_delta": 473.0,
"vllm_generation_tokens_delta": 7252.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0
},
"fixed2K": {
"completion_tokens": 2000,
"elapsed_seconds": 55.69153628099593,
"vllm_prefill_seconds_delta": 0.774255597993033,
"vllm_decode_seconds_delta": 54.904747335996944,
"vllm_decode_tps": 36.42672258849918,
"vllm_prompt_tokens_delta": 473.0,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"server_e2e_seconds": 55.68696188926697
},
"fixed10K": {
"completion_tokens": 10000,
"elapsed_seconds": 397.1708856849873,
"vllm_prefill_seconds_delta": 0.772096765023889,
"vllm_decode_seconds_delta": 396.386681768985,
"vllm_decode_tps": 25.227890996166266,
"vllm_prompt_tokens_delta": 473.0,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_decode_request_count_delta": 1.0,
"server_e2e_seconds": 397.16564750671387
}
},
"MTP_fixed2K_accounting": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1104.0,
"vllm:spec_decode_num_draft_tokens_total": 1790.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_drafts_total": 895.0,
"vllm:spec_decode_num_accepted_tokens_total": 1104.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_generation_tokens_sum": 2000.0
},
"accepted_per_position": {
"0": 663.0,
"1": 441.0
},
"accepted_fraction": 0.6167597765363129,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/vnext-v620-qwen38-rtn4-tp2-mtp2-service-r2-20261022/ladder1/c1_2000_proxy/metrics-before.txt": "03a95fac30c1499bd5a55e705fc31f7ea7e30dbc77cc4488c26bcf0eb33bd326",
"/home/ai/vnext-v620-qwen38-rtn4-tp2-mtp2-service-r2-20261022/ladder1/c1_2000_proxy/metrics-after.txt": "ca4f7307de7e6e6661190706fdfea725617d84290c6ea1b6098a146b78d92499"
}
},
"loaded_packed_bytes_per_rank": {
"target": 6325437440,
"draft": 110315520,
"total": 6435752960
},
"maximum_sampled_VRAM_bytes_during_loaded_serving": {
"0000:16:00.0": 28912955392,
"0000:19:00.0": 28915073024
},
"VRAM_limit": "Periodic own-device observations; not a continuous peak measurement. Includes runtime/KV and cannot be equated to packed weights.",
"actual_KV": {
"evidence": ".work/rtn4-tp-mtp-matrix/tp2-mtp2-service-r2/kv-audit-observation",
"dtype": "FP16",
"shape_per_target_and_draft_view": [
585,
2,
800,
2,
256
],
"target_views_per_rank": 16,
"draft_views_per_rank": 1,
"available_KV_GiB": 15.19,
"runtime_reported_tokens": 116800,
"runtime_reported_maximum_concurrency_at_131072": 3.38,
"configured_context": 131072,
"HTTP_ready_utc": "2026-10-22T21:05:05.112523Z",
"independent_review": {
"path": ".work/helpers/projection-analyst/.work/native-mtp-review/TP2-r2-KV-observation-addendum.json",
"SHA256": "e96f3a09681c1fe2b5d2e7afc43e5ccf4d3e963216fdcad9da1550e999945921"
},
"limits": "Distinct hybrid runtime estimates; not full resident context proof. Logical views may alias. Cache budget differs from TP2off1.05GiB/7840/0.25x; The original KV observation preceded inference; the completed strict-valid ladder is retained separately in this result."
},
"TP2_off_comparison": {
"official_fixed2K_TPS": 32.806665219045115,
"fixed2K_delta_percent": 11.034517971526503,
"diagnostic_fixed10K_TPS": 30.822545818435916,
"fixed10K_delta_percent": -18.151176918433887,
"limits": "One sample per cell; changed MTP/cache budget and different completion length. No repeatability or isolated causal claim."
},
"closure": {
"container_id": "b413eb7b3441ca6e3d817ee3b55419667627355373c0160876dff95dbff3d4c9",
"image_id": "sha256:7f987d7773de6755f27160795f693fb1374eb42f54f2f322e917432c0886fd84",
"exit_code": 0,
"OOMKilled": false,
"restart_count": 0,
"bank_idle": true,
"devices": [
{
"pci": "0000:16:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
},
{
"pci": "0000:19:00.0",
"vram_used_bytes": 21401600,
"busy_percent": 0,
"dpm": "auto"
}
],
"started_at": "2026-10-22T20:41:17.418699404Z",
"finished_at": "2026-10-22T21:29:41.181240206Z",
"closed_at": "2026-10-22T21:29:41.328370+00:00",
"kernel_fault_matches": [],
"startup_requests_completed_and_drained": 3,
"official_phase_samples": 11,
"complete_ladders": 1,
"strict_valid": true,
"official_fixed_2000_tps": 36.42672258849918,
"qualifying_ladders": 0,
"FP16_qualification": false,
"baseline_kind": "RTN4"
},
"archive": {
"path": "/home/ai/vnext-v620-qwen38-rtn4-tp2-mtp2-service-r2-20261022/raw-evidence.tar.gz",
"bytes": 787728,
"sha256": "fde54375455d96f4963ab3e33dd01a55d58fc2125e7a03b347140d5988ef05ee",
"files": 240,
"index_sha256": "8a576a5a3498120a15bf429ea77a59a394774f2662f6f446fda006b4f7e44d7e"
},
"evidence": ".work/rtn4-tp-mtp-matrix/tp2-mtp2-service-r2/sealed-mirror/retained/evidence",
"all_240_files_verified": true,
"independent_review": {
"path": ".work/helpers/projection-analyst/.work/native-mtp-review/TP2-MTP-sealed/review.json",
"SHA256": "6ac5f5145a81338f728c4df684b8686a73efed2785624497b76a7ca5c7e1da2c",
"decision": "One strict-valid completed TP2 RTN4 native-MTP2 ladder; admitted official2K result, no repeatability or FP16 release qualification"
},
"FP16_qualification": false,
"per_tier_MTP": {
"strict": {
"draft_batches": 2896.0,
"proposed": 5792.0,
"accepted": 4355.0,
"accepted_per_position": {
"0": 2406.0,
"1": 1949.0
},
"committed": 7252.0,
"request_tokens_agree": true,
"accepted_fraction": 0.7518991712707183,
"committed_per_reported_spec_batch": 2.50414364640884,
"audit_sha256": "8904ecfb2e41db11ecb76b952e5913e932ea8fa9f479366b94f473353043cf44"
},
"official2K": {
"draft_batches": 895.0,
"proposed": 1790.0,
"accepted": 1104.0,
"accepted_per_position": {
"0": 663.0,
"1": 441.0
},
"committed": 2000.0,
"request_tokens_agree": true,
"accepted_fraction": 0.6167597765363129,
"committed_per_reported_spec_batch": 2.2346368715083798,
"audit_sha256": "91ce1c75f9f2618df4f28a82be243d8cc33243d19587853e7f4b535e1e7e0f95"
},
"diagnostic10K": {
"draft_batches": 4157.0,
"proposed": 8314.0,
"accepted": 5843.0,
"accepted_per_position": {
"0": 3318.0,
"1": 2525.0
},
"committed": 10000.0,
"request_tokens_agree": true,
"accepted_fraction": 0.7027904738994467,
"committed_per_reported_spec_batch": 2.4055809477988936,
"audit_sha256": "71b2602c5fa4ca02be8a008f0297a9089cd1c7c10fb01ba5ef3b5ace1a1a2ace"
}
},
"MTP_ratio_limits": "2000/895=committed tokens per recorded speculative verification batch only, not all scheduler/GPU steps. Accepted1104 precedes stop trimming; proposals1790 never enter speed numerator.",
"accounting": {
"startup": 3,
"warmups": 8,
"strict": 1,
"fixed": 2,
"requests": 14,
"prompt": 6622,
"generated": 41252,
"drained": true
}
},
"MTP_accounting_by_tier": {
"strict": {
"draft_batches": 2896.0,
"proposed": 5792.0,
"accepted": 4355.0,
"accepted_per_position": {
"0": 2406.0,
"1": 1949.0
},
"committed": 7252.0,
"request_tokens_agree": true,
"accepted_fraction": 0.7518991712707183,
"committed_per_reported_spec_batch": 2.50414364640884,
"audit_sha256": "8904ecfb2e41db11ecb76b952e5913e932ea8fa9f479366b94f473353043cf44"
},
"official2K": {
"draft_batches": 895.0,
"proposed": 1790.0,
"accepted": 1104.0,
"accepted_per_position": {
"0": 663.0,
"1": 441.0
},
"committed": 2000.0,
"request_tokens_agree": true,
"accepted_fraction": 0.6167597765363129,
"committed_per_reported_spec_batch": 2.2346368715083798,
"audit_sha256": "91ce1c75f9f2618df4f28a82be243d8cc33243d19587853e7f4b535e1e7e0f95"
},
"diagnostic10K": {
"draft_batches": 4157.0,
"proposed": 8314.0,
"accepted": 5843.0,
"accepted_per_position": {
"0": 3318.0,
"1": 2525.0
},
"committed": 10000.0,
"request_tokens_agree": true,
"accepted_fraction": 0.7027904738994467,
"committed_per_reported_spec_batch": 2.4055809477988936,
"audit_sha256": "71b2602c5fa4ca02be8a008f0297a9089cd1c7c10fb01ba5ef3b5ace1a1a2ace"
}
},
"MTP_2K_draft_batches": 895.0,
"MTP_2K_proposed": 1790.0,
"MTP_2K_accepted": 1104.0,
"MTP_2K_acceptance_percent": 61.675977653631286,
"MTP_2K_committed": 2000.0,
"detail_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json",
"detail_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
"detail_source_pointer": "local_RTN4_TP_MTP_matrix.cells[2]",
"sampled_peak_VRAM_bytes_by_PCI": {
"0000:16:00.0": 28912955392,
"0000:19:00.0": 28915073024
},
"MTP_fixed2K_accounting": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1104.0,
"vllm:spec_decode_num_draft_tokens_total": 1790.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_drafts_total": 895.0,
"vllm:spec_decode_num_accepted_tokens_total": 1104.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_generation_tokens_sum": 2000.0
},
"accepted_per_position": {
"0": 663.0,
"1": 441.0
},
"accepted_fraction": 0.6167597765363129,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/vnext-v620-qwen38-rtn4-tp2-mtp2-service-r2-20261022/ladder1/c1_2000_proxy/metrics-before.txt": "03a95fac30c1499bd5a55e705fc31f7ea7e30dbc77cc4488c26bcf0eb33bd326",
"/home/ai/vnext-v620-qwen38-rtn4-tp2-mtp2-service-r2-20261022/ladder1/c1_2000_proxy/metrics-after.txt": "ca4f7307de7e6e6661190706fdfea725617d84290c6ea1b6098a146b78d92499"
}
},
"memory_accounting_limit": "Periodic own-device observations; not a continuous peak measurement. Includes runtime/KV and cannot be equated to packed weights.",
"memory_source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#local_RTN4_TP_MTP_matrix",
"memory_source_sha256": "47fe24712908ad85f8ad8acf91c530204c898bb03e3b024ab54812ace6e1b3ca",
"retest_strict_tokens": 8588,
"retest_MTP_accounting_by_tier": {
"strict": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:generation_tokens_total": 8588.0,
"vllm:spec_decode_num_draft_tokens_total": 6822.0,
"vllm:request_generation_tokens_sum": 8588.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 5177.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_drafts_total": 3411.0,
"vllm:spec_decode_num_accepted_tokens_total": 5177.0
},
"accepted_per_position": {
"0": 2845.0,
"1": 2332.0
},
"accepted_fraction": 0.7588683670477866,
"committed_emitted_tokens": 8588.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-before.txt": "7bb29d1e750c70d3e30f302f68bce4a3031519b03fbaa7f796ac3ef15f050166",
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_128_strict_proxy/metrics-after.txt": "e4a731472fb2f1212b357ecc4d1bee15736d1a42fd7dc0c79956048e43d3c4a2"
}
},
"2K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_accepted_tokens_total": 1089.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_draft_tokens_total": 1820.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 1089.0,
"vllm:spec_decode_num_drafts_total": 910.0,
"vllm:request_generation_tokens_sum": 2000.0
},
"accepted_per_position": {
"0": 653.0,
"1": 436.0
},
"accepted_fraction": 0.5983516483516483,
"committed_emitted_tokens": 2000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-before.txt": "71672c9bc8139659a7c942a3d660e713cc869f85dbaa51d989cd9c7a54d0c06e",
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_2000_proxy/metrics-after.txt": "5f853b7052353c1649940ce9512aab00fab221e37ee9e1b7accfe0196947d95b"
}
},
"10K": {
"status": "observed",
"actual_MTP_work_observed": true,
"counter_deltas": {
"vllm:spec_decode_num_draft_tokens_total": 7686.0,
"vllm:request_generation_tokens_count": 1.0,
"vllm:spec_decode_num_accepted_tokens_per_pos_total": 6157.0,
"vllm:request_generation_tokens_sum": 10000.0,
"vllm:spec_decode_num_drafts_total": 3843.0,
"vllm:spec_decode_num_accepted_tokens_total": 6157.0,
"vllm:generation_tokens_total": 10000.0
},
"accepted_per_position": {
"0": 3321.0,
"1": 2836.0
},
"accepted_fraction": 0.8010668748373666,
"committed_emitted_tokens": 10000.0,
"request_tokens_agree": true,
"limits": "Acceptance counters precede final stop trimming. Do not equate accepted plus bonus to emitted tokens. Proposals never enter speed numerator. No strict verdict or performance eligibility is assigned here.",
"sources": {
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-before.txt": "22a11ba0d334e8412b04fd0cc21e3c4ad023fd1dcce9a8129d9122ecc9d6d707",
"/home/ai/qwen38-release-retests-20261026/v620-w4-tp2-mtp2-r0/ladder/c1_10000_proxy/metrics-after.txt": "08899f1240a16630dd09114338fbfe2ddb3bae95b4a9cf14f0af8266e0d86e9b"
}
}
},
"retest_MTP_2K_accepted": 1089.0,
"retest_MTP_2K_proposed": 1820.0,
"retest_MTP_2K_draft_batches": 910.0,
"retest_MTP_2K_acceptance_percent": 59.83516483516483,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30544822272,
"1": 30544805888
},
"retest_memory_observation_count": 45,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-w4-tp2-mtp2/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 15.19,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 15.19,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 116800,
"retest_memory_log_report": {
"profile": "v620-w4-tp2-mtp2",
"image": "sha256:7f987d7773de6755f27160795f693fb1374eb42f54f2f322e917432c0886fd84",
"container": "70e39d1b172d119241eb03dbd6619fcb0c16edb72e4d6c821d31d0d3628f146b",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 10.97,
"retest_logged_model_load_seconds": 106.4305,
"retest_logged_available_KV_GiB": 15.19,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 15.19,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 116800,
"retest_logged_context_capacity_x": 3.38,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w4-tp2-mtp2/service-final.log",
"source_log_sha256": "38a702105b6b0ffe6eca92cfc94fe05a20949d83e51f76d0326d098ca29c1e05",
"sealed_index_sha256": "89aaf41caccca05a57040a8728c276f5acab0ccd3fc718e6700e103b8e71b36e",
"source_lines": {
"model_load": {
"line": 118,
"text": "(Worker pid=102) (Worker_TP0 pid=102) INFO 10-26 17:05:15 [gpu_model_runner.py:4598] Model loading took 10.97 GiB memory and 106.430500 seconds"
},
"automatic_KV_budget": {
"line": 168,
"text": "(Worker pid=102) (Worker_TP0 pid=102) INFO 10-26 17:21:47 [gpu_worker.py:453] Available KV cache memory: 15.19 GiB"
},
"grouped_KV_tokens": {
"line": 172,
"text": "(EngineCore_DP0 pid=84) INFO 10-26 17:21:47 [kv_cache_utils.py:1314] GPU KV cache size: 116,800 tokens"
},
"context_capacity": {
"line": 173,
"text": "(EngineCore_DP0 pid=84) INFO 10-26 17:21:47 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 3.38x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w4-tp2-mtp2/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.44708251953125,
"sampled_peak_VRAM_GiB": 26.92926025390625
} |
| V620Additional candidate | 4-bit RTN X (unpromoted)Qwen/Qwen3.8-27B | TP4MTP off | PASS6851 tokens | 41.176 | 39.369 | 1.100 / 49.68410K: 1.098 / 255.115Strict: — / — | 2.95 / 21.04packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:fa94293d3aa1bc4db60d2a27d6dafbefdd5c0709e736315b64faf66bc2e4da4dRuntime image config: sha256:23efb9f20a02e5b7bd69ed0004a7a1976d35ba728e4fd52f644b4645eadd5ed1 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-serve-r5.tar.gz.sha256 mkdir -p launch-tools-v620-serve-r5 tar -xzf launch-tools-v620-serve-r5.tar.gz -C launch-tools-v620-serve-r5 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp4-mtp0-x GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-v620-serve-r5/portable/launch.py --config v620-w4-tp4-mtp0-x --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Preflight cleanup compatibilityAfter its own selected-GPU preflight exits, serving waits at most 30 seconds for those allocations to drain before reapplying the original idle check. Device identity, idle thresholds, DPM and numerical/runtime settings are unchanged. Benchmark commands already have their original bounded drain check and are unchanged. Retained lab Docker command · not portabledocker run -d \
--name serve-v620-w4-tp4-mtp0-x-20261028 \
--network host \
--ipc host \
--cpuset-cpus 0-7,16-23 \
--cpuset-mems 0 \
--security-opt label=disable \
--shm-size 67108864 \
--device /dev/kfd:/dev/kfd:rwm \
--device /dev/dri/card5:/dev/dri/card5:rwm \
--device /dev/dri/card6:/dev/dri/card6:rwm \
--device /dev/dri/card7:/dev/dri/card7:rwm \
--device /dev/dri/card8:/dev/dri/card8:rwm \
--device /dev/dri/renderD133:/dev/dri/renderD133:rwm \
--device /dev/dri/renderD134:/dev/dri/renderD134:rwm \
--device /dev/dri/renderD135:/dev/dri/renderD135:rwm \
--device /dev/dri/renderD136:/dev/dri/renderD136:rwm \
--user root \
--workdir /tmp \
-e VLLM_DISABLE_COMPILE_CACHE=1 \
-e VNEXT_RTN_W4_ORDERED=1 \
-e PATH=/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e ROCM_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e ROCM_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e HIP_HOME=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel \
-e LD_LIBRARY_PATH=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib:/opt/venv/lib/python3.12/site-packages/torch/lib:/usr/local/lib \
-e ROCM_VERSION=7.14.0 \
-e TORCH_VERSION=2.12.0+rocm7.14.0 \
-e VLLM_TARGET_DEVICE=rocm \
-e VLLM_VERSION_OVERRIDE=0.0.0+gfx1030 \
-e PYTORCH_ROCM_ARCH=gfx1030 \
-e GPU_ARCHS=gfx1030 \
-e AMDGPU_TARGETS=gfx1030 \
-e CMAKE_HIP_ARCHITECTURES=gfx1030 \
-e PYTHONDONTWRITEBYTECODE=1 \
-e PIP_NO_CACHE_DIR=1 \
-e PIP_DISABLE_PIP_VERSION_CHECK=1 \
-e VNEXT_GDN_HEAD32=1 \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0-x/cache/vllm,dst=/root/.cache/vllm \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0-x/cache/torch_extensions,dst=/root/.cache/torch_extensions \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0-x/cache/.triton,dst=/root/.triton \
--mount type=bind,src=/home/ai/qwen38-serving-20261028/v620-w4-tp4-mtp0-x/cache/torchinductor,dst=/tmp/torchinductor_root \
--mount type=bind,src=/usr/share/ollama/localaiservers-vnext/models/Qwen3.8-27B-1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,dst=/opt/local-models/qwen38-27b-hf,readonly \
--entrypoint /usr/bin/env \
sha256:23efb9f20a02e5b7bd69ed0004a7a1976d35ba728e4fd52f644b4645eadd5ed1 \
V756F94_BOUNDARY_MODE=posted \
DO_NOT_TRACK=1 \
GPU_DEVICE_ORDINAL=0,1,2,3 \
GPU_MAX_HW_QUEUES=1 \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
OMP_NUM_THREADS=4 \
TEMP=/tmp/torchinductor_root \
TMPDIR=/tmp/torchinductor_root \
TMP=/tmp/torchinductor_root \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
TRITON_CACHE_DIR=/root/.triton \
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
VLLM_GPU_MEMORY_UTILIZATION=0.95 \
VLLM_LOGGING_LEVEL=INFO \
VLLM_PORT=9002 \
VLLM_RDNA_FORCE_FP16=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
NCCL_P2P_DISABLE=0 \
NCCL_ALGO=Tree \
NCCL_PROTO=LL \
NCCL_MIN_NCHANNELS=4 \
NCCL_MAX_NCHANNELS=4 \
NCCL_NTHREADS=256 \
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root \
HIP_VISIBLE_DEVICES=0,1,2,3 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
ROCR_VISIBLE_DEVICES=0,1,2,3 \
VLLM_CACHE_ROOT=/root/.cache/vllm \
PYTORCH_ROCM_ARCH=gfx1030 \
PYTHONHASHSEED=0 \
/opt/venv/bin/python \
-m \
vllm.vnext_rtn_entry \
serve \
/opt/local-models/qwen38-27b-hf \
--served-model-name \
Qwen/Qwen3.8-27B \
--enable-auto-tool-choice \
--tool-call-parser \
qwen3_coder \
--dtype \
half \
--host \
0.0.0.0 \
--port \
9002 \
--tensor-parallel-size \
4 \
--max-model-len \
131072 \
--gpu-memory-utilization \
0.95 \
--trust-remote-code \
--generation-config \
vllm \
-O=3 \
--reasoning-parser \
qwen3 \
--async-scheduling \
--hf-overrides \
'{"architectures":["Qwen3_5ForCausalLM"],"quantization_config":{"quant_method":"vnext_rtn","bits":4,"group_size":128}}' \
--language-model-only \
--max-num-seqs \
2 \
--max-num-batched-tokens \
2048 \
--mamba-ssm-cache-dtype \
float32 \
--no-enable-prefix-caching \
--no-enforce-eager \
--attention-backend \
TRITON_ATTN \
--disable-custom-all-reduce \
--seed \
0 \
--worker-cls \
vllm.vnext_rtn_worker.Worker \
--quantization \
vnext_rtnHistorical host paths and device addresses are retained as evidence. Use the serving command above.Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-v620-r4.tar.gz.sha256 mkdir -p launch-tools-v620-r4 tar -xzf launch-tools-v620-r4.tar.gz -C launch-tools-v620-r4 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/v620-w4-tp4-mtp0-x GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-v620-r4/portable/launch.py --config v620-w4-tp4-mtp0-x --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-v620-r4.tar.gz · SHA-256 launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. Portable observer compatibilityThe evidence collector verifies the caller-selected run/cache filesystem identity instead of requiring the original literal SATA label. Images, native arithmetic, prompts, timing and original route/numerical checks are unchanged. | All four commands verifiedTest host: 10.0.0.50acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 41.539 · eligible10K 39.761 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 8394,
"content_chars": 5030,
"content_has_think_close": false,
"elapsed_seconds": 210.26249684090726,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T06:14:00Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.013499699998646975,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 5030,
"qwen_answer_text_sha256": "a9a4d27533a8a0819fb3b1b995314fa366cdb37d74a8f623b76570a16020cb95",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 5030,
"qwen_visible_text_sha256": "a9a4d27533a8a0819fb3b1b995314fa366cdb37d74a8f623b76570a16020cb95",
"raw_has_think_close": false,
"reasoning_chars": 30971,
"text_chars": 36002,
"text_sha256": "e50e5c9b3705f244b624038ee60d85bce937e8071f04fa10daef3d8edc680da8",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 209.1481526561547,
"vllm_decode_tps": 40.134229699843274,
"vllm_generation_tokens_delta": 8394.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.0931877410039306,
"vllm_prefill_tps": 432.6795684386469,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 39.92152726290141
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 49.2615391809959,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T06:14:50Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.01474340702407062,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 8176,
"text_chars": 8176,
"text_sha256": "a547216ac88e1666e05daa4fe0dde87d974517b3902aaefa2b345516b1df762d",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 48.14751691184938,
"vllm_decode_tps": 41.53900612697617,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.0958944531157613,
"vllm_prefill_tps": 431.6109080169203,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 40.59962464127713
},
"10K": {
"completion_tokens": 10000,
"content_chars": 13264,
"content_has_think_close": false,
"elapsed_seconds": 252.61663698311895,
"endpoint": "http://127.0.0.1:19092/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T06:19:03Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.015581564977765083,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:9002/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 13264,
"qwen_answer_text_sha256": "cae96e1476d2bdd78c4c7701b9ef70f8882093ae3a3bbdcd5e209da6637c21a9",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 13264,
"qwen_visible_text_sha256": "cae96e1476d2bdd78c4c7701b9ef70f8882093ae3a3bbdcd5e209da6637c21a9",
"raw_has_think_close": false,
"reasoning_chars": 30971,
"text_chars": 44236,
"text_sha256": "004705f78884acd7d254e9109d3e79e2e49bedc484368a349825d41bebec297a",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 251.50327909295447,
"vllm_decode_tps": 39.76091300306285,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 1.0949219870381057,
"vllm_prefill_tps": 431.99424762628183,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 39.585674638952014
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestEarlier setup attempt retainedAdmission observer repair and replayPASS7,550 natural tokens2K 41.19510K 39.455 · diagnostic Retest latency / counters / KV{
"retest_prior_attempts": [
{
"attempt": "v620-w4-tp4-mtp0-x-r0",
"complete": false,
"error": "CalledProcessError(2, ['/usr/bin/python3', '/home/ai/qwen38-release-retests-20261026/package-r1/profiles/v620-w4-tp4-mtp0-x/original-inputs/audit_rtn_admission.py', '/home/ai/qwen38-release-retests-20261026/v620-w4-tp4-mtp0-x-r0', '--bits', '4', '--pins', '/home/ai/qwen38-release-retests-20261026/package-r1/profiles/v620-w4-tp4-mtp0-x/original-inputs/rtn-admission-pins.json'])",
"evidence": "retest-evidence/v620-w4-tp4-mtp0-x/attempts/v620-w4-tp4-mtp0-x-r0/evidence.tar.gz",
"result": "retest-evidence/v620-w4-tp4-mtp0-x/attempts/v620-w4-tp4-mtp0-x-r0/result.json",
"verification": "retest-evidence/v620-w4-tp4-mtp0-x/attempts/v620-w4-tp4-mtp0-x-r0/local-verification.json",
"archive_sha256": "776e2317325cdc10fce98d45e6b7793e5b590b16c811a40e18338576cc466c30"
}
],
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-27T01:35:53.161956943Z",
"FinishedAt": "2026-10-27T02:15:21.513031262Z"
},
"devices_after": [
{
"card": 5,
"render": 133,
"pci": "0000:16:00.0",
"actual_card": "card5",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:19:00.0",
"actual_card": "card6",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 7,
"render": 135,
"pci": "0000:1c:00.0",
"actual_card": "card7",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
},
{
"card": 8,
"render": 136,
"pci": "0000:1f:00.0",
"actual_card": "card8",
"busy_percent": 0,
"vram_bytes": 21401600,
"dpm": "auto"
}
],
"finished": "2026-10-27T02:15:21.654509+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 41.19541881788233,
"retest_10K_TPS": 39.45477749262219,
"retest_strict_tokens": 7550,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 7550,
"elapsed_seconds": 189.7973721170565,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 188.68804516794626,
"vllm_decode_tps": 40.01313381184242,
"vllm_generation_tokens_delta": 7550.0,
"vllm_prefill_seconds_delta": 1.0972324630711228,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 49.65721135598142,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 48.54908767505549,
"vllm_decode_tps": 41.19541881788233,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 1.0951066319830716,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 254.5613265290158,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 253.4547305930173,
"vllm_decode_tps": 39.45477749262219,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 1.0946464809821919,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 1.0951066319830716,
"retest_2K_total_seconds": 49.65721135598142,
"retest_10K_prefill_seconds": 1.0946464809821919,
"retest_10K_total_seconds": 254.5613265290158,
"retest_strict_prefill_seconds": 1.0972324630711228,
"retest_strict_total_seconds": 189.7973721170565,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": "Complete record: chart-details/row-24-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 22575697920,
"1": 22592425984,
"2": 22592434176,
"3": 22542094336
},
"retest_memory_observation_count": 37,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-w4-tp4-mtp0-x/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 13.57,
"retest_logged_model_load_seconds": 65.883079,
"retest_logged_available_KV_GiB": 13.65,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 13.65,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 223440,
"retest_logged_context_capacity_x": 6.67,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "v620-w4-tp4-mtp0-x",
"image": "sha256:23efb9f20a02e5b7bd69ed0004a7a1976d35ba728e4fd52f644b4645eadd5ed1",
"container": "ec3518784deb32352092d77a2decf4d6324f174f83299da367262e1603a263bb",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 13.57,
"retest_logged_model_load_seconds": 65.883079,
"retest_logged_available_KV_GiB": 13.65,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 13.65,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 223440,
"retest_logged_context_capacity_x": 6.67,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w4-tp4-mtp0-x/service-final.log",
"source_log_sha256": "3ef08b7b4c885cc867a908039180b10d89226cb9991c59b40a34cce08d3cfc4c",
"sealed_index_sha256": "f0b1471b89d3b00a16d5794a170ae274389ce3e0a9a1ca8f0b2ff7e9866e5f7e",
"source_lines": {
"model_load": {
"line": 120,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-27 01:38:37 [gpu_model_runner.py:4579] Model loading took 13.57 GiB memory and 65.883079 seconds"
},
"automatic_KV_budget": {
"line": 140,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-27 01:42:20 [gpu_worker.py:453] Available KV cache memory: 13.65 GiB"
},
"grouped_KV_tokens": {
"line": 145,
"text": "(EngineCore_DP0 pid=65) INFO 10-27 01:42:20 [kv_cache_utils.py:1314] GPU KV cache size: 223,440 tokens"
},
"context_capacity": {
"line": 146,
"text": "(EngineCore_DP0 pid=65) INFO 10-27 01:42:20 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.67x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w4-tp4-mtp0-x/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 21.040843963623047
}Complete retest loaded admission | Evidence and limitsnewer X candidate; unpromoted, historical reference retained {
"strict_tokens": 6851,
"source": "/home/joe/PhoneSync/Documents/gfx1030/provenance/qwen38-quantized-engineering-result.json#$.RTN4_ordered_half_partials.serving_result",
"source_revision": "4f5967174dde96d94c30f9f347f93ffe3da73b56",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"sampled_VRAM_startup_max_bytes_by_rank": "[22575652864, 22592380928, 22592438272, 22542057472]",
"comparison_limit": "One sample; fresh versus seeded caches; KV1140 vs1149; no dynamic X invocation trace; original launch/startup/ladder parent exits unknown.",
"retest_strict_tokens": 7550,
"retest_MTP_accounting_by_tier": {},
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 22575697920,
"1": 22592425984,
"2": 22592434176,
"3": 22542094336
},
"retest_memory_observation_count": 37,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/v620-w4-tp4-mtp0-x/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 13.65,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 13.65,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 223440,
"retest_memory_log_report": {
"profile": "v620-w4-tp4-mtp0-x",
"image": "sha256:23efb9f20a02e5b7bd69ed0004a7a1976d35ba728e4fd52f644b4645eadd5ed1",
"container": "ec3518784deb32352092d77a2decf4d6324f174f83299da367262e1603a263bb",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 13.57,
"retest_logged_model_load_seconds": 65.883079,
"retest_logged_available_KV_GiB": 13.65,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 13.65,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 223440,
"retest_logged_context_capacity_x": 6.67,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/v620-w4-tp4-mtp0-x/service-final.log",
"source_log_sha256": "3ef08b7b4c885cc867a908039180b10d89226cb9991c59b40a34cce08d3cfc4c",
"sealed_index_sha256": "f0b1471b89d3b00a16d5794a170ae274389ce3e0a9a1ca8f0b2ff7e9866e5f7e",
"source_lines": {
"model_load": {
"line": 120,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-27 01:38:37 [gpu_model_runner.py:4579] Model loading took 13.57 GiB memory and 65.883079 seconds"
},
"automatic_KV_budget": {
"line": 140,
"text": "(Worker pid=83) (Worker_TP0 pid=83) INFO 10-27 01:42:20 [gpu_worker.py:453] Available KV cache memory: 13.65 GiB"
},
"grouped_KV_tokens": {
"line": 145,
"text": "(EngineCore_DP0 pid=65) INFO 10-27 01:42:20 [kv_cache_utils.py:1314] GPU KV cache size: 223,440 tokens"
},
"context_capacity": {
"line": 146,
"text": "(EngineCore_DP0 pid=65) INFO 10-27 01:42:20 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 6.67x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/v620-w4-tp4-mtp0-x/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 21.040843963623047,
"sampled_peak_VRAM_GiB": 21.040847778320312
} |
| MI50Earlier Qwen3.8 TP8 | FP16Qwen/Qwen3.8-27B | TP8MTP off | PASS6151 tokens | 67.305 | 63.426 | 14.437 / 44.16110K: 14.813 / 172.486Strict: 13.197 / 107.360 | 6.30 / 31.33FP16 parameter extents / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:7d9a5e4969001af7785d888350428ad81fec613027782d326b42516fd6bc3dcdRuntime image config: sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp8-mtp0-previous GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-r2/portable/launch.py --config mi50-fp16-tp8-mtp0-previous --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-fp16-tp8-mtp0-previous-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-15 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD128:/dev/dri/renderD128:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \ --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \ --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \ --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \ --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \ --user root \ --workdir /tmp \ -e TRANSFORMERS_OFFLINE=1 \ -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e HF_HUB_OFFLINE=1 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0-previous/cache/.triton,dst=/root/.triton \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0-previous/inputs/profiles/mi50-fp16-tp8-mtp0-previous/overlays/vllm_command.sh,dst=/opt/vnext/profile/vllm_command.sh,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0-previous/inputs/profiles/mi50-fp16-tp8-mtp0-previous/overlays/mi50_tp8_worker.py,dst=/opt/mi50-tp8-observer/mi50_tp8_worker.py,readonly \ --mount type=bind,src=/usr/share/ollama/vnext-offload/models/Qwen3.8-27B-1d4bf0f2ff60,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0-previous/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0-previous,dst=/runwork \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0-previous/cache/torchinductor,dst=/tmp/torchinductor_root \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0-previous/cache/vllm,dst=/root/.cache/vllm \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04Historical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp8-mtp0-previous GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-r2/portable/launch.py --config mi50-fp16-tp8-mtp0-previous --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict FAIL2K 67.367 · diagnostic / ineligible10K 63.511 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 10805,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 186.761883944273,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T21:57:45Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.012507418170571327,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 48647,
"text_chars": 48647,
"text_sha256": "9d9059ffcf98b5f4c6a9e0da75090231aaaf3a77127a7e5ca3da19c24aa0afc0",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 171.0692141884938,
"vllm_decode_tps": 63.16156914180033,
"vllm_generation_tokens_delta": 10805.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.682509177364409,
"vllm_prefill_tps": 30.160989842283136,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 57.85441746359795
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 45.63218120113015,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T21:58:33Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008618236519396305,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 8718,
"text_chars": 8718,
"text_sha256": "e85c1b58cf5ead754a97bf391e47b1010ea5c827678a1fd1e4e73600326c0709",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 29.68798005953431,
"vllm_decode_tps": 67.36733169415137,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.935353471897542,
"vllm_prefill_tps": 29.68242912428326,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 43.82871796517294
},
"10K": {
"completion_tokens": 10000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 172.98016954772174,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T22:01:26Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008071170188486576,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 45045,
"text_chars": 45045,
"text_sha256": "e2394ceddab1ea404538250f6ffcba7785f236db39ec71498d31ae36b66c8e1b",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 157.45232519879937,
"vllm_decode_tps": 63.511288178018305,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.5185188902542,
"vllm_prefill_tps": 30.47971287369758,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 57.81009479957297
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS6,117 natural tokens2K 67.60310K 63.714 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T17:09:31.497941141Z",
"FinishedAt": "2026-10-26T17:30:28.269542448Z"
},
"devices_after": [
{
"card": 0,
"render": 128,
"pci": "0000:06:00.0",
"actual_card": "card1",
"busy_percent": 0,
"vram_bytes": 20000714752,
"dpm": "auto"
},
{
"card": 1,
"render": 129,
"pci": "0000:09:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 21793456128,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:45:00.0",
"actual_card": "card3",
"busy_percent": 0,
"vram_bytes": 21791809536,
"dpm": "auto"
},
{
"card": 3,
"render": 131,
"pci": "0000:48:00.0",
"actual_card": "card4",
"busy_percent": 0,
"vram_bytes": 21790257152,
"dpm": "auto"
},
{
"card": 4,
"render": 132,
"pci": "0000:89:00.0",
"actual_card": "card5",
"busy_percent": 0,
"vram_bytes": 21793210368,
"dpm": "auto"
},
{
"card": 5,
"render": 133,
"pci": "0000:8c:00.0",
"actual_card": "card6",
"busy_percent": 0,
"vram_bytes": 21792894976,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:c5:00.0",
"actual_card": "card7",
"busy_percent": 0,
"vram_bytes": 21791252480,
"dpm": "auto"
},
{
"card": 7,
"render": 135,
"pci": "0000:c8:00.0",
"actual_card": "card8",
"busy_percent": 0,
"vram_bytes": 21738868736,
"dpm": "auto"
}
],
"finished": "2026-10-26T17:30:28.397358+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 67.60343526281542,
"retest_10K_TPS": 63.7137636364964,
"retest_strict_tokens": 6117,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 6117,
"elapsed_seconds": 108.61691073700786,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 93.25116309709847,
"vllm_decode_tps": 65.59703704318012,
"vllm_generation_tokens_delta": 6117.0,
"vllm_prefill_seconds_delta": 15.356500206049532,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 45.094846380874515,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 29.584295416716486,
"vllm_decode_tps": 67.60343526281542,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 15.502325372304767,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 171.70430587418377,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 156.95195871731266,
"vllm_decode_tps": 63.7137636364964,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 14.742029319982976,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 15.502325372304767,
"retest_2K_total_seconds": 45.094846380874515,
"retest_10K_prefill_seconds": 14.742029319982976,
"retest_10K_total_seconds": 171.70430587418377,
"retest_strict_prefill_seconds": 15.356500206049532,
"retest_strict_total_seconds": 108.61691073700786,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": "Complete record: chart-details/row-25-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33637744640,
"1": 33584029696,
"2": 33584132096,
"3": 33584074752,
"4": 33584177152,
"5": 33584128000,
"6": 33584078848,
"7": 33533796352
},
"retest_memory_observation_count": 21,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-fp16-tp8-mtp0-previous/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 6.82,
"retest_logged_model_load_seconds": 7.578667,
"retest_logged_available_KV_GiB": 22.67,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 22.67,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 371072,
"retest_logged_context_capacity_x": 11.26,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-fp16-tp8-mtp0-previous",
"image": "sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04",
"container": "8b859435de245533d593ce5501e9d10ad6d8dcc76d593b11ac845b427b5c56c2",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 6.82,
"retest_logged_model_load_seconds": 7.578667,
"retest_logged_available_KV_GiB": 22.67,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 22.67,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 371072,
"retest_logged_context_capacity_x": 11.26,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-fp16-tp8-mtp0-previous/service-final.log",
"source_log_sha256": "071c179f82976ea238ccede6a3e54f18071f17adf763c73baff256e6cfaa164f",
"sealed_index_sha256": "36dc26d55a69b2bb0238dd8b67f921f2e7195d0d7794c8a7336849ac6cc94e73",
"source_lines": {
"model_load": {
"line": 207,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:11:11 [gpu_model_runner.py:4584] Model loading took 6.82 GiB memory and 7.578667 seconds"
},
"automatic_KV_budget": {
"line": 256,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:12:42 [gpu_worker.py:453] Available KV cache memory: 22.67 GiB"
},
"grouped_KV_tokens": {
"line": 264,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 17:12:42 [kv_cache_utils.py:1314] GPU KV cache size: 371,072 tokens"
},
"context_capacity": {
"line": 265,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 17:12:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.26x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-fp16-tp8-mtp0-previous/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 31.327590942382812
}Complete retest loaded admission | Evidence and limitsCompleted, independently reviewed strict-valid TP8 sample. Stopped because official fixed2K did not exceed 71.0 TPS; no promotion or TP4 goal credit. {
"strict_tokens": 6151,
"source": "/home/joe/PhoneSync/Documents/gfx906/tuning/qwen38-fp16-matrix/tp8-as-is/result-20261023.json#$",
"source_revision": "57f22fd70c859e4df9036fcc075a798fb032c31d",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"attention_KV_dtype": "FP16",
"memory_accounting_limit": "Logical tensor/cache views may alias. Allocator counts and sampled physical VRAM are distinct; block geometry is not proof of a full-context request.",
"sampled_VRAM_before_ladder_bytes_by_rank": "{\"0\": 33637703680, \"1\": 33584119808, \"2\": 33584070656, \"3\": 33584037888, \"4\": 33584087040, \"5\": 33584066560, \"6\": 33584033792, \"7\": 33533755392}",
"sampled_VRAM_after_ladder_bytes_by_rank": "{\"0\": 33637703680, \"1\": 33584119808, \"2\": 33584070656, \"3\": 33584037888, \"4\": 33584087040, \"5\": 33584066560, \"6\": 33584033792, \"7\": 33533755392}",
"KV_capacity": "{\"GDN_SSM\": {\"bytes\": 1403584512, \"dtype\": \"torch.float16\", \"shape\": [7139, 6, 128, 128]}, \"GDN_conv\": {\"bytes\": 54827520, \"dtype\": \"torch.float16\", \"shape\": [7139, 3, 1280]}, \"SSM_precision\": \"actual torch.float16 (original auto policy)\", \"attention\": {\"bytes\": 1520549888, \"dtype\": \"torch.float16\", \"shape\": [7139, 2, 208, 1, 256]}, \"block_size\": 208, \"blocks\": 7139}",
"post_KV_allocator_allocated_bytes_by_rank": "{\"0\": 31288186880, \"1\": 31288186880, \"2\": 31288186880, \"3\": 31288186880, \"4\": 31288186880, \"5\": 31288186880, \"6\": 31288186880, \"7\": 31288186880}",
"post_KV_allocator_reserved_bytes_by_rank": "{\"0\": 31669092352, \"1\": 31669092352, \"2\": 31669092352, \"3\": 31669092352, \"4\": 31669092352, \"5\": 31669092352, \"6\": 31669092352, \"7\": 31669092352}",
"source_evidence_commit": "c7b96bd",
"comparison_limit": "Exact original 57f image/native arithmetic; mounted TP8 profile and observation worker only. | Eight .30 cards and CPU0-15 differ from .50 TP4 control hardware/affinity/storage. No transferred causal gain. | TP4-only packed/interleaved/posted guards fall back; no forced TP4 shapes or new dispatch. | Actual target, attention KV, convolution state and SSM are FP16; not the separate MTP FP32-SSM composition. | Non-eager O3, maxseq2/maxbatched4, context131072, greedy original requests/scorer preserved. | Single sample: no repeatability, causal performance or historical Qwen3.6 transfer claim. | Strict is sole correctness authority; forced-length 2K/10K verdict fields retain their raw meanings. Strict TPS and 10K are diagnostic. | Logical tensor bytes are not unique physical storage; cache views can alias shared allocations. Do not sum all per-layer view bytes into physical VRAM. | Metadata and generated source/maps do not establish executed kernel traces. Wrapper source hash is directly observed; underlying qwen source is separately retained in effective-profile evidence. | 7139x208 describes allocated block geometry, not an executed 131072-token request or a naive usable-token guarantee. | Outer counters do not prove scorer internal scrape alignment or complete settlement. | Read-only retention preserves regular files; original modes/times, symlinks and socket inventory separately retained, sockets are not byte-hashed. | Assigned PCI06 kernel log appends 11 vm_bo_update_detail diagnostic lines during ladder and three more before closure; no identical-kernel-log or host-exclusivity claim. | Outer transport statuses supplied by owner; preserve remote successful command receipts separately. Login SSH1 is not a runtime failure.",
"recorded_configuration_details": {
"cell": "FP16_TP8_MTP_off",
"status": "completed",
"TP": 8,
"native_MTP_tokens": 0,
"host": "10.0.0.30",
"strict_pass": true,
"strict_tokens": 6151,
"strict_diagnostic_TPS": 65.3299318056544,
"official_2K_TPS": 67.30474071347291,
"official_2K_tokens": 2000,
"official_2K_prefill_seconds": 14.436688052956015,
"official_2K_total_seconds": 44.16095239017159,
"diagnostic_10K_TPS": 63.42627998549319,
"diagnostic_10K_tokens": 10000,
"diagnostic_10K_prefill_seconds": 14.813438802957535,
"diagnostic_10K_total_seconds": 172.48586302390322,
"SSM_dtype": "torch.float16",
"source": "57f22fd70c859e4df9036fcc075a798fb032c31d",
"image": "sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04",
"primary": "tuning/qwen38-fp16-matrix/tp8-as-is/result-20261023.json",
"strict_prefill_seconds": 13.196696631144732,
"strict_total_seconds": 107.35985162388533,
"per_rank_memory_JSON": [
{
"rank": 0,
"pid": 196,
"parameter_count": 611,
"parameter_logical_bytes": 6767123584,
"parameter_dtype": "torch.float16",
"quantized_packed_bytes": 0,
"observed_fixed3_buffer_bytes": 0,
"after_load_allocated_bytes": 6958575104,
"after_load_reserved_bytes": 7308574720,
"after_cache_allocated_bytes": 31288186880,
"after_cache_reserved_bytes": 31669092352,
"attention_layers": 16,
"GDN_layers": 48
},
{
"rank": 1,
"pid": 197,
"parameter_count": 611,
"parameter_logical_bytes": 6767123584,
"parameter_dtype": "torch.float16",
"quantized_packed_bytes": 0,
"observed_fixed3_buffer_bytes": 0,
"after_load_allocated_bytes": 6958575104,
"after_load_reserved_bytes": 7308574720,
"after_cache_allocated_bytes": 31288186880,
"after_cache_reserved_bytes": 31669092352,
"attention_layers": 16,
"GDN_layers": 48
},
{
"rank": 2,
"pid": 198,
"parameter_count": 611,
"parameter_logical_bytes": 6767123584,
"parameter_dtype": "torch.float16",
"quantized_packed_bytes": 0,
"observed_fixed3_buffer_bytes": 0,
"after_load_allocated_bytes": 6958575104,
"after_load_reserved_bytes": 7308574720,
"after_cache_allocated_bytes": 31288186880,
"after_cache_reserved_bytes": 31669092352,
"attention_layers": 16,
"GDN_layers": 48
},
{
"rank": 3,
"pid": 199,
"parameter_count": 611,
"parameter_logical_bytes": 6767123584,
"parameter_dtype": "torch.float16",
"quantized_packed_bytes": 0,
"observed_fixed3_buffer_bytes": 0,
"after_load_allocated_bytes": 6958575104,
"after_load_reserved_bytes": 7308574720,
"after_cache_allocated_bytes": 31288186880,
"after_cache_reserved_bytes": 31669092352,
"attention_layers": 16,
"GDN_layers": 48
},
{
"rank": 4,
"pid": 200,
"parameter_count": 611,
"parameter_logical_bytes": 6767123584,
"parameter_dtype": "torch.float16",
"quantized_packed_bytes": 0,
"observed_fixed3_buffer_bytes": 0,
"after_load_allocated_bytes": 6958575104,
"after_load_reserved_bytes": 7308574720,
"after_cache_allocated_bytes": 31288186880,
"after_cache_reserved_bytes": 31669092352,
"attention_layers": 16,
"GDN_layers": 48
},
{
"rank": 5,
"pid": 201,
"parameter_count": 611,
"parameter_logical_bytes": 6767123584,
"parameter_dtype": "torch.float16",
"quantized_packed_bytes": 0,
"observed_fixed3_buffer_bytes": 0,
"after_load_allocated_bytes": 6958575104,
"after_load_reserved_bytes": 7308574720,
"after_cache_allocated_bytes": 31288186880,
"after_cache_reserved_bytes": 31669092352,
"attention_layers": 16,
"GDN_layers": 48
},
{
"rank": 6,
"pid": 202,
"parameter_count": 611,
"parameter_logical_bytes": 6767123584,
"parameter_dtype": "torch.float16",
"quantized_packed_bytes": 0,
"observed_fixed3_buffer_bytes": 0,
"after_load_allocated_bytes": 6958575104,
"after_load_reserved_bytes": 7308574720,
"after_cache_allocated_bytes": 31288186880,
"after_cache_reserved_bytes": 31669092352,
"attention_layers": 16,
"GDN_layers": 48
},
{
"rank": 7,
"pid": 203,
"parameter_count": 611,
"parameter_logical_bytes": 6767123584,
"parameter_dtype": "torch.float16",
"quantized_packed_bytes": 0,
"observed_fixed3_buffer_bytes": 0,
"after_load_allocated_bytes": 6958575104,
"after_load_reserved_bytes": 7308574720,
"after_cache_allocated_bytes": 31288186880,
"after_cache_reserved_bytes": 31669092352,
"attention_layers": 16,
"GDN_layers": 48
}
],
"KV_JSON": {
"attention": {
"shape": [
7139,
2,
208,
1,
256
],
"dtype": "torch.float16",
"bytes": 1520549888
},
"GDN_conv": {
"shape": [
7139,
3,
1280
],
"dtype": "torch.float16",
"bytes": 54827520
},
"GDN_SSM": {
"shape": [
7139,
6,
128,
128
],
"dtype": "torch.float16",
"bytes": 1403584512
},
"blocks": 7139,
"block_size": 208,
"SSM_precision": "actual torch.float16 (original auto policy)"
},
"VRAM_JSON": {
"before": "renderD128 33637703680\nrenderD129 33584119808\nrenderD130 33584070656\nrenderD131 33584037888\nrenderD132 33584087040\nrenderD133 33584066560\nrenderD134 33584033792\nrenderD135 33533755392",
"after": "renderD128 33637703680\nrenderD129 33584119808\nrenderD130 33584070656\nrenderD131 33584037888\nrenderD132 33584087040\nrenderD133 33584066560\nrenderD134 33584033792\nrenderD135 33533755392"
},
"attention_KV_dtype": "torch.float16",
"context": 131072,
"speculative_accounting_JSON": null
},
"detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/fp16-matrix.csv",
"detail_source_sha256": "8643d90cf1f211a7f77be9f6212cd7706e6dbae3ae84b9bc80b5f42fa199ffe0",
"detail_source_pointer": "FP16_TP8_MTP_off",
"retest_strict_tokens": 6117,
"retest_MTP_accounting_by_tier": {},
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33637744640,
"1": 33584029696,
"2": 33584132096,
"3": 33584074752,
"4": 33584177152,
"5": 33584128000,
"6": 33584078848,
"7": 33533796352
},
"retest_memory_observation_count": 21,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-fp16-tp8-mtp0-previous/runtime-observations.jsonl",
"retest_logged_available_KV_GiB": 22.67,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 22.67,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 371072,
"retest_memory_log_report": {
"profile": "mi50-fp16-tp8-mtp0-previous",
"image": "sha256:5c4e7b7a071d89eee0e8941aa9f9a857b687864ee7bb5d9d59eb7cc50ff48c04",
"container": "8b859435de245533d593ce5501e9d10ad6d8dcc76d593b11ac845b427b5c56c2",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 6.82,
"retest_logged_model_load_seconds": 7.578667,
"retest_logged_available_KV_GiB": 22.67,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 22.67,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 371072,
"retest_logged_context_capacity_x": 11.26,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-fp16-tp8-mtp0-previous/service-final.log",
"source_log_sha256": "071c179f82976ea238ccede6a3e54f18071f17adf763c73baff256e6cfaa164f",
"sealed_index_sha256": "36dc26d55a69b2bb0238dd8b67f921f2e7195d0d7794c8a7336849ac6cc94e73",
"source_lines": {
"model_load": {
"line": 207,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:11:11 [gpu_model_runner.py:4584] Model loading took 6.82 GiB memory and 7.578667 seconds"
},
"automatic_KV_budget": {
"line": 256,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 17:12:42 [gpu_worker.py:453] Available KV cache memory: 22.67 GiB"
},
"grouped_KV_tokens": {
"line": 264,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 17:12:42 [kv_cache_utils.py:1314] GPU KV cache size: 371,072 tokens"
},
"context_capacity": {
"line": 265,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 17:12:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.26x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-fp16-tp8-mtp0-previous/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 31.327590942382812,
"sampled_peak_VRAM_GiB": 31.327552795410156
} |
| MI50Current Qwen3.8 TP8 | FP16Qwen/Qwen3.8-27B | TP8MTP off | PASS14795 tokens | 70.372 | 66.406 | 15.524 / 43.95310K: 14.785 / 165.383Strict: 14.693 / 245.164 | 6.30 / 31.46retained FP16 parameter extents; projection buffers separate / VRAM GiBSeparate packed-buffer extents: 2.188 GiB/rank; includes alternate views, not additive physical storage. | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:6d33e0476cbb771490316875aaa8bbc4d2a24c84c5c82cb979f3729518971b46Runtime image config: sha256:0aab9fc8441d0d25b576bdb70ef0c9cbb6982eb7ac37a7f108fee2dba77a40ce | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp8-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-r2/portable/launch.py --config mi50-fp16-tp8-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-fp16-tp8-mtp0-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-15 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD128:/dev/dri/renderD128:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \ --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \ --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \ --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \ --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \ --user root \ --workdir /tmp \ -e TRANSFORMERS_OFFLINE=1 \ -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e HF_HUB_OFFLINE=1 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ --mount type=bind,src=/usr/share/ollama/vnext-offload/models/Qwen3.8-27B-1d4bf0f2ff60,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0/cache/.triton,dst=/root/.triton \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0/inputs/profiles/mi50-fp16-tp8-mtp0/overlays/vllm_command.sh,dst=/opt/vnext/profile/vllm_command.sh,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0/inputs/profiles/mi50-fp16-tp8-mtp0/overlays/mi50_tp8_worker.py,dst=/opt/mi50-tp8-observer/mi50_tp8_worker.py,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0,dst=/runwork \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-fp16-tp8-mtp0/inputs/profiles/mi50-fp16-tp8-mtp0/overlays/serving-pins.json,dst=/opt/mi50-tp8-observer/serving-pins.json,readonly \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:0aab9fc8441d0d25b576bdb70ef0c9cbb6982eb7ac37a7f108fee2dba77a40ceHistorical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-fp16-tp8-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-r2/portable/launch.py --config mi50-fp16-tp8-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict FAIL2K 70.046 · diagnostic / ineligible10K 66.129 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 15240,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 254.38865686114877,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T12:30:08Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.013873212039470673,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 32507,
"text_chars": 32507,
"text_sha256": "97b5e51a52f86183973ec7e490147143eccaa1ab9f5cec6ee32a7a52a9f138b9",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 239.16702868323773,
"vllm_decode_tps": 63.721157903351546,
"vllm_generation_tokens_delta": 15240.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.211745806969702,
"vllm_prefill_tps": 31.094392846301794,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 59.908331558660436
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 43.61238981690258,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T12:30:53Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.009367557242512703,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 7986,
"text_chars": 7986,
"text_sha256": "a90fec54997946ef62a25d7700199fcb23aa456107b173b2007db62135aeae9d",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 28.552836760878563,
"vllm_decode_tps": 70.04557959509941,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.050994507968426,
"vllm_prefill_tps": 31.426494757511225,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 45.85852800996639
},
"10K": {
"completion_tokens": 10000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 166.6537444461137,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T12:33:40Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008907177485525608,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 22835,
"text_chars": 22835,
"text_sha256": "2bff7094c90509554fe661af4c5fa3e872ed92c8f90ff7ad0cca77dd4bd7ab89",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 151.21882070507854,
"vllm_decode_tps": 66.12933465142517,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.426380563527346,
"vllm_prefill_tps": 30.66176139322764,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 60.004652360112004
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS10,210 natural tokens2K 70.34010K 66.386 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T15:57:46.384320926Z",
"FinishedAt": "2026-10-26T16:20:30.268261294Z"
},
"devices_after": [
{
"card": 0,
"render": 128,
"pci": "0000:06:00.0",
"actual_card": "card1",
"busy_percent": 0,
"vram_bytes": 21298290688,
"dpm": "auto"
},
{
"card": 1,
"render": 129,
"pci": "0000:09:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 21178646528,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:45:00.0",
"actual_card": "card3",
"busy_percent": 0,
"vram_bytes": 21178335232,
"dpm": "auto"
},
{
"card": 3,
"render": 131,
"pci": "0000:48:00.0",
"actual_card": "card4",
"busy_percent": 0,
"vram_bytes": 22552678400,
"dpm": "auto"
},
{
"card": 4,
"render": 132,
"pci": "0000:89:00.0",
"actual_card": "card5",
"busy_percent": 0,
"vram_bytes": 22555885568,
"dpm": "auto"
},
{
"card": 5,
"render": 133,
"pci": "0000:8c:00.0",
"actual_card": "card6",
"busy_percent": 0,
"vram_bytes": 22555213824,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:c5:00.0",
"actual_card": "card7",
"busy_percent": 0,
"vram_bytes": 22556356608,
"dpm": "auto"
},
{
"card": 7,
"render": 135,
"pci": "0000:c8:00.0",
"actual_card": "card8",
"busy_percent": 0,
"vram_bytes": 22505455616,
"dpm": "auto"
}
],
"finished": "2026-10-26T16:20:30.387589+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 70.33971228281604,
"retest_10K_TPS": 66.38633126361084,
"retest_strict_tokens": 10210,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 10210,
"elapsed_seconds": 168.26249494263902,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 153.90207020984963,
"vllm_decode_tps": 66.34088798206801,
"vllm_generation_tokens_delta": 10210.0,
"vllm_prefill_seconds_delta": 14.347171621862799,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 44.575539912097156,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 28.433440159074962,
"vllm_decode_tps": 70.33971228281604,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 16.134185431990772,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 164.31250084890053,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 150.6334182000719,
"vllm_decode_tps": 66.38633126361084,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 13.67035356303677,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 16.134185431990772,
"retest_2K_total_seconds": 44.575539912097156,
"retest_10K_prefill_seconds": 13.67035356303677,
"retest_10K_total_seconds": 164.31250084890053,
"retest_strict_prefill_seconds": 14.347171621862799,
"retest_strict_total_seconds": 168.26249494263902,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": "Complete record: chart-details/row-26-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_logged_model_load_delta_GiB": 9.07,
"retest_logged_model_load_seconds": 22.691548,
"retest_logged_available_KV_GiB": 20.57,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 20.57,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 336752,
"retest_logged_context_capacity_x": 10.22,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-fp16-tp8-mtp0",
"image": "sha256:0aab9fc8441d0d25b576bdb70ef0c9cbb6982eb7ac37a7f108fee2dba77a40ce",
"container": "ae82bff4b20a076f77a910c5721aa5ac0fc946b40a75fd1ab511e08b08f418aa",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 9.07,
"retest_logged_model_load_seconds": 22.691548,
"retest_logged_available_KV_GiB": 20.57,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 20.57,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 336752,
"retest_logged_context_capacity_x": 10.22,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-fp16-tp8-mtp0/service-final.log",
"source_log_sha256": "73b42cdf85e17a4b8fc446101f58a84f60f280529af15dc1711fd4dada425b46",
"sealed_index_sha256": "4578ad1dc43c02e27e9e24e23f5962720d6906c26ef4715afe24cb4f342f5279",
"source_lines": {
"model_load": {
"line": 216,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 15:59:42 [gpu_model_runner.py:4584] Model loading took 9.07 GiB memory and 22.691548 seconds"
},
"automatic_KV_budget": {
"line": 267,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:02:06 [gpu_worker.py:453] Available KV cache memory: 20.57 GiB"
},
"grouped_KV_tokens": {
"line": 274,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:02:06 [kv_cache_utils.py:1314] GPU KV cache size: 336,752 tokens"
},
"context_capacity": {
"line": 275,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:02:06 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 10.22x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-fp16-tp8-mtp0/memory-log-report.json"
}Complete retest loaded admission | Evidence and limitsAccepted TP8 deployment baseline under strict PASS and >=70 TPS. Exact tested image/profile restored as a new service instance on .30; completed inference and HTTP200 verified. Original measured cohort remains stopped under historical >71 rule. One eligible benchmark sample, no TP4 qualification. {
"source": "/home/joe/PhoneSync/Documents/gfx906/tuning/fixed3-tp8-gateup/serving/result-20261023.json#$",
"source_revision": "f3aff262c7ce2ed5b65a156ce1f7cd5dc5bb6d7c",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"strict_tokens": 14795,
"attention_KV_dtype": "FP16",
"memory_accounting_limit": "Target parameter extents and lossless packed gate/up buffers are separate representations, not a disjoint unique whole-model footprint. Logical KV views may alias. Allocator counters and sampled physical VRAM are separate. No full-context request was tested.",
"sampled_VRAM_before_ladder_bytes_by_rank": "{\"0\": 33783861248, \"1\": 33730301952, \"2\": 33730269184, \"3\": 33730314240, \"4\": 33730240512, \"5\": 33730314240, \"6\": 33730297856, \"7\": 33679798272}",
"sampled_VRAM_after_ladder_bytes_by_rank": "{\"0\": 33783861248, \"1\": 33730301952, \"2\": 33730269184, \"3\": 33730314240, \"4\": 33730240512, \"5\": 33730314240, \"6\": 33730297856, \"7\": 33679798272}",
"KV_capacity": "{\"actual_attention_shape\": [6477, 2, 208, 1, 256], \"actual_blocks\": 6477, \"available_KV_GiB\": 20.57, \"limit\": \"Engine grouped-cache interpretation differs from raw view extent; no full-context inference performed.\", \"reported_GPU_KV_tokens\": 336752, \"reported_max_concurrency_at_131072\": 10.22}",
"attention_KV_logical_view_shape": "[6477, 2, 208, 1, 256]",
"post_KV_allocator_allocated_bytes_by_rank": "{\"0\": 31438801408, \"1\": 31438786560, \"2\": 31438564864, \"3\": 31438415360, \"4\": 31438394880, \"5\": 31438513664, \"6\": 31438672896, \"7\": 31438678016}",
"post_KV_allocator_reserved_bytes_by_rank": "{\"0\": 31803310080, \"1\": 31803310080, \"2\": 31803310080, \"3\": 31803310080, \"4\": 31803310080, \"5\": 31803310080, \"6\": 31803310080, \"7\": 31803310080}",
"source_evidence_commit": "ca367c2",
"comparison_limit": "New host-only TP8 dispatch and separate gate/up packing; full fixed3 device ELF unchanged. | Dense gate-then-up weight layout preserved. M1 fused K5120; M2..4 original opaque GEMM plus exact retained compiled TI7 activation; M>4 original Python path. | Only VLLM_GFX906_TP8_GATE_FIXED3=1 added to original TP8 profile. | Mounted intended profile deliberately differs from stock image profile in installed.sha256. Full manifest passes at image build; runtime observes this replacement separately. | New private rootFS cache; other projection/collective eligibility and all requests/scorer unchanged. | Outer scrapes in either cohort do not expose internal scorer scrapes or the full settlement trajectory. | One official sample per configuration; no repeatability or causal claim. | Fixed-tier strict flags are not model correctness verdicts; uncapped strict is sole correctness authority. | No TP4 qualification, formal promotion or model causal attribution. | Gate/up-only packed route on the same .30 host; actual eight FP16 ranks admitted. Other projection families retain fallback. Generated callsites are not dynamic per-token kernel traces. | Fresh private rootSSD caches; actual KV blocks 6477 versus original dense 7139. One sample per configuration; observed gain is not isolated causal attribution. | Scoped kernel logs changed; retained bounded fault/reset/error signatures have zero matches. Active debug logging is not assumed performance-neutral. | Original >71 rule closed the measured cohort. Joe subsequently accepted strict-valid >=70 TPS; a service restoration does not create a new benchmark sample. | Fixed-tier raw strict flags are not model verdicts; uncapped strict PASS determines eligibility.",
"recorded_configuration_details": {
"cell": "FP16_TP8_gateup_MTP_off",
"status": "completed",
"TP": 8,
"native_MTP_tokens": 0,
"host": "10.0.0.30",
"strict_pass": true,
"strict_tokens": 14795,
"strict_diagnostic_TPS": 64.19733379026958,
"official_2K_TPS": 70.37197138638763,
"official_2K_tokens": 2000,
"official_2K_prefill_seconds": 15.52441488718614,
"official_2K_total_seconds": 43.95317275123671,
"diagnostic_10K_TPS": 66.40603683633941,
"diagnostic_10K_tokens": 10000,
"diagnostic_10K_prefill_seconds": 14.784574894234538,
"diagnostic_10K_total_seconds": 165.38285010913387,
"SSM_dtype": "torch.float16",
"source": "f3aff262c7ce2ed5b65a156ce1f7cd5dc5bb6d7c",
"image": "sha256:0aab9fc8441d0d25b576bdb70ef0c9cbb6982eb7ac37a7f108fee2dba77a40ce",
"primary": "tuning/fixed3-tp8-gateup/serving/result-20261023.json",
"strict_prefill_seconds": 14.693329483736306,
"strict_total_seconds": 245.16398947313428,
"per_rank_memory_JSON": [
{
"rank": 0,
"worker_pid": 196,
"device": "cuda:0",
"target_parameter_logical_bytes": 6767123584,
"gateup_packed_owners": 64,
"gateup_packed_bytes": 2349609008,
"after_load_allocated_bytes": 9359171584,
"after_load_reserved_bytes": 9724493824,
"after_cache_allocated_bytes": 31438801408,
"after_cache_reserved_bytes": 31803310080
},
{
"rank": 1,
"worker_pid": 197,
"device": "cuda:1",
"target_parameter_logical_bytes": 6767123584,
"gateup_packed_owners": 64,
"gateup_packed_bytes": 2349593600,
"after_load_allocated_bytes": 9359156736,
"after_load_reserved_bytes": 9724493824,
"after_cache_allocated_bytes": 31438786560,
"after_cache_reserved_bytes": 31803310080
},
{
"rank": 2,
"worker_pid": 198,
"device": "cuda:2",
"target_parameter_logical_bytes": 6767123584,
"gateup_packed_owners": 64,
"gateup_packed_bytes": 2349373240,
"after_load_allocated_bytes": 9358935040,
"after_load_reserved_bytes": 9724493824,
"after_cache_allocated_bytes": 31438564864,
"after_cache_reserved_bytes": 31803310080
},
{
"rank": 3,
"worker_pid": 199,
"device": "cuda:3",
"target_parameter_logical_bytes": 6767123584,
"gateup_packed_owners": 64,
"gateup_packed_bytes": 2349224056,
"after_load_allocated_bytes": 9358785536,
"after_load_reserved_bytes": 9724493824,
"after_cache_allocated_bytes": 31438415360,
"after_cache_reserved_bytes": 31803310080
},
{
"rank": 4,
"worker_pid": 200,
"device": "cuda:4",
"target_parameter_logical_bytes": 6767123584,
"gateup_packed_owners": 64,
"gateup_packed_bytes": 2349203776,
"after_load_allocated_bytes": 9358765056,
"after_load_reserved_bytes": 9724493824,
"after_cache_allocated_bytes": 31438394880,
"after_cache_reserved_bytes": 31803310080
},
{
"rank": 5,
"worker_pid": 201,
"device": "cuda:5",
"target_parameter_logical_bytes": 6767123584,
"gateup_packed_owners": 64,
"gateup_packed_bytes": 2349321712,
"after_load_allocated_bytes": 9358883840,
"after_load_reserved_bytes": 9724493824,
"after_cache_allocated_bytes": 31438513664,
"after_cache_reserved_bytes": 31803310080
},
{
"rank": 6,
"worker_pid": 202,
"device": "cuda:6",
"target_parameter_logical_bytes": 6767123584,
"gateup_packed_owners": 64,
"gateup_packed_bytes": 2349479896,
"after_load_allocated_bytes": 9359043072,
"after_load_reserved_bytes": 9724493824,
"after_cache_allocated_bytes": 31438672896,
"after_cache_reserved_bytes": 31803310080
},
{
"rank": 7,
"worker_pid": 203,
"device": "cuda:7",
"target_parameter_logical_bytes": 6767123584,
"gateup_packed_owners": 64,
"gateup_packed_bytes": 2349484816,
"after_load_allocated_bytes": 9359048192,
"after_load_reserved_bytes": 9724493824,
"after_cache_allocated_bytes": 31438678016,
"after_cache_reserved_bytes": 31803310080
}
],
"KV_JSON": {
"blocks": 6477,
"block_size": 208,
"attention_example": {
"shape": [
6477,
2,
208,
1,
256
],
"stride": [
106496,
53248,
256,
256,
1
],
"dtype": "torch.float16",
"device": "cuda:0",
"bytes": 1379549184,
"storage_offset": 0
},
"conv_example": {
"shape": [
6477,
3,
1280
],
"stride": [
106496,
1280,
1
],
"dtype": "torch.float16",
"device": "cuda:0",
"bytes": 49743360,
"storage_offset": 0
},
"SSM_example": {
"shape": [
6477,
6,
128,
128
],
"stride": [
106496,
16384,
128,
1
],
"dtype": "torch.float16",
"device": "cuda:0",
"bytes": 1273430016,
"storage_offset": 3840
},
"attention_layers_per_rank": 16,
"GDN_layers_per_rank": 48,
"configured_SSM_dtype": "auto",
"configured_conv_dtype": "auto",
"logged_available_KV_GiB": 20.57,
"logged_GPU_KV_tokens": 336752,
"logged_max_concurrency_at_131072": 10.22,
"limits": "Actual views may share/interleave physical grouped allocation. Logical bytes and raw block extents are not additive physical storage or a full-context execution proof; engine grouped token/concurrency reports use different accounting."
},
"VRAM_JSON": {
"before": [
{
"render": 128,
"PCI": "0000:06:00.0",
"bytes": 33783861248,
"busy": 0
},
{
"render": 129,
"PCI": "0000:09:00.0",
"bytes": 33730301952,
"busy": 0
},
{
"render": 130,
"PCI": "0000:45:00.0",
"bytes": 33730269184,
"busy": 0
},
{
"render": 131,
"PCI": "0000:48:00.0",
"bytes": 33730314240,
"busy": 0
},
{
"render": 132,
"PCI": "0000:89:00.0",
"bytes": 33730240512,
"busy": 0
},
{
"render": 133,
"PCI": "0000:8c:00.0",
"bytes": 33730314240,
"busy": 0
},
{
"render": 134,
"PCI": "0000:c5:00.0",
"bytes": 33730297856,
"busy": 0
},
{
"render": 135,
"PCI": "0000:c8:00.0",
"bytes": 33679798272,
"busy": 0
}
],
"after": [
{
"render": 128,
"PCI": "0000:06:00.0",
"bytes": 33783861248,
"busy": 0
},
{
"render": 129,
"PCI": "0000:09:00.0",
"bytes": 33730301952,
"busy": 0
},
{
"render": 130,
"PCI": "0000:45:00.0",
"bytes": 33730269184,
"busy": 0
},
{
"render": 131,
"PCI": "0000:48:00.0",
"bytes": 33730314240,
"busy": 0
},
{
"render": 132,
"PCI": "0000:89:00.0",
"bytes": 33730240512,
"busy": 0
},
{
"render": 133,
"PCI": "0000:8c:00.0",
"bytes": 33730314240,
"busy": 0
},
{
"render": 134,
"PCI": "0000:c5:00.0",
"bytes": 33730297856,
"busy": 0
},
{
"render": 135,
"PCI": "0000:c8:00.0",
"bytes": 33679798272,
"busy": 0
}
],
"limits": "Two own-card samples, not peaks/continuous measurement or model-only allocation."
},
"attention_KV_dtype": "torch.float16",
"context": 131072,
"speculative_accounting_JSON": null
},
"detail_source": "/home/joe/PhoneSync/Documents/gfx906/baseline/qwen38/fp16-matrix.csv",
"detail_source_sha256": "8643d90cf1f211a7f77be9f6212cd7706e6dbae3ae84b9bc80b5f42fa199ffe0",
"detail_source_pointer": "FP16_TP8_gateup_MTP_off",
"retest_strict_tokens": 10210,
"retest_MTP_accounting_by_tier": {},
"retest_logged_available_KV_GiB": 20.57,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 20.57,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 336752,
"retest_memory_log_report": {
"profile": "mi50-fp16-tp8-mtp0",
"image": "sha256:0aab9fc8441d0d25b576bdb70ef0c9cbb6982eb7ac37a7f108fee2dba77a40ce",
"container": "ae82bff4b20a076f77a910c5721aa5ac0fc946b40a75fd1ab511e08b08f418aa",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 9.07,
"retest_logged_model_load_seconds": 22.691548,
"retest_logged_available_KV_GiB": 20.57,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 20.57,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 336752,
"retest_logged_context_capacity_x": 10.22,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-fp16-tp8-mtp0/service-final.log",
"source_log_sha256": "73b42cdf85e17a4b8fc446101f58a84f60f280529af15dc1711fd4dada425b46",
"sealed_index_sha256": "4578ad1dc43c02e27e9e24e23f5962720d6906c26ef4715afe24cb4f342f5279",
"source_lines": {
"model_load": {
"line": 216,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 15:59:42 [gpu_model_runner.py:4584] Model loading took 9.07 GiB memory and 22.691548 seconds"
},
"automatic_KV_budget": {
"line": 267,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:02:06 [gpu_worker.py:453] Available KV cache memory: 20.57 GiB"
},
"grouped_KV_tokens": {
"line": 274,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:02:06 [kv_cache_utils.py:1314] GPU KV cache size: 336,752 tokens"
},
"context_capacity": {
"line": 275,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:02:06 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 10.22x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-fp16-tp8-mtp0/memory-log-report.json",
"sampled_peak_VRAM_GiB": 31.463672637939453
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.20 · Validated Profile Results / table 1 | GGUF F16Qwen/Qwen3.6-27B | TP8MTP off | PASS3527 tokens | 70.759 | 66.316 | 13.168 / 41.44210K: 13.567 / 164.371Strict: 13.609 / 64.065Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 22.79 GiB · engine report373,360 grouped KV tokens · 11.32× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-27 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-27 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row27-9d44a1a055-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-27/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-27/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-27/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-27/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-27/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-27/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense" python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-27 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-27 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 71.013 · valid historical-model sample10K 66.570 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3421,
"content_chars": 5352,
"content_has_think_close": false,
"elapsed_seconds": 63.232292518951,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T23:52:09Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.012749253772199154,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 5352,
"qwen_answer_text_sha256": "f123bac2c95b41007733c66683e97c2cd19f181be25caa76dd4d86ef924aab84",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 5352,
"qwen_visible_text_sha256": "f123bac2c95b41007733c66683e97c2cd19f181be25caa76dd4d86ef924aab84",
"raw_has_think_close": false,
"reasoning_chars": 11201,
"text_chars": 16554,
"text_sha256": "95314c43ec2e43a890c71f9cbe34a9813336a76dbae28f504f567f35986f987a",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 48.71209065709263,
"vllm_decode_tps": 70.22897095675962,
"vllm_generation_tokens_delta": 3421.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.510479955002666,
"vllm_prefill_tps": 29.702670162292424,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 54.10210295593365
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 43.64001283608377,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T23:52:53Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.006201061420142651,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9558,
"text_chars": 9558,
"text_sha256": "f2524b3bdc797c6ebdb939c9ed1502650b1763a8a7da2a6a6a6b4b4f3cc0c1a9",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 28.163926037028432,
"vllm_decode_tps": 71.01282674050863,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.466903982684016,
"vllm_prefill_tps": 27.865951743317627,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 45.82950072705521
},
"10K": {
"completion_tokens": 10000,
"content_chars": 43611,
"content_has_think_close": false,
"elapsed_seconds": 163.7763210022822,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T23:55:37Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.00586194172501564,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 43611,
"qwen_answer_text_sha256": "f92ff16a3422cc7f045c52069c185be251115758eea20cd7ee6491dbc461eea6",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 43611,
"qwen_visible_text_sha256": "f92ff16a3422cc7f045c52069c185be251115758eea20cd7ee6491dbc461eea6",
"raw_has_think_close": false,
"reasoning_chars": 11201,
"text_chars": 54813,
"text_sha256": "77ec6bd95bfb18605654bceeb0d512f67f3524a5bca93893f178f8e563bf88f2",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 150.21777959540486,
"vllm_decode_tps": 66.57001605891064,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 13.549391930922866,
"vllm_prefill_tps": 31.809545564650595,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 61.058887748862375
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 27, 28, 32, 33, 34. One new sample.PASS4,508 natural tokens2K 71.02510K 66.531 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-08T22:18:52.716405342Z",
"FinishedAt": "2026-11-08T22:40:00.260467584Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "9a55da176fe75043aa7dc1ca12883e4f952738b9ebf2616b6f0edce4cb417ed7",
"retest_strict": "PASS",
"retest_strict_tokens": 4508,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 4508,
"decode_tps": 69.61732325093088,
"prefill_seconds": 14.042354583740234,
"client_total_seconds": 78.80580217391253,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 64.75399784836918,
"vllm:request_prefill_time_seconds_sum": 14.042354583740234,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 4508.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
"summary_sha256": "754f0c5eeaf987089a936b03a08af821069b38f7e14fa59e17115de3fc4d2fae"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 71.02517985930231,
"prefill_seconds": 13.268245195038617,
"client_total_seconds": 41.43482113070786,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 28.15902760066092,
"vllm:request_prefill_time_seconds_sum": 13.268245195038617,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
"summary_sha256": "aff3ec02e75684eb8bd8ecdd4936b31fc9fbb3754970d3b0a39d80e61d360495"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 66.53116656725801,
"prefill_seconds": 12.96059922594577,
"client_total_seconds": 163.27578648924828,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 150.3054961450398,
"vllm:request_prefill_time_seconds_sum": 12.96059922594577,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
"summary_sha256": "29705fa055020f3834638960e4af19073f76c5672f7f2d3c621b46c7bf61f6ef"
}
},
"retest_strict_TPS": 69.61732325093088,
"retest_strict_prefill_seconds": 14.042354583740234,
"retest_strict_total_seconds": 78.80580217391253,
"retest_2K_TPS": 71.02517985930231,
"retest_2K_prefill_seconds": 13.268245195038617,
"retest_2K_total_seconds": 41.43482113070786,
"retest_10K_TPS": 66.53116656725801,
"retest_10K_prefill_seconds": 12.96059922594577,
"retest_10K_total_seconds": 163.27578648924828,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33758752768,
"1": 33634103296,
"2": 33684492288,
"3": 33634070528,
"4": 33684193280,
"5": 33633898496,
"6": 33684754432,
"7": 33634291712
},
"retest_sampled_peak_VRAM_GiB": 31.440288543701172,
"retest_memory_observation_count": 601,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_logged_model_load_delta_GiB": 6.75,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_logged_context_capacity_x": 11.32,
"retest_memory_log_source": "historical-recovered/fresh-row-27/result.json"
} | Evidence and limitsstrict-valid; ai-info 10K gate cleared Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 3527,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r2.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-27.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 27,
"original_host": "10.0.0.20",
"logged_model_load_delta_GiB": 6.75,
"model_load_report_rank": 0,
"logged_model_load_seconds": 109.895363,
"logged_available_KV_GiB": 22.79,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 373360,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 11.32,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "1983ec5523a46271ddc4be71898407fe3211b4ad867af0641f94f757d35f9721",
"source_archive": "historical-recovered/host10.0.0.20-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/gguf-vllm-test/vnext-contract-runs/20260629T003817Z-gguf-dense27b-tp8",
"source_lines": {
"load": {
"number": 4589,
"text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 00:43:29 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 109.895363 seconds"
},
"KV": {
"number": 5172,
"text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 00:46:17 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB"
},
"tokens": {
"number": 5175,
"text": "(EngineCore_DP0 pid=200) INFO 06-29 00:46:17 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens"
},
"capacity": {
"number": 5176,
"text": "(EngineCore_DP0 pid=200) INFO 06-29 00:46:17 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x"
},
"engine": {
"number": 64,
"text": "(EngineCore_DP0 pid=200) INFO 06-29 00:41:15 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 22.79,
"logged_GPU_KV_tokens": 373360,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-27/result.json",
"retest_strict_tokens": 4508,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33758752768,
"1": 33634103296,
"2": 33684492288,
"3": 33634070528,
"4": 33684193280,
"5": 33633898496,
"6": 33684754432,
"7": 33634291712
},
"retest_sampled_peak_VRAM_GiB": 31.440288543701172,
"retest_memory_observation_count": 601,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_memory_log_source": "historical-recovered/fresh-row-27/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.30 · Validated Profile Results / table 1 | GGUF F16Qwen/Qwen3.6-27B | TP8MTP off | PASS3932 tokens | 70.959 | 66.437 | 14.844 / 43.03810K: 13.178 / 163.706Strict: 12.720 / 69.021Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 22.79 GiB · engine report373,360 grouped KV tokens · 11.32× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-28 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-28 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row28-77dcebbf74-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-28/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-28/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-28/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-28/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-28/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-28/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense" python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-28 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-28 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 70.958 · valid historical-model sample10K 66.457 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3694,
"content_chars": 7845,
"content_has_think_close": false,
"elapsed_seconds": 66.78168513812125,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T00:29:30Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.012624590657651424,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 7845,
"qwen_answer_text_sha256": "19bb233b93af9aeff8cb300a49e6cecc8bb250160e82ad7eb86c8f9f5cab80b1",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 7845,
"qwen_visible_text_sha256": "19bb233b93af9aeff8cb300a49e6cecc8bb250160e82ad7eb86c8f9f5cab80b1",
"raw_has_think_close": false,
"reasoning_chars": 11535,
"text_chars": 19381,
"text_sha256": "fa4ebf6dc67dc272987c639514d4a2b8e1dae60a4d10405870575fa479b48aef",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 52.78669147565961,
"vllm_decode_tps": 69.97975998748349,
"vllm_generation_tokens_delta": 3694.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 13.98595976922661,
"vllm_prefill_tps": 30.816619460634502,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 55.31456704573841
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 43.153645584359765,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T00:30:13Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.0065198661759495735,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 10166,
"text_chars": 10166,
"text_sha256": "4969f1b7c1cd06df2bd4cf27b864e30d6b33cd319dd4dbf7cbd9d77dd8c1fc43",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 28.185694557614625,
"vllm_decode_tps": 70.95798174892523,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.959511124528944,
"vllm_prefill_tps": 28.811101941245532,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 46.34602645772441
},
"10K": {
"completion_tokens": 10000,
"content_chars": 43778,
"content_has_think_close": false,
"elapsed_seconds": 165.04912922065705,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T00:32:58Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008748846128582954,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 43778,
"qwen_answer_text_sha256": "8d4913aa03499a61f358951edc53ab88f28f697c35447d590c08cf4c001f1f33",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 43778,
"qwen_visible_text_sha256": "8d4913aa03499a61f358951edc53ab88f28f697c35447d590c08cf4c001f1f33",
"raw_has_think_close": false,
"reasoning_chars": 11535,
"text_chars": 55314,
"text_sha256": "42b70c87469625bee9338143ee5b8cd53052c0cbcb52664510fdfc9aa2ea4312",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 150.4740112675354,
"vllm_decode_tps": 66.45665863336687,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.564758247695863,
"vllm_prefill_tps": 29.591977612686016,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 60.588020350176016
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 27, 28, 32, 33, 34. One new sample.PASS4,508 natural tokens2K 71.02510K 66.531 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-08T22:18:52.716405342Z",
"FinishedAt": "2026-11-08T22:40:00.260467584Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "9a55da176fe75043aa7dc1ca12883e4f952738b9ebf2616b6f0edce4cb417ed7",
"retest_strict": "PASS",
"retest_strict_tokens": 4508,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 4508,
"decode_tps": 69.61732325093088,
"prefill_seconds": 14.042354583740234,
"client_total_seconds": 78.80580217391253,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 64.75399784836918,
"vllm:request_prefill_time_seconds_sum": 14.042354583740234,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 4508.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
"summary_sha256": "754f0c5eeaf987089a936b03a08af821069b38f7e14fa59e17115de3fc4d2fae"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 71.02517985930231,
"prefill_seconds": 13.268245195038617,
"client_total_seconds": 41.43482113070786,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 28.15902760066092,
"vllm:request_prefill_time_seconds_sum": 13.268245195038617,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
"summary_sha256": "aff3ec02e75684eb8bd8ecdd4936b31fc9fbb3754970d3b0a39d80e61d360495"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 66.53116656725801,
"prefill_seconds": 12.96059922594577,
"client_total_seconds": 163.27578648924828,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 150.3054961450398,
"vllm:request_prefill_time_seconds_sum": 12.96059922594577,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
"summary_sha256": "29705fa055020f3834638960e4af19073f76c5672f7f2d3c621b46c7bf61f6ef"
}
},
"retest_strict_TPS": 69.61732325093088,
"retest_strict_prefill_seconds": 14.042354583740234,
"retest_strict_total_seconds": 78.80580217391253,
"retest_2K_TPS": 71.02517985930231,
"retest_2K_prefill_seconds": 13.268245195038617,
"retest_2K_total_seconds": 41.43482113070786,
"retest_10K_TPS": 66.53116656725801,
"retest_10K_prefill_seconds": 12.96059922594577,
"retest_10K_total_seconds": 163.27578648924828,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33758752768,
"1": 33634103296,
"2": 33684492288,
"3": 33634070528,
"4": 33684193280,
"5": 33633898496,
"6": 33684754432,
"7": 33634291712
},
"retest_sampled_peak_VRAM_GiB": 31.440288543701172,
"retest_memory_observation_count": 601,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_logged_model_load_delta_GiB": 6.75,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_logged_context_capacity_x": 11.32,
"retest_memory_log_source": "historical-recovered/fresh-row-27/result.json"
} | Evidence and limitsstrict-valid; ai-info 10K gate cleared Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 3932,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r2.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-28.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 28,
"original_host": "10.0.0.30",
"logged_model_load_delta_GiB": 6.75,
"model_load_report_rank": 0,
"logged_model_load_seconds": 121.718155,
"logged_available_KV_GiB": 22.79,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 373360,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 11.32,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "c954e80c7e73a96ae72d149bfefde298e35ede878ea1d50e3d8a8b4051bd22aa",
"source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/gguf-vllm-test/vnext-contract-runs/20260629T003817Z-gguf-dense27b-tp8",
"source_lines": {
"load": {
"number": 4589,
"text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 00:43:42 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 121.718155 seconds"
},
"KV": {
"number": 5172,
"text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 00:46:32 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB"
},
"tokens": {
"number": 5175,
"text": "(EngineCore_DP0 pid=200) INFO 06-29 00:46:32 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens"
},
"capacity": {
"number": 5176,
"text": "(EngineCore_DP0 pid=200) INFO 06-29 00:46:32 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x"
},
"engine": {
"number": 64,
"text": "(EngineCore_DP0 pid=200) INFO 06-29 00:41:16 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 22.79,
"logged_GPU_KV_tokens": 373360,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-27/result.json",
"retest_strict_tokens": 4508,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33758752768,
"1": 33634103296,
"2": 33684492288,
"3": 33634070528,
"4": 33684193280,
"5": 33633898496,
"6": 33684754432,
"7": 33634291712
},
"retest_sampled_peak_VRAM_GiB": 31.440288543701172,
"retest_memory_observation_count": 601,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_memory_log_source": "historical-recovered/fresh-row-27/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.20 · Validated Profile Results / table 1 | HF FP16Qwen/Qwen3.6-27B | TP8MTP off | PASS4120 tokens | 71.316 | 66.824 | 13.802 / 41.85410K: 14.736 / 164.390Strict: 14.254 / 72.978Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.84 GiB (rank 0)Available KV: 22.64 GiB · engine report370,864 grouped KV tokens · 11.25× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-29 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-29 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row29-db10e5f4ad-reproduction \ --network \ host \ --ipc \ host \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-29/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-29/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-29/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-29/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-29/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-29/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/opt/local-models/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-29 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-29 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 71.157 · valid historical-model sample10K 66.643 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3234,
"content_chars": 5935,
"content_has_think_close": false,
"elapsed_seconds": 60.3153067426756,
"endpoint": "http://127.0.0.1:19078/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T00:59:25Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.013067858293652534,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8078/metrics",
"model": "Qwen3.6-27B-HF-CausalLM",
"prompt_repeat": 32,
"qwen_answer_text_chars": 5935,
"qwen_answer_text_sha256": "52e4d8f1dd1b31e6269fcaa4b1424c12f64ce1d16e326317fb8f98e0833b2856",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 5935,
"qwen_visible_text_sha256": "52e4d8f1dd1b31e6269fcaa4b1424c12f64ce1d16e326317fb8f98e0833b2856",
"raw_has_think_close": false,
"reasoning_chars": 9949,
"text_chars": 15885,
"text_sha256": "da8b23f5c947a638d508077cbc6787afe6aeb529f98d038fbbee08dccc0c1c1e",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 45.86109949275851,
"vllm_decode_tps": 70.51728012998576,
"vllm_generation_tokens_delta": 3234.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.444526282139122,
"vllm_prefill_tps": 29.838292484049,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 53.61823017492523
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 42.0066326521337,
"endpoint": "http://127.0.0.1:19078/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T01:00:07Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.009246834553778172,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8078/metrics",
"model": "Qwen3.6-27B-HF-CausalLM",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9629,
"text_chars": 9629,
"text_sha256": "166f1a5cf5e0080402ebf7099d53ca119066ece8cb52882a4fe8cd5ec401156f",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 28.1066905381158,
"vllm_decode_tps": 71.15743482100027,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 13.89147546235472,
"vllm_prefill_tps": 31.02622188463643,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 47.61152879266582
},
"10K": {
"completion_tokens": 10000,
"content_chars": 40547,
"content_has_think_close": false,
"elapsed_seconds": 164.41756441816688,
"endpoint": "http://127.0.0.1:19078/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T01:02:51Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008683929219841957,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8078/metrics",
"model": "Qwen3.6-27B-HF-CausalLM",
"prompt_repeat": 32,
"qwen_answer_text_chars": 40547,
"qwen_answer_text_sha256": "6b37071096f0f0ba680c6493ff5fd736b9a43e9e66fcc01a39b831b99863482a",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 40547,
"qwen_visible_text_sha256": "6b37071096f0f0ba680c6493ff5fd736b9a43e9e66fcc01a39b831b99863482a",
"raw_has_think_close": false,
"reasoning_chars": 9949,
"text_chars": 50497,
"text_sha256": "10fcf1e7cefef0bd9ff1f9498f82bd0f5694e9329aceaffff40ea4d827635d54",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 150.05429869424552,
"vllm_decode_tps": 66.64254264635402,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.353619427420199,
"vllm_prefill_tps": 30.027269580287623,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 60.82075254786512
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 29. One new sample.PASS3,894 natural tokens2K 71.36010K 66.825 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-08T22:58:14.831580175Z",
"FinishedAt": "2026-11-08T23:16:33.728315227Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "1d2c3350fafbf79cf6d1aca4a68e17a670c7f2e384ce7589e2d7f8bce876d262",
"retest_strict": "PASS",
"retest_strict_tokens": 3894,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 3894,
"decode_tps": 70.29589397473099,
"prefill_seconds": 14.098794215358794,
"client_total_seconds": 69.50212777592242,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 3894.0,
"vllm:request_prefill_time_seconds_sum": 14.098794215358794,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 55.39441608637571
},
"request_sha256": "d6b5acc7f81647ef18db224f701221c1f68a009b4548f6849cf87c3c8e419a9a",
"summary_sha256": "7edb828e4d7722b0c1fd383897c084e95cdb225e7c5a037b23b28761ada4bc47"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 71.35954591433956,
"prefill_seconds": 13.803319579921663,
"client_total_seconds": 41.83979881927371,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_prefill_time_seconds_sum": 13.803319579921663,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 28.02708417456597
},
"request_sha256": "3d137c416021d94f838f557d4f42b0fbfe62d81a216b8c0a16b65a1f529aca97",
"summary_sha256": "35b66db175a416934d4cd1895d171f23baa26dd6de6eeecc1ae3c60ef2905b15"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 66.82477816892438,
"prefill_seconds": 14.576330919750035,
"client_total_seconds": 164.2314986139536,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:request_prefill_time_seconds_sum": 14.576330919750035,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 149.64509084820747
},
"request_sha256": "d03cae7830251b8462de5cfc7ed217442c2137d550fefb2baf4f6c92b612c39a",
"summary_sha256": "09a50ce3034dc4380f7c301323163264cf522177a86be5b4be1fa6e0bdcbc20c"
}
},
"retest_strict_TPS": 70.29589397473099,
"retest_strict_prefill_seconds": 14.098794215358794,
"retest_strict_total_seconds": 69.50212777592242,
"retest_2K_TPS": 71.35954591433956,
"retest_2K_prefill_seconds": 13.803319579921663,
"retest_2K_total_seconds": 41.83979881927371,
"retest_10K_TPS": 66.82477816892438,
"retest_10K_prefill_seconds": 14.576330919750035,
"retest_10K_total_seconds": 164.2314986139536,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33791266816,
"1": 33666322432,
"2": 33716637696,
"3": 33666342912,
"4": 33716674560,
"5": 33666342912,
"6": 33716662272,
"7": 33666334720
},
"retest_sampled_peak_VRAM_GiB": 31.470569610595703,
"retest_memory_observation_count": 517,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 723,
"text": "2026-11-08T22:59:57.180783513Z (Worker pid=327) (Worker_TP0 pid=327) INFO 11-08 22:59:57 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 9.563808 seconds",
"values": [
"6.84"
]
}
],
"available_KV_GiB": [
{
"line": 1271,
"text": "2026-11-08T23:01:39.506595611Z (Worker pid=327) (Worker_TP0 pid=327) INFO 11-08 23:01:39 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
"values": [
"22.64"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 1284,
"text": "2026-11-08T23:01:40.213405262Z (EngineCore_DP0 pid=273) INFO 11-08 23:01:40 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
"values": [
"370,864"
]
}
],
"context_capacity": [
{
"line": 1285,
"text": "2026-11-08T23:01:40.213475072Z (EngineCore_DP0 pid=273) INFO 11-08 23:01:40 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
"values": [
"131,072",
"11.25"
]
}
]
},
"retest_logged_model_load_delta_GiB": 6.84,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 22.64,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 370864,
"retest_logged_context_capacity_x": 11.25,
"retest_memory_log_source": "historical-recovered/fresh-row-29/result.json"
} | Evidence and limitsfresh-cache release-band control Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 4120,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r3.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-29.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 29,
"original_host": "10.0.0.20",
"logged_model_load_delta_GiB": 6.84,
"model_load_report_rank": 0,
"logged_model_load_seconds": 7.221931,
"logged_available_KV_GiB": 22.64,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 370864,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 11.25,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "7b29916dd30992ee4d28b5e1b69537e1adfb8405d4bb0bc254c1314d926e1efb",
"source_archive": "historical-recovered/host10.0.0.20-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/gguf-vllm-test/vnext-package-runs/hf_dense27b_dot20_releasebundle_freshcache_20260628T203743Z/launch",
"source_lines": {
"load": {
"number": 723,
"text": "(Worker pid=327) (Worker_TP0 pid=327) INFO 06-28 20:39:17 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 7.221931 seconds"
},
"KV": {
"number": 1273,
"text": "(Worker pid=327) (Worker_TP0 pid=327) INFO 06-28 20:40:54 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB"
},
"tokens": {
"number": 1284,
"text": "(EngineCore_DP0 pid=273) INFO 06-28 20:40:55 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens"
},
"capacity": {
"number": 1285,
"text": "(EngineCore_DP0 pid=273) INFO 06-28 20:40:55 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x"
},
"engine": {
"number": 57,
"text": "(EngineCore_DP0 pid=273) INFO 06-28 20:38:44 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9', speculative_config=None, tokenizer='/opt/local-models/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-HF-CausalLM, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 22.64,
"logged_GPU_KV_tokens": 370864,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-29/result.json",
"retest_strict_tokens": 3894,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33791266816,
"1": 33666322432,
"2": 33716637696,
"3": 33666342912,
"4": 33716674560,
"5": 33666342912,
"6": 33716662272,
"7": 33666334720
},
"retest_sampled_peak_VRAM_GiB": 31.470569610595703,
"retest_memory_observation_count": 517,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 723,
"text": "2026-11-08T22:59:57.180783513Z (Worker pid=327) (Worker_TP0 pid=327) INFO 11-08 22:59:57 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 9.563808 seconds",
"values": [
"6.84"
]
}
],
"available_KV_GiB": [
{
"line": 1271,
"text": "2026-11-08T23:01:39.506595611Z (Worker pid=327) (Worker_TP0 pid=327) INFO 11-08 23:01:39 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
"values": [
"22.64"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 1284,
"text": "2026-11-08T23:01:40.213405262Z (EngineCore_DP0 pid=273) INFO 11-08 23:01:40 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
"values": [
"370,864"
]
}
],
"context_capacity": [
{
"line": 1285,
"text": "2026-11-08T23:01:40.213475072Z (EngineCore_DP0 pid=273) INFO 11-08 23:01:40 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
"values": [
"131,072",
"11.25"
]
}
]
},
"retest_KV_budget_GiB": 22.64,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 370864,
"retest_memory_log_source": "historical-recovered/fresh-row-29/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.20 · Validated Profile Results / table 1 | HF FP16Qwen/Qwen3.6-35B-A3B | TP8MTP off | PASS2978 tokens | 115.549 | 108.805 | 0.811 / 18.12810K: 0.803 / 92.720Strict: 0.808 / 26.704Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 8.88 GiB (rank 0)Available KV: 19.91 GiB · engine report521,696 grouped KV tokens · 15.82× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-30 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-30 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row30-aaf874ca19-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-30/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-30/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-30/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-30/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-30/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-30/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-moe,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/995ad96eacd98c81ed38be0c5b274b04031597b0,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-moe "$MODEL_DIR/qwen36-moe" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-30 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-30 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 115.297 · valid historical-model sample10K 108.623 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 4123,
"content_chars": 8534,
"content_has_think_close": false,
"elapsed_seconds": 37.004668129608035,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T01:39:58Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.008319173008203506,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 8534,
"qwen_answer_text_sha256": "ba0a6d00b3670679073d7eb1a571b6b29db4a839be80a8c8f3d86afb09ff8220",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 8534,
"qwen_visible_text_sha256": "ba0a6d00b3670679073d7eb1a571b6b29db4a839be80a8c8f3d86afb09ff8220",
"raw_has_think_close": false,
"reasoning_chars": 13258,
"text_chars": 21793,
"text_sha256": "28fd44aa56787b93626bd24f8604c3893ad729899b4d70e8490b500bb3564071",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 36.186177032999694,
"vllm_decode_tps": 113.93853504447466,
"vllm_generation_tokens_delta": 4123.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8094438994303346,
"vllm_prefill_tps": 532.4643255738001,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 111.41837525901552
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 18.165475289337337,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T01:40:16Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.00866091437637806,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 10313,
"text_chars": 10313,
"text_sha256": "6af2a531c6bc5cacd9e2bcf5903c638455cd83162d39d986b0f3f6116ea655fc",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 17.346558371558785,
"vllm_decode_tps": 115.2966460066901,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8107803724706173,
"vllm_prefill_tps": 531.5866227578412,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 110.0989634537087
},
"10K": {
"completion_tokens": 10000,
"content_chars": 42964,
"content_has_think_close": false,
"elapsed_seconds": 92.88607190269977,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T01:41:49Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.006269831210374832,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 42964,
"qwen_answer_text_sha256": "1b3fc40a96673d78a8c2338d0b94b336673bd4d367e7b03132e9b010a80191fc",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 42964,
"qwen_visible_text_sha256": "1b3fc40a96673d78a8c2338d0b94b336673bd4d367e7b03132e9b010a80191fc",
"raw_has_think_close": false,
"reasoning_chars": 13258,
"text_chars": 56223,
"text_sha256": "d50a79daf4fa5658877f56d67da4aab8062cd4ffee794376866171c17fb4be48",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 92.06168713700026,
"vllm_decode_tps": 108.62281923118186,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8135625869035721,
"vllm_prefill_tps": 529.7687073349705,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 107.65876729586782
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 30, 31. One new sample.PASS3,315 natural tokens2K 116.30310K 109.479 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-08T23:18:06.809781202Z",
"FinishedAt": "2026-11-08T23:38:48.306386482Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 174,
"retest_historical_manifest_sha256": "9a2772d75889092782cc9a65e91fb732b333c30fc7469365a71fbf2c8f50798e",
"retest_strict": "PASS",
"retest_strict_tokens": 3315,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 3315,
"decode_tps": 115.49001974098105,
"prefill_seconds": 0.8086845744401217,
"client_total_seconds": 29.521306382492185,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 3315.0,
"vllm:request_prefill_time_seconds_sum": 0.8086845744401217,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 28.703778970986605,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "e31c712c0771301f2e4d592bc966448d3143f2660e1b4cab33a91704f19e9370",
"summary_sha256": "6de45c7cc013914550cba9d9e4fb391546d47711db7c87c3248b42dcf7698c6d"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 116.30302059984058,
"prefill_seconds": 0.81096709985286,
"client_total_seconds": 18.01519708149135,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 2000.0,
"vllm:request_prefill_time_seconds_sum": 0.81096709985286,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 17.19645792245865,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "bca01af4a8338cd6274c1c9e0b281d8e59066ebae2ece3e3a636620066da3a03",
"summary_sha256": "0acf3601e108fbbfd1a891c8d6a16ccbe40e590b194451e109627d82f3c5e96d"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 109.47943821033532,
"prefill_seconds": 0.8086706819012761,
"client_total_seconds": 92.15940057393163,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 10000.0,
"vllm:request_prefill_time_seconds_sum": 0.8086706819012761,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 91.34135289210826,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "46fd7808353544eb86016bdd1746bafb67def6864a3f5352647b66cb9149f95a",
"summary_sha256": "a8ba495779711e5602e6164c8bec0f7728c656bfa02a9661baafa8ef58f71683"
}
},
"retest_strict_TPS": 115.49001974098105,
"retest_strict_prefill_seconds": 0.8086845744401217,
"retest_strict_total_seconds": 29.521306382492185,
"retest_2K_TPS": 116.30302059984058,
"retest_2K_prefill_seconds": 0.81096709985286,
"retest_2K_total_seconds": 18.01519708149135,
"retest_10K_TPS": 109.47943821033532,
"retest_10K_prefill_seconds": 0.8086706819012761,
"retest_10K_total_seconds": 92.15940057393163,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 34334515200,
"1": 34314481664,
"2": 34314469376,
"3": 34314502144,
"4": 34314551296,
"5": 34314530816,
"6": 34314477568,
"7": 34318667776
},
"retest_sampled_peak_VRAM_GiB": 31.97650909423828,
"retest_memory_observation_count": 588,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 193,
"text": "2026-11-08T23:23:02.194194967Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:23:02 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 193.073592 seconds",
"values": [
"8.88"
]
}
],
"available_KV_GiB": [
{
"line": 311,
"text": "2026-11-08T23:24:25.909888257Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:24:25 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
"values": [
"19.91"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 315,
"text": "2026-11-08T23:24:26.013151548Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
"values": [
"521,696"
]
}
],
"context_capacity": [
{
"line": 316,
"text": "2026-11-08T23:24:26.013251748Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
"values": [
"131,072",
"15.82"
]
}
]
},
"retest_logged_model_load_delta_GiB": 8.88,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 19.91,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 521696,
"retest_logged_context_capacity_x": 15.82,
"retest_memory_log_source": "historical-recovered/fresh-row-30/result.json"
} | Evidence and limitsstrict-valid Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 2978,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "995ad96eacd98c81ed38be0c5b274b04031597b0",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r3.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-30.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 30,
"original_host": "10.0.0.20",
"logged_model_load_delta_GiB": 8.88,
"model_load_report_rank": 0,
"logged_model_load_seconds": 189.378052,
"logged_available_KV_GiB": 19.91,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 521696,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 15.82,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "b89b1bada4eac6d54c85c9f77c9c1aa0aee0fb9ade0f826fbeb8e919bf371f4c",
"source_archive": "historical-recovered/host10.0.0.20-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/gguf-vllm-test/vnext-contract-runs/20260629T003817Z-hf-moe35b-tp8",
"source_lines": {
"load": {
"number": 193,
"text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-29 01:42:17 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 189.378052 seconds"
},
"KV": {
"number": 308,
"text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-29 01:43:40 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB"
},
"tokens": {
"number": 315,
"text": "(EngineCore_DP0 pid=171) INFO 06-29 01:43:41 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens"
},
"capacity": {
"number": 316,
"text": "(EngineCore_DP0 pid=171) INFO 06-29 01:43:41 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x"
},
"engine": {
"number": 33,
"text": "(EngineCore_DP0 pid=171) INFO 06-29 01:38:37 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/995ad96eacd98c81ed38be0c5b274b04031597b0', speculative_config=None, tokenizer='/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/995ad96eacd98c81ed38be0c5b274b04031597b0', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.6-35B-A3B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 19.91,
"logged_GPU_KV_tokens": 521696,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-30/result.json",
"retest_strict_tokens": 3315,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 34334515200,
"1": 34314481664,
"2": 34314469376,
"3": 34314502144,
"4": 34314551296,
"5": 34314530816,
"6": 34314477568,
"7": 34318667776
},
"retest_sampled_peak_VRAM_GiB": 31.97650909423828,
"retest_memory_observation_count": 588,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 193,
"text": "2026-11-08T23:23:02.194194967Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:23:02 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 193.073592 seconds",
"values": [
"8.88"
]
}
],
"available_KV_GiB": [
{
"line": 311,
"text": "2026-11-08T23:24:25.909888257Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:24:25 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
"values": [
"19.91"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 315,
"text": "2026-11-08T23:24:26.013151548Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
"values": [
"521,696"
]
}
],
"context_capacity": [
{
"line": 316,
"text": "2026-11-08T23:24:26.013251748Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
"values": [
"131,072",
"15.82"
]
}
]
},
"retest_KV_budget_GiB": 19.91,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 521696,
"retest_memory_log_source": "historical-recovered/fresh-row-30/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.30 · Validated Profile Results / table 1 | HF FP16Qwen/Qwen3.6-35B-A3B | TP8MTP off | PASS3103 tokens | 115.532 | 108.673 | 0.809 / 18.12810K: 0.807 / 92.836Strict: 0.808 / 27.870Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 8.88 GiB (rank 0)Available KV: 19.91 GiB · engine report521,696 grouped KV tokens · 15.82× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-31 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-31 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row31-90a75956f3-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-31/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-31/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-31/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-31/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-31/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-31/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-moe,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/995ad96eacd98c81ed38be0c5b274b04031597b0,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-moe "$MODEL_DIR/qwen36-moe" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-31 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-31 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 115.825 · valid historical-model sample10K 108.889 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3074,
"content_chars": 6062,
"content_has_think_close": false,
"elapsed_seconds": 27.538480462506413,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T02:18:25Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.01051213126629591,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 6062,
"qwen_answer_text_sha256": "1da5eaf2d1a2df7954628036116f036612a2e11e62d19ab301d81891297acee2",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 6062,
"qwen_visible_text_sha256": "1da5eaf2d1a2df7954628036116f036612a2e11e62d19ab301d81891297acee2",
"raw_has_think_close": false,
"reasoning_chars": 9805,
"text_chars": 15868,
"text_sha256": "c3953bbf14c8bf2a03c07aa8de91765d98935a3422c8b9473e825d5678ae9947",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 26.723272138275206,
"vllm_decode_tps": 115.03082347453893,
"vllm_generation_tokens_delta": 3074.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8065544962882996,
"vllm_prefill_tps": 534.3718273017238,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 111.62562161645937
},
"2K": {
"completion_tokens": 2000,
"content_chars": 200,
"content_has_think_close": false,
"elapsed_seconds": 18.08423571381718,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T02:18:44Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.006676186807453632,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 200,
"qwen_answer_text_sha256": "2f213ad86009aacf37f15220812d39d2ddf2e8288ddb317a100ac0d32c11b69c",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 200,
"qwen_visible_text_sha256": "2f213ad86009aacf37f15220812d39d2ddf2e8288ddb317a100ac0d32c11b69c",
"raw_has_think_close": false,
"reasoning_chars": 9805,
"text_chars": 10006,
"text_sha256": "622b7aa36324f0b39addb94e14df1edc548b4e7e33e4a6780113bca38cdf628a",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 17.267360970377922,
"vllm_decode_tps": 115.82545841434546,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8081611143425107,
"vllm_prefill_tps": 533.3095002357857,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 110.59355958692294
},
"10K": {
"completion_tokens": 10000,
"content_chars": 48347,
"content_has_think_close": false,
"elapsed_seconds": 92.6585035957396,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T02:20:16Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.005911114625632763,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 48347,
"qwen_answer_text_sha256": "ae73d5c7cbbdccaf573352ad53b4ff705746df31b9b2da9b40e640fc29123613",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 48347,
"qwen_visible_text_sha256": "ae73d5c7cbbdccaf573352ad53b4ff705746df31b9b2da9b40e640fc29123613",
"raw_has_think_close": false,
"reasoning_chars": 9805,
"text_chars": 58153,
"text_sha256": "0dbb2aae366ae7556a2270143f96ce9c0efbad0938d8f1b734817ccb8d9939b1",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 91.83650601096451,
"vllm_decode_tps": 108.88916003409454,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8104435317218304,
"vllm_prefill_tps": 531.8075635501928,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 107.9231760921703
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 30, 31. One new sample.PASS3,315 natural tokens2K 116.30310K 109.479 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-08T23:18:06.809781202Z",
"FinishedAt": "2026-11-08T23:38:48.306386482Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 174,
"retest_historical_manifest_sha256": "9a2772d75889092782cc9a65e91fb732b333c30fc7469365a71fbf2c8f50798e",
"retest_strict": "PASS",
"retest_strict_tokens": 3315,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 3315,
"decode_tps": 115.49001974098105,
"prefill_seconds": 0.8086845744401217,
"client_total_seconds": 29.521306382492185,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 3315.0,
"vllm:request_prefill_time_seconds_sum": 0.8086845744401217,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 28.703778970986605,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "e31c712c0771301f2e4d592bc966448d3143f2660e1b4cab33a91704f19e9370",
"summary_sha256": "6de45c7cc013914550cba9d9e4fb391546d47711db7c87c3248b42dcf7698c6d"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 116.30302059984058,
"prefill_seconds": 0.81096709985286,
"client_total_seconds": 18.01519708149135,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 2000.0,
"vllm:request_prefill_time_seconds_sum": 0.81096709985286,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 17.19645792245865,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "bca01af4a8338cd6274c1c9e0b281d8e59066ebae2ece3e3a636620066da3a03",
"summary_sha256": "0acf3601e108fbbfd1a891c8d6a16ccbe40e590b194451e109627d82f3c5e96d"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 109.47943821033532,
"prefill_seconds": 0.8086706819012761,
"client_total_seconds": 92.15940057393163,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 10000.0,
"vllm:request_prefill_time_seconds_sum": 0.8086706819012761,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 91.34135289210826,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "46fd7808353544eb86016bdd1746bafb67def6864a3f5352647b66cb9149f95a",
"summary_sha256": "a8ba495779711e5602e6164c8bec0f7728c656bfa02a9661baafa8ef58f71683"
}
},
"retest_strict_TPS": 115.49001974098105,
"retest_strict_prefill_seconds": 0.8086845744401217,
"retest_strict_total_seconds": 29.521306382492185,
"retest_2K_TPS": 116.30302059984058,
"retest_2K_prefill_seconds": 0.81096709985286,
"retest_2K_total_seconds": 18.01519708149135,
"retest_10K_TPS": 109.47943821033532,
"retest_10K_prefill_seconds": 0.8086706819012761,
"retest_10K_total_seconds": 92.15940057393163,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 34334515200,
"1": 34314481664,
"2": 34314469376,
"3": 34314502144,
"4": 34314551296,
"5": 34314530816,
"6": 34314477568,
"7": 34318667776
},
"retest_sampled_peak_VRAM_GiB": 31.97650909423828,
"retest_memory_observation_count": 588,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 193,
"text": "2026-11-08T23:23:02.194194967Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:23:02 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 193.073592 seconds",
"values": [
"8.88"
]
}
],
"available_KV_GiB": [
{
"line": 311,
"text": "2026-11-08T23:24:25.909888257Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:24:25 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
"values": [
"19.91"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 315,
"text": "2026-11-08T23:24:26.013151548Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
"values": [
"521,696"
]
}
],
"context_capacity": [
{
"line": 316,
"text": "2026-11-08T23:24:26.013251748Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
"values": [
"131,072",
"15.82"
]
}
]
},
"retest_logged_model_load_delta_GiB": 8.88,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 19.91,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 521696,
"retest_logged_context_capacity_x": 15.82,
"retest_memory_log_source": "historical-recovered/fresh-row-30/result.json"
} | Evidence and limitsstrict-valid Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 3103,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "995ad96eacd98c81ed38be0c5b274b04031597b0",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r3.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-31.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 31,
"original_host": "10.0.0.30",
"logged_model_load_delta_GiB": 8.88,
"model_load_report_rank": 0,
"logged_model_load_seconds": 191.364916,
"logged_available_KV_GiB": 19.91,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 521696,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 15.82,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "72649d16114bbe7ecda6325c3b0eff1b7e054304b6dcfb050476b2307b2ee7ba",
"source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/gguf-vllm-test/vnext-contract-runs/20260629T003817Z-hf-moe35b-tp8",
"source_lines": {
"load": {
"number": 193,
"text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-29 01:42:22 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 191.364916 seconds"
},
"KV": {
"number": 308,
"text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-29 01:43:44 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB"
},
"tokens": {
"number": 315,
"text": "(EngineCore_DP0 pid=171) INFO 06-29 01:43:45 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens"
},
"capacity": {
"number": 316,
"text": "(EngineCore_DP0 pid=171) INFO 06-29 01:43:45 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x"
},
"engine": {
"number": 33,
"text": "(EngineCore_DP0 pid=171) INFO 06-29 01:38:36 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/995ad96eacd98c81ed38be0c5b274b04031597b0', speculative_config=None, tokenizer='/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/995ad96eacd98c81ed38be0c5b274b04031597b0', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.6-35B-A3B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 19.91,
"logged_GPU_KV_tokens": 521696,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-30/result.json",
"retest_strict_tokens": 3315,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 34334515200,
"1": 34314481664,
"2": 34314469376,
"3": 34314502144,
"4": 34314551296,
"5": 34314530816,
"6": 34314477568,
"7": 34318667776
},
"retest_sampled_peak_VRAM_GiB": 31.97650909423828,
"retest_memory_observation_count": 588,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 193,
"text": "2026-11-08T23:23:02.194194967Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:23:02 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 193.073592 seconds",
"values": [
"8.88"
]
}
],
"available_KV_GiB": [
{
"line": 311,
"text": "2026-11-08T23:24:25.909888257Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-08 23:24:25 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
"values": [
"19.91"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 315,
"text": "2026-11-08T23:24:26.013151548Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
"values": [
"521,696"
]
}
],
"context_capacity": [
{
"line": 316,
"text": "2026-11-08T23:24:26.013251748Z (EngineCore_DP0 pid=171) INFO 11-08 23:24:26 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
"values": [
"131,072",
"15.82"
]
}
]
},
"retest_KV_budget_GiB": 19.91,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 521696,
"retest_memory_log_source": "historical-recovered/fresh-row-30/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.30 · Public-Staged POSIX-Runner Reproduction Audit / table 1 | GGUF F16Qwen/Qwen3.6-27B | TP8MTP off | PASS3636 tokens | 70.935 | 66.403 | 12.073 / 40.27610K: 12.556 / 163.163Strict: 13.232 / 65.194Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 23.15 GiB · engine report379,184 grouped KV tokens · 11.50× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-32 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-32 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row32-8e5b34f5cf-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-32/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-32/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-32/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-32/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-32/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-32/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense" python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-32 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-32 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 70.912 · valid historical-model sample10K 66.452 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3687,
"content_chars": 5329,
"content_has_think_close": false,
"elapsed_seconds": 65.07993041258305,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T02:56:48Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.012216160073876381,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 5329,
"qwen_answer_text_sha256": "3a7a6b602375e1ddc6c8e18ad6a2710bf65108182452f41ace63f1ab0f80812d",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 5329,
"qwen_visible_text_sha256": "3a7a6b602375e1ddc6c8e18ad6a2710bf65108182452f41ace63f1ab0f80812d",
"raw_has_think_close": false,
"reasoning_chars": 12365,
"text_chars": 17695,
"text_sha256": "5b3f450c991bceef669328df77eae3d6b223ed3d3ba745f87d3c95e9ee2d591e",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 52.69167762435973,
"vllm_decode_tps": 69.973099476633,
"vllm_generation_tokens_delta": 3687.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 12.377928133122623,
"vllm_prefill_tps": 34.8200438203118,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 56.65341030062206
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 42.01967437751591,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T02:57:30Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008349246345460415,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9610,
"text_chars": 9610,
"text_sha256": "b6e619acbecd7b98e7bd3fafcd526951e3260b60cf5909be4ad3a5f325565bff",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 28.204140285030007,
"vllm_decode_tps": 70.91157467620262,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 13.807229696772993,
"vllm_prefill_tps": 31.215530520270317,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 47.59675151290962
},
"10K": {
"completion_tokens": 10000,
"content_chars": 45133,
"content_has_think_close": false,
"elapsed_seconds": 165.0049143144861,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T03:00:15Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.009004133753478527,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 45133,
"qwen_answer_text_sha256": "79555b8c674ed6168687a63258550a0f08ef0033b17b0f5b24d96f2e305de681",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 45133,
"qwen_visible_text_sha256": "79555b8c674ed6168687a63258550a0f08ef0033b17b0f5b24d96f2e305de681",
"raw_has_think_close": false,
"reasoning_chars": 12365,
"text_chars": 57499,
"text_sha256": "54c178bd283aaf13a7da9ffef6cbdb53f7833574997cb0e5421dce9f6dd6cf24",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 150.48369926866144,
"vllm_decode_tps": 66.45238021526045,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.511613338254392,
"vllm_prefill_tps": 29.70035033002369,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 60.60425558562944
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 27, 28, 32, 33, 34. One new sample.PASS4,508 natural tokens2K 71.02510K 66.531 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-08T22:18:52.716405342Z",
"FinishedAt": "2026-11-08T22:40:00.260467584Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "9a55da176fe75043aa7dc1ca12883e4f952738b9ebf2616b6f0edce4cb417ed7",
"retest_strict": "PASS",
"retest_strict_tokens": 4508,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 4508,
"decode_tps": 69.61732325093088,
"prefill_seconds": 14.042354583740234,
"client_total_seconds": 78.80580217391253,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 64.75399784836918,
"vllm:request_prefill_time_seconds_sum": 14.042354583740234,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 4508.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
"summary_sha256": "754f0c5eeaf987089a936b03a08af821069b38f7e14fa59e17115de3fc4d2fae"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 71.02517985930231,
"prefill_seconds": 13.268245195038617,
"client_total_seconds": 41.43482113070786,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 28.15902760066092,
"vllm:request_prefill_time_seconds_sum": 13.268245195038617,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
"summary_sha256": "aff3ec02e75684eb8bd8ecdd4936b31fc9fbb3754970d3b0a39d80e61d360495"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 66.53116656725801,
"prefill_seconds": 12.96059922594577,
"client_total_seconds": 163.27578648924828,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 150.3054961450398,
"vllm:request_prefill_time_seconds_sum": 12.96059922594577,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
"summary_sha256": "29705fa055020f3834638960e4af19073f76c5672f7f2d3c621b46c7bf61f6ef"
}
},
"retest_strict_TPS": 69.61732325093088,
"retest_strict_prefill_seconds": 14.042354583740234,
"retest_strict_total_seconds": 78.80580217391253,
"retest_2K_TPS": 71.02517985930231,
"retest_2K_prefill_seconds": 13.268245195038617,
"retest_2K_total_seconds": 41.43482113070786,
"retest_10K_TPS": 66.53116656725801,
"retest_10K_prefill_seconds": 12.96059922594577,
"retest_10K_total_seconds": 163.27578648924828,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33758752768,
"1": 33634103296,
"2": 33684492288,
"3": 33634070528,
"4": 33684193280,
"5": 33633898496,
"6": 33684754432,
"7": 33634291712
},
"retest_sampled_peak_VRAM_GiB": 31.440288543701172,
"retest_memory_observation_count": 601,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_logged_model_load_delta_GiB": 6.75,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_logged_context_capacity_x": 11.32,
"retest_memory_log_source": "historical-recovered/fresh-row-27/result.json"
} | Evidence and limitsstrict-valid; ai-info 10K gate cleared Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 3636,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r2.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-32.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 32,
"original_host": "10.0.0.30",
"logged_model_load_delta_GiB": 6.75,
"model_load_report_rank": 0,
"logged_model_load_seconds": 109.957728,
"logged_available_KV_GiB": 23.15,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 379184,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 11.5,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "759f71b82d879600a80aa25cd5340f4bcf248dd73784a58ffaa097e640117f39",
"source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/vnext-release-repro/localaiservers/qwen36-gfx906/vnext-launch-runs/repro-gguf-dense27b-tp8",
"source_lines": {
"load": {
"number": 4589,
"text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 12:36:09 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 109.957728 seconds"
},
"KV": {
"number": 5245,
"text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 12:37:30 [gpu_worker.py:453] Available KV cache memory: 23.15 GiB"
},
"tokens": {
"number": 5252,
"text": "(EngineCore_DP0 pid=200) INFO 06-29 12:37:31 [kv_cache_utils.py:1314] GPU KV cache size: 379,184 tokens"
},
"capacity": {
"number": 5253,
"text": "(EngineCore_DP0 pid=200) INFO 06-29 12:37:31 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.50x"
},
"engine": {
"number": 64,
"text": "(EngineCore_DP0 pid=200) INFO 06-29 12:33:52 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 23.15,
"logged_GPU_KV_tokens": 379184,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-27/result.json",
"retest_strict_tokens": 4508,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33758752768,
"1": 33634103296,
"2": 33684492288,
"3": 33634070528,
"4": 33684193280,
"5": 33633898496,
"6": 33684754432,
"7": 33634291712
},
"retest_sampled_peak_VRAM_GiB": 31.440288543701172,
"retest_memory_observation_count": 601,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_memory_log_source": "historical-recovered/fresh-row-27/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.30 · Clean-Room GGUF Package-Path Reproduction Audit / table 1 | GGUF F16Qwen/Qwen3.6-27B | TP8MTP off | PASS4210 tokens | 70.837 | 66.309 | 11.616 / 39.85710K: 13.530 / 164.349Strict: 14.289 / 74.803Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 22.79 GiB · engine report373,360 grouped KV tokens · 11.32× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-33 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-33 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row33-6830e5d368-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-33/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-33/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-33/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-33/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-33/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-33/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense" python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-33 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-33 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 70.883 · valid historical-model sample10K 66.394 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3742,
"content_chars": 6468,
"content_has_think_close": false,
"elapsed_seconds": 66.08779009897262,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T03:34:15Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.012077721767127514,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 6468,
"qwen_answer_text_sha256": "a3e745fc71d97304f1caba5a41e98d3560a4affeb33e477aa907ba6e5fef5c87",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 6468,
"qwen_visible_text_sha256": "a3e745fc71d97304f1caba5a41e98d3560a4affeb33e477aa907ba6e5fef5c87",
"raw_has_think_close": false,
"reasoning_chars": 12127,
"text_chars": 18596,
"text_sha256": "9a94117e6de620eee1f2549f35ad6e94911474e61cd6dda14a52b9981ae2d03e",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 53.53867237456143,
"vllm_decode_tps": 69.89340291856001,
"vllm_generation_tokens_delta": 3742.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 12.540770625695586,
"vllm_prefill_tps": 34.36790392425299,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 56.62165423289244
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 41.21531672030687,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T03:34:57Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.006134975701570511,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9940,
"text_chars": 9940,
"text_sha256": "f4724c0e635226e325d2779aeb7bb6a3d34abdbcbbbd9d5f951f8b11442e0e74",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 28.215577918104827,
"vllm_decode_tps": 70.88282954206933,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 12.991800364106894,
"vllm_prefill_tps": 33.17477084936939,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 48.525649179704004
},
"10K": {
"completion_tokens": 10000,
"content_chars": 44552,
"content_has_think_close": false,
"elapsed_seconds": 164.95538353640586,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T03:37:42Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008456559851765633,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 44552,
"qwen_answer_text_sha256": "8c459b3f86193f648e1132dd7bf503eda77fc453896ca4d7f7e99d9c81e92cfd",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 44552,
"qwen_visible_text_sha256": "8c459b3f86193f648e1132dd7bf503eda77fc453896ca4d7f7e99d9c81e92cfd",
"raw_has_think_close": false,
"reasoning_chars": 12127,
"text_chars": 56680,
"text_sha256": "1d1b28e8252e3f6248432dbef2a6bc6a6421cef72bf73d7a6f2cfa02605c97fe",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 150.61556784249842,
"vllm_decode_tps": 66.39419910734055,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.329687963239849,
"vllm_prefill_tps": 30.077416975558044,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 60.62245308770409
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 27, 28, 32, 33, 34. One new sample.PASS4,508 natural tokens2K 71.02510K 66.531 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-08T22:18:52.716405342Z",
"FinishedAt": "2026-11-08T22:40:00.260467584Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "9a55da176fe75043aa7dc1ca12883e4f952738b9ebf2616b6f0edce4cb417ed7",
"retest_strict": "PASS",
"retest_strict_tokens": 4508,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 4508,
"decode_tps": 69.61732325093088,
"prefill_seconds": 14.042354583740234,
"client_total_seconds": 78.80580217391253,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 64.75399784836918,
"vllm:request_prefill_time_seconds_sum": 14.042354583740234,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 4508.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
"summary_sha256": "754f0c5eeaf987089a936b03a08af821069b38f7e14fa59e17115de3fc4d2fae"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 71.02517985930231,
"prefill_seconds": 13.268245195038617,
"client_total_seconds": 41.43482113070786,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 28.15902760066092,
"vllm:request_prefill_time_seconds_sum": 13.268245195038617,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
"summary_sha256": "aff3ec02e75684eb8bd8ecdd4936b31fc9fbb3754970d3b0a39d80e61d360495"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 66.53116656725801,
"prefill_seconds": 12.96059922594577,
"client_total_seconds": 163.27578648924828,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 150.3054961450398,
"vllm:request_prefill_time_seconds_sum": 12.96059922594577,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
"summary_sha256": "29705fa055020f3834638960e4af19073f76c5672f7f2d3c621b46c7bf61f6ef"
}
},
"retest_strict_TPS": 69.61732325093088,
"retest_strict_prefill_seconds": 14.042354583740234,
"retest_strict_total_seconds": 78.80580217391253,
"retest_2K_TPS": 71.02517985930231,
"retest_2K_prefill_seconds": 13.268245195038617,
"retest_2K_total_seconds": 41.43482113070786,
"retest_10K_TPS": 66.53116656725801,
"retest_10K_prefill_seconds": 12.96059922594577,
"retest_10K_total_seconds": 163.27578648924828,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33758752768,
"1": 33634103296,
"2": 33684492288,
"3": 33634070528,
"4": 33684193280,
"5": 33633898496,
"6": 33684754432,
"7": 33634291712
},
"retest_sampled_peak_VRAM_GiB": 31.440288543701172,
"retest_memory_observation_count": 601,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_logged_model_load_delta_GiB": 6.75,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_logged_context_capacity_x": 11.32,
"retest_memory_log_source": "historical-recovered/fresh-row-27/result.json"
} | Evidence and limitsstrict-valid; ai-info 10K gate cleared Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 4210,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r2.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-33.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 33,
"original_host": "10.0.0.30",
"logged_model_load_delta_GiB": 6.75,
"model_load_report_rank": 0,
"logged_model_load_seconds": 114.586086,
"logged_available_KV_GiB": 22.79,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 373360,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 11.32,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "37540bd9d6d3a34812dcc6e2d956c59bc7047f62aa91288bafefb2fc8773f312",
"source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/vnext-release-repro/cleanroom-20260629T144538Z/localaiservers/qwen36-gfx906/vnext-launch-runs/gguf-dense27b-tp8",
"source_lines": {
"load": {
"number": 4589,
"text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 14:59:15 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 114.586086 seconds"
},
"KV": {
"number": 5168,
"text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-29 15:02:04 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB"
},
"tokens": {
"number": 5175,
"text": "(EngineCore_DP0 pid=200) INFO 06-29 15:02:04 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens"
},
"capacity": {
"number": 5176,
"text": "(EngineCore_DP0 pid=200) INFO 06-29 15:02:04 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x"
},
"engine": {
"number": 64,
"text": "(EngineCore_DP0 pid=200) INFO 06-29 14:56:55 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 22.79,
"logged_GPU_KV_tokens": 373360,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-27/result.json",
"retest_strict_tokens": 4508,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33758752768,
"1": 33634103296,
"2": 33684492288,
"3": 33634070528,
"4": 33684193280,
"5": 33633898496,
"6": 33684754432,
"7": 33634291712
},
"retest_sampled_peak_VRAM_GiB": 31.440288543701172,
"retest_memory_observation_count": 601,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_memory_log_source": "historical-recovered/fresh-row-27/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.30 · Stand-Alone GGUF Release-Package Audit / table 1 | GGUF F16Qwen/Qwen3.6-27B | TP8MTP off | PASS3715 tokens | 71.129 | 66.530 | 13.592 / 41.71810K: 12.328 / 162.646Strict: 13.505 / 66.534Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 22.79 GiB · engine report373,360 grouped KV tokens · 11.32× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-34 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-34 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row34-15f69466a8-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-34/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-34/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-34/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-34/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-34/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108/serve-row-34/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense" python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-34 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-34 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 70.896 · valid historical-model sample10K 66.395 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 4044,
"content_chars": 7548,
"content_has_think_close": false,
"elapsed_seconds": 72.31937592290342,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T04:11:40Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.011942341923713684,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 7548,
"qwen_answer_text_sha256": "b8dcb06babe50c57ac2d999a7a4366bd9156506cbb59b91ba497f9e84fb74165",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 7548,
"qwen_visible_text_sha256": "b8dcb06babe50c57ac2d999a7a4366bd9156506cbb59b91ba497f9e84fb74165",
"raw_has_think_close": false,
"reasoning_chars": 12371,
"text_chars": 19920,
"text_sha256": "ce311899a1f84ab85c42383d67628ec0138a96d7e3d4f788a18abcc7387ecd9c",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 57.98553304374218,
"vllm_decode_tps": 69.74153358130472,
"vllm_generation_tokens_delta": 4044.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.324300050735474,
"vllm_prefill_tps": 30.08873023278164,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 55.91862413623611
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 41.197517121210694,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T04:12:21Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008826046250760555,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9647,
"text_chars": 9647,
"text_sha256": "1d259b96408168939c15694a69070d10a6bea9eaa997e3817f3dba8b5d0a375b",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 28.21035666577518,
"vllm_decode_tps": 70.89594873596197,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 12.978673229925334,
"vllm_prefill_tps": 33.20832510107657,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 48.546614935934876
},
"10K": {
"completion_tokens": 10000,
"content_chars": 39469,
"content_has_think_close": false,
"elapsed_seconds": 165.68056911695749,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T04:15:07Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008547734469175339,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 39469,
"qwen_answer_text_sha256": "8cf441537e4d89e9f0d90733dad51c162387787efbc9ee5fa4d325d5f4a62c34",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 39469,
"qwen_visible_text_sha256": "8cf441537e4d89e9f0d90733dad51c162387787efbc9ee5fa4d325d5f4a62c34",
"raw_has_think_close": false,
"reasoning_chars": 12371,
"text_chars": 51841,
"text_sha256": "d76a00c8f065f38b2b3551bad9418362baf93c16fed14699be81777846b1f433",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 150.61303294822574,
"vllm_decode_tps": 66.3953165556235,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.058221222832799,
"vllm_prefill_tps": 28.62223855142161,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 60.35710797770609
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 27, 28, 32, 33, 34. One new sample.PASS4,508 natural tokens2K 71.02510K 66.531 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-08T22:18:52.716405342Z",
"FinishedAt": "2026-11-08T22:40:00.260467584Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "9a55da176fe75043aa7dc1ca12883e4f952738b9ebf2616b6f0edce4cb417ed7",
"retest_strict": "PASS",
"retest_strict_tokens": 4508,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 4508,
"decode_tps": 69.61732325093088,
"prefill_seconds": 14.042354583740234,
"client_total_seconds": 78.80580217391253,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 64.75399784836918,
"vllm:request_prefill_time_seconds_sum": 14.042354583740234,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 4508.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
"summary_sha256": "754f0c5eeaf987089a936b03a08af821069b38f7e14fa59e17115de3fc4d2fae"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 71.02517985930231,
"prefill_seconds": 13.268245195038617,
"client_total_seconds": 41.43482113070786,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 28.15902760066092,
"vllm:request_prefill_time_seconds_sum": 13.268245195038617,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2000.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
"summary_sha256": "aff3ec02e75684eb8bd8ecdd4936b31fc9fbb3754970d3b0a39d80e61d360495"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 66.53116656725801,
"prefill_seconds": 12.96059922594577,
"client_total_seconds": 163.27578648924828,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_decode_time_seconds_sum": 150.3054961450398,
"vllm:request_prefill_time_seconds_sum": 12.96059922594577,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 10000.0,
"vllm:request_prefill_time_seconds_count": 1.0
},
"request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
"summary_sha256": "29705fa055020f3834638960e4af19073f76c5672f7f2d3c621b46c7bf61f6ef"
}
},
"retest_strict_TPS": 69.61732325093088,
"retest_strict_prefill_seconds": 14.042354583740234,
"retest_strict_total_seconds": 78.80580217391253,
"retest_2K_TPS": 71.02517985930231,
"retest_2K_prefill_seconds": 13.268245195038617,
"retest_2K_total_seconds": 41.43482113070786,
"retest_10K_TPS": 66.53116656725801,
"retest_10K_prefill_seconds": 12.96059922594577,
"retest_10K_total_seconds": 163.27578648924828,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33758752768,
"1": 33634103296,
"2": 33684492288,
"3": 33634070528,
"4": 33684193280,
"5": 33633898496,
"6": 33684754432,
"7": 33634291712
},
"retest_sampled_peak_VRAM_GiB": 31.440288543701172,
"retest_memory_observation_count": 601,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_logged_model_load_delta_GiB": 6.75,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_logged_context_capacity_x": 11.32,
"retest_memory_log_source": "historical-recovered/fresh-row-27/result.json"
} | Evidence and limitsstrict-valid; ai-info 10K gate cleared Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 3715,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r2.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-34.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 34,
"original_host": "10.0.0.30",
"logged_model_load_delta_GiB": 6.75,
"model_load_report_rank": 0,
"logged_model_load_seconds": 112.330588,
"logged_available_KV_GiB": 22.79,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 373360,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 11.32,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "440fc7e6d449bf85ebc64565dfb0544de1e59b96415d3a09f31df4c5871a8a7c",
"source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/vnext-standalone-audit/20260629T171707Z/localaiservers/vnext-launch-runs/gguf-dense27b-tp8",
"source_lines": {
"load": {
"number": 4589,
"text": "(Worker pid=251) (Worker_TP0 pid=251) INFO 06-29 17:28:29 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 112.330588 seconds"
},
"KV": {
"number": 5169,
"text": "(Worker pid=251) (Worker_TP0 pid=251) INFO 06-29 17:31:23 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB"
},
"tokens": {
"number": 5175,
"text": "(EngineCore_DP0 pid=224) INFO 06-29 17:31:23 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens"
},
"capacity": {
"number": 5176,
"text": "(EngineCore_DP0 pid=224) INFO 06-29 17:31:23 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x"
},
"engine": {
"number": 64,
"text": "(EngineCore_DP0 pid=224) INFO 06-29 17:26:12 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 22.79,
"logged_GPU_KV_tokens": 373360,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-27/result.json",
"retest_strict_tokens": 4508,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33758752768,
"1": 33634103296,
"2": 33684492288,
"3": 33634070528,
"4": 33684193280,
"5": 33633898496,
"6": 33684754432,
"7": 33634291712
},
"retest_sampled_peak_VRAM_GiB": 31.440288543701172,
"retest_memory_observation_count": 601,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-08T22:22:06.240435797Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:22:06 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 99.197417 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-08T22:25:00.830568798Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-08 22:25:00 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-08T22:25:00.891573213Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-08T22:25:00.891705873Z (EngineCore_DP0 pid=200) INFO 11-08 22:25:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_memory_log_source": "historical-recovered/fresh-row-27/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.30 · Public-Staged HF Reproduction Audit / table 1 | HF FP16Qwen/Qwen3.6-27B | TP8MTP off | PASS4884 tokens | 71.346 | 66.729 | 14.065 / 42.10610K: 13.218 / 163.087Strict: 13.993 / 84.091Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.84 GiB (rank 0)Available KV: 22.64 GiB · engine report370,864 grouped KV tokens · 11.25× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-35 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-35 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row35-99603ed262-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-35/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-35/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-35/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-35/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-35/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-35/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/opt/local-models/qwen36-27b-hf,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-35 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-35 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 71.295 · valid historical-model sample10K 66.800 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3718,
"content_chars": 5402,
"content_has_think_close": false,
"elapsed_seconds": 68.07331505976617,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T04:41:48Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.0126217445358634,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 5402,
"qwen_answer_text_sha256": "44119bdba6f8fe6fa4fec4357ee8a88c27eda314f12159987fe0552df4205ef4",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 5402,
"qwen_visible_text_sha256": "44119bdba6f8fe6fa4fec4357ee8a88c27eda314f12159987fe0552df4205ef4",
"raw_has_think_close": false,
"reasoning_chars": 12461,
"text_chars": 17864,
"text_sha256": "9399dfc9caf27d0063a4ea1cfe02070c9d0382ef098359845c7cc888a3123158",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 52.871985126286745,
"vllm_decode_tps": 70.32079448349472,
"vllm_generation_tokens_delta": 3718.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.192512446083128,
"vllm_prefill_tps": 28.36923790779047,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 54.61758394953611
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 42.9475625436753,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T04:42:31Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.006040042266249657,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9610,
"text_chars": 9610,
"text_sha256": "b6e619acbecd7b98e7bd3fafcd526951e3260b60cf5909be4ad3a5f325565bff",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 28.05252104625106,
"vllm_decode_tps": 71.29484001464746,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.887143962085247,
"vllm_prefill_tps": 28.95115417018038,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 46.56841696117469
},
"10K": {
"completion_tokens": 10000,
"content_chars": 41015,
"content_has_think_close": false,
"elapsed_seconds": 164.69132177624851,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T04:45:16Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008324465714395046,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 41015,
"qwen_answer_text_sha256": "ceaec15a92ffffd97f2dbd1c9884a52f22187f067808c2022d82e8ed942b9d31",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 41015,
"qwen_visible_text_sha256": "ceaec15a92ffffd97f2dbd1c9884a52f22187f067808c2022d82e8ed942b9d31",
"raw_has_think_close": false,
"reasoning_chars": 12461,
"text_chars": 53477,
"text_sha256": "b1abee2192f7316f13106288d0a3b7898999e3bfa2fdf13496f9212608eb5782",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 149.7005802281201,
"vllm_decode_tps": 66.80000828828837,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.980803821235895,
"vllm_prefill_tps": 28.77015179846625,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 60.719653544259685
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 35, 38, 41, 44. One new sample.PASS3,609 natural tokens2K 70.95110K 66.469 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-08T23:40:06.929336115Z",
"FinishedAt": "2026-11-08T23:58:45.910937408Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "a96218fee0c78b5145db76c43cdf273ff01d169be52e0662ddcc70106f936f79",
"retest_strict": "PASS",
"retest_strict_tokens": 3609,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 3609,
"decode_tps": 70.06009773846488,
"prefill_seconds": 12.782165881246328,
"client_total_seconds": 64.30384199228138,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 3609.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_sum": 12.782165881246328,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 51.512917002663016,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "ee97c35547770515d3bd63d45b2557687a8278bbeb38ce72abbaad2fee0ab250",
"summary_sha256": "6ae03f5709d05f854c0ff4b3e846b625d2d4a691aac551f35ae21d6a4b36c17e"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 70.95081153473755,
"prefill_seconds": 15.072314956225455,
"client_total_seconds": 43.269196966663,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 2000.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_sum": 15.072314956225455,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 28.188542974181473,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "d476546705501abd0468f2fc6527e9ace49f732079e53b75abf7bcfb98635f44",
"summary_sha256": "f70df1fbfc18c6af1f765a73663863bb438fbe1dd644049da3c9e283ae692d0e"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 66.46936099788869,
"prefill_seconds": 14.515854343771935,
"client_total_seconds": 164.97092120815068,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 10000.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_sum": 14.515854343771935,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 150.4452555263415,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "ee5df402413359190aaa7ed74f4a298f749233768156b8fc14f2ea4bef0e8017",
"summary_sha256": "1dfc4acf114048076b24fd3ed43f4a13dd3800ec8892ff4352974a55fd56aba2"
}
},
"retest_strict_TPS": 70.06009773846488,
"retest_strict_prefill_seconds": 12.782165881246328,
"retest_strict_total_seconds": 64.30384199228138,
"retest_2K_TPS": 70.95081153473755,
"retest_2K_prefill_seconds": 15.072314956225455,
"retest_2K_total_seconds": 43.269196966663,
"retest_10K_TPS": 66.46936099788869,
"retest_10K_prefill_seconds": 14.515854343771935,
"retest_10K_total_seconds": 164.97092120815068,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33776762880,
"1": 33666486272,
"2": 33716842496,
"3": 33666523136,
"4": 33716817920,
"5": 33666486272,
"6": 33716846592,
"7": 33666510848
},
"retest_sampled_peak_VRAM_GiB": 31.457061767578125,
"retest_memory_observation_count": 527,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 712,
"text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
"values": [
"6.84"
]
}
],
"available_KV_GiB": [
{
"line": 1267,
"text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
"values": [
"22.64"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 1273,
"text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
"values": [
"370,864"
]
}
],
"context_capacity": [
{
"line": 1274,
"text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
"values": [
"131,072",
"11.25"
]
}
]
},
"retest_logged_model_load_delta_GiB": 6.84,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 22.64,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 370864,
"retest_logged_context_capacity_x": 11.25,
"retest_memory_log_source": "historical-recovered/fresh-row-35/result.json"
} | Evidence and limitsstrict-valid; ai-info 10K gate cleared Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 4884,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r3.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-35.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 35,
"original_host": "10.0.0.30",
"logged_model_load_delta_GiB": 6.84,
"model_load_report_rank": 0,
"logged_model_load_seconds": 22.097343,
"logged_available_KV_GiB": 22.64,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 370864,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 11.25,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "102a7ff5f97bc9e718827f1860435d52aba59065ead3318be73d7a2872aef11a",
"source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/vnext-release-repro/localaiservers/qwen36-gfx906/vnext-launch-runs/repro-hf-dense27b-tp8-public",
"source_lines": {
"load": {
"number": 712,
"text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-29 13:41:17 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.097343 seconds"
},
"KV": {
"number": 1271,
"text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-29 13:42:58 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB"
},
"tokens": {
"number": 1273,
"text": "(EngineCore_DP0 pid=184) INFO 06-29 13:42:58 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens"
},
"capacity": {
"number": 1274,
"text": "(EngineCore_DP0 pid=184) INFO 06-29 13:42:58 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x"
},
"engine": {
"number": 55,
"text": "(EngineCore_DP0 pid=184) INFO 06-29 13:40:27 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-27b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 22.64,
"logged_GPU_KV_tokens": 370864,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-35/result.json",
"retest_strict_tokens": 3609,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33776762880,
"1": 33666486272,
"2": 33716842496,
"3": 33666523136,
"4": 33716817920,
"5": 33666486272,
"6": 33716846592,
"7": 33666510848
},
"retest_sampled_peak_VRAM_GiB": 31.457061767578125,
"retest_memory_observation_count": 527,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 712,
"text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
"values": [
"6.84"
]
}
],
"available_KV_GiB": [
{
"line": 1267,
"text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
"values": [
"22.64"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 1273,
"text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
"values": [
"370,864"
]
}
],
"context_capacity": [
{
"line": 1274,
"text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
"values": [
"131,072",
"11.25"
]
}
]
},
"retest_KV_budget_GiB": 22.64,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 370864,
"retest_memory_log_source": "historical-recovered/fresh-row-35/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.30 · Public-Staged HF Reproduction Audit / table 1 | HF FP16Qwen/Qwen3.6-35B-A3B | TP8MTP off | PASS2975 tokens | 116.258 | 109.242 | 0.817 / 18.03010K: 0.818 / 92.369Strict: 0.808 / 26.577Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 8.88 GiB (rank 0)Available KV: 19.91 GiB · engine report521,696 grouped KV tokens · 15.82× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-36 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-36 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row36-f3436e8210-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-36/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-36/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-36/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-36/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-36/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-36/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-moe,dst=/opt/local-models/qwen36-35b-a3b-hf,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-moe "$MODEL_DIR/qwen36-moe" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-36 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-36 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 116.158 · valid historical-model sample10K 109.274 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3867,
"content_chars": 7448,
"content_has_think_close": false,
"elapsed_seconds": 34.548473133705556,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T05:24:29Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.010292066261172295,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 7448,
"qwen_answer_text_sha256": "df3adaa80cdc4fb582fc7654e356001d375fc9f61acfe39fbbc3a45bbf20995e",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 7448,
"qwen_visible_text_sha256": "df3adaa80cdc4fb582fc7654e356001d375fc9f61acfe39fbbc3a45bbf20995e",
"raw_has_think_close": false,
"reasoning_chars": 12244,
"text_chars": 19693,
"text_sha256": "419dea0c3413266f4ff53d4a698d31273de2d306b49681fb788af33368af5c96",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 33.72320320736617,
"vllm_decode_tps": 114.66882241943523,
"vllm_generation_tokens_delta": 3867.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8162200124934316,
"vllm_prefill_tps": 528.0439016477416,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 111.9296932467718
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 18.036245052702725,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T05:24:47Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008652694523334503,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 10030,
"text_chars": 10030,
"text_sha256": "0ab784d0ec5e96d9efb3934f80cfd8b0cf2b3536f46f7c75264e5114bda0c04e",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 17.217981002293527,
"vllm_decode_tps": 116.15763774704998,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8099861172959208,
"vllm_prefill_tps": 532.1078853040863,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 110.88782582826467
},
"10K": {
"completion_tokens": 10000,
"content_chars": 43387,
"content_has_think_close": false,
"elapsed_seconds": 92.34119913075119,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T05:26:20Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.00704532116651535,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 43387,
"qwen_answer_text_sha256": "d35733135a89d8effc0a7c67690e9d4414d78fd965a979b0150343275ef63dbf",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 43387,
"qwen_visible_text_sha256": "d35733135a89d8effc0a7c67690e9d4414d78fd965a979b0150343275ef63dbf",
"raw_has_think_close": false,
"reasoning_chars": 12244,
"text_chars": 55632,
"text_sha256": "af45e17100f84637c61eadb247d364e6be4ccd3fde6062efc0b81cb3cceddba6",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 91.51303980406374,
"vllm_decode_tps": 109.27404467615487,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8177437912672758,
"vllm_prefill_tps": 527.0599478744678,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 108.29402362254824
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 36, 39, 42, 45. One new sample.PASS2,911 natural tokens2K 117.36710K 110.169 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-09T00:00:14.441833328Z",
"FinishedAt": "2026-11-09T00:20:09.78005595Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "d3e8150b92459eb9bfeb0628cfbc7f1ba14417d265985bd295006c8f223de819",
"retest_strict": "PASS",
"retest_strict_tokens": 2911,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 2911,
"decode_tps": 116.74599429536063,
"prefill_seconds": 0.8083759220317006,
"client_total_seconds": 25.750953457318246,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 0.8083759220317006,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 24.93447434809059,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2911.0
},
"request_sha256": "e31c712c0771301f2e4d592bc966448d3143f2660e1b4cab33a91704f19e9370",
"summary_sha256": "eebfaae9c73243c913cb60f1c6f4ec103cac551b4f8d5c5529c5364a6f3465ca"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 117.36665638338557,
"prefill_seconds": 0.8091103108599782,
"client_total_seconds": 17.857541773468256,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 0.8091103108599782,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 17.04061495512724,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2000.0
},
"request_sha256": "bca01af4a8338cd6274c1c9e0b281d8e59066ebae2ece3e3a636620066da3a03",
"summary_sha256": "2c37396cb2c1b23f8ae028e9beb941f96a022ffb8a5da29d3363d63003b69672"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 110.16866681199451,
"prefill_seconds": 0.8098680507391691,
"client_total_seconds": 91.58954442292452,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 0.8098680507391691,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 90.76991026010364,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 10000.0
},
"request_sha256": "46fd7808353544eb86016bdd1746bafb67def6864a3f5352647b66cb9149f95a",
"summary_sha256": "b44fdcf5c45447a4a3ba6eda737cc6adcf2f183f3569935f415e850a380531e5"
}
},
"retest_strict_TPS": 116.74599429536063,
"retest_strict_prefill_seconds": 0.8083759220317006,
"retest_strict_total_seconds": 25.750953457318246,
"retest_2K_TPS": 117.36665638338557,
"retest_2K_prefill_seconds": 0.8091103108599782,
"retest_2K_total_seconds": 17.857541773468256,
"retest_10K_TPS": 110.16866681199451,
"retest_10K_prefill_seconds": 0.8098680507391691,
"retest_10K_total_seconds": 91.58954442292452,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 34334547968,
"1": 34314465280,
"2": 34314543104,
"3": 34314469376,
"4": 34334240768,
"5": 34314461184,
"6": 34314457088,
"7": 34318581760
},
"retest_sampled_peak_VRAM_GiB": 31.976539611816406,
"retest_memory_observation_count": 565,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 193,
"text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
"values": [
"8.88"
]
}
],
"available_KV_GiB": [
{
"line": 313,
"text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
"values": [
"19.91"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 315,
"text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
"values": [
"521,696"
]
}
],
"context_capacity": [
{
"line": 316,
"text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
"values": [
"131,072",
"15.82"
]
}
]
},
"retest_logged_model_load_delta_GiB": 8.88,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 19.91,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 521696,
"retest_logged_context_capacity_x": 15.82,
"retest_memory_log_source": "historical-recovered/fresh-row-36/result.json"
} | Evidence and limitsstrict-valid Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 2975,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "995ad96eacd98c81ed38be0c5b274b04031597b0",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r3.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-36.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 36,
"original_host": "10.0.0.30",
"logged_model_load_delta_GiB": 8.88,
"model_load_report_rank": 0,
"logged_model_load_seconds": 108.136458,
"logged_available_KV_GiB": 19.91,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 521696,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 15.82,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "4eb4f84cc3c3a759f1a904f06f6727f7d5a570c87d16ff334c5f48045c2ef969",
"source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/vnext-release-repro/localaiservers/qwen36-gfx906/vnext-launch-runs/repro-hf-moe35b-tp8-public",
"source_lines": {
"load": {
"number": 193,
"text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-29 14:15:32 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 108.136458 seconds"
},
"KV": {
"number": 313,
"text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-29 14:17:00 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB"
},
"tokens": {
"number": 315,
"text": "(EngineCore_DP0 pid=171) INFO 06-29 14:17:00 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens"
},
"capacity": {
"number": 316,
"text": "(EngineCore_DP0 pid=171) INFO 06-29 14:17:00 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x"
},
"engine": {
"number": 33,
"text": "(EngineCore_DP0 pid=171) INFO 06-29 14:13:13 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-35b-a3b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-35b-a3b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.6-35B-A3B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 19.91,
"logged_GPU_KV_tokens": 521696,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-36/result.json",
"retest_strict_tokens": 2911,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 34334547968,
"1": 34314465280,
"2": 34314543104,
"3": 34314469376,
"4": 34334240768,
"5": 34314461184,
"6": 34314457088,
"7": 34318581760
},
"retest_sampled_peak_VRAM_GiB": 31.976539611816406,
"retest_memory_observation_count": 565,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 193,
"text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
"values": [
"8.88"
]
}
],
"available_KV_GiB": [
{
"line": 313,
"text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
"values": [
"19.91"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 315,
"text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
"values": [
"521,696"
]
}
],
"context_capacity": [
{
"line": 316,
"text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
"values": [
"131,072",
"15.82"
]
}
]
},
"retest_KV_budget_GiB": 19.91,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 521696,
"retest_memory_log_source": "historical-recovered/fresh-row-36/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.20 · Full vNext Release-Readiness Serving Replay / table 1 | GGUF F16Qwen/Qwen3.6-27B | TP8MTP off | PASS3701 tokens | 70.881 | 66.446 | 13.934 / 42.15910K: 14.571 / 165.078Strict: 14.468 / 67.418Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 22.79 GiB · engine report373,360 grouped KV tokens · 11.32× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-37 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-37 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row37-2c5900fca7-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-37/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-37/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-37/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-37/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-37/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-37/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense" python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-37 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-37 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 70.841 · valid historical-model sample10K 66.373 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3878,
"content_chars": 6406,
"content_has_think_close": false,
"elapsed_seconds": 69.53517793025821,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T06:02:00Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.009362303651869297,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 6406,
"qwen_answer_text_sha256": "d01a470e93c380e6a60db3467c7434ffff73248597d6ea2708468e7141a34d17",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 6406,
"qwen_visible_text_sha256": "d01a470e93c380e6a60db3467c7434ffff73248597d6ea2708468e7141a34d17",
"raw_has_think_close": false,
"reasoning_chars": 12683,
"text_chars": 19090,
"text_sha256": "2d71822f464cf3ae4264986cd1e5f82bfc75faa6829d04e192e6eafead517f2a",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 55.5404448620975,
"vllm_decode_tps": 69.82299132872927,
"vllm_generation_tokens_delta": 3878.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 13.984859772957861,
"vllm_prefill_tps": 30.819043379570587,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 55.77033259179293
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 42.994658844545484,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T06:02:43Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008095500990748405,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9905,
"text_chars": 9905,
"text_sha256": "5ab7bffe8c721a17390b8d131cd339b67b68be1e9b06f97e8ee51276791519bb",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 28.23240473214537,
"vllm_decode_tps": 70.84058262039589,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.754096891731024,
"vllm_prefill_tps": 29.212225130604583,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 46.51740596968895
},
"10K": {
"completion_tokens": 10000,
"content_chars": 40852,
"content_has_think_close": false,
"elapsed_seconds": 165.7533283000812,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T06:05:29Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.00877947174012661,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 40852,
"qwen_answer_text_sha256": "10f12fb0806376aa6c74232eb8a7892839336925fbd86df0e95736cbf71d0808",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 40852,
"qwen_visible_text_sha256": "10f12fb0806376aa6c74232eb8a7892839336925fbd86df0e95736cbf71d0808",
"raw_has_think_close": false,
"reasoning_chars": 12683,
"text_chars": 53536,
"text_sha256": "d198d0c8b309b50972d4e85140d9d7b39c69b4141296698daa7c5fcb0112302d",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 150.66444395296276,
"vllm_decode_tps": 66.37266058023609,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.079772244207561,
"vllm_prefill_tps": 28.581333525481835,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 60.33061358439764
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 37, 40, 43. One new sample.PASS4,263 natural tokens2K 70.87610K 66.474 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-09T00:22:27.410864084Z",
"FinishedAt": "2026-11-09T00:43:40.972906165Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "ae0dba47e379e10ae120c82dbe767b915856ebc0a56771c0751407ceb9b17f98",
"retest_strict": "PASS",
"retest_strict_tokens": 4263,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 4263,
"decode_tps": 69.65298485444228,
"prefill_seconds": 13.264510760083795,
"client_total_seconds": 74.47932942770422,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 13.264510760083795,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 61.20340727549046,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 4263.0
},
"request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
"summary_sha256": "2a4ec12838f2054993cfc40a0ce46452682e1e06ac8c785eefd538bcc5266ae6"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 70.8759028403918,
"prefill_seconds": 15.0672946870327,
"client_total_seconds": 43.29480273462832,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 15.0672946870327,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 28.218335426412523,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2000.0
},
"request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
"summary_sha256": "304d5e8b0f2afd6fcf019e0b454f8917c037eece212906cfcf2c1686136c1ee7"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 66.47442948531119,
"prefill_seconds": 14.876342931762338,
"client_total_seconds": 165.31903470959514,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 14.876342931762338,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 150.43378450069577,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 10000.0
},
"request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
"summary_sha256": "895a7e44c83b5069acf38a1a85397205ac0d8a1cefc38010e259fa29cda426f9"
}
},
"retest_strict_TPS": 69.65298485444228,
"retest_strict_prefill_seconds": 13.264510760083795,
"retest_strict_total_seconds": 74.47932942770422,
"retest_2K_TPS": 70.8759028403918,
"retest_2K_prefill_seconds": 15.0672946870327,
"retest_2K_total_seconds": 43.29480273462832,
"retest_10K_TPS": 66.47442948531119,
"retest_10K_prefill_seconds": 14.876342931762338,
"retest_10K_total_seconds": 165.31903470959514,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33744068608,
"1": 33634144256,
"2": 33684496384,
"3": 33634095104,
"4": 33684230144,
"5": 33633964032,
"6": 33684738048,
"7": 33634250752
},
"retest_sampled_peak_VRAM_GiB": 31.426612854003906,
"retest_memory_observation_count": 604,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-09T00:25:47.689313425Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:25:47 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 105.144041 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-09T00:28:42.161027038Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:28:42 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-09T00:28:42.250244890Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-09T00:28:42.250362411Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_logged_model_load_delta_GiB": 6.75,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_logged_context_capacity_x": 11.32,
"retest_memory_log_source": "historical-recovered/fresh-row-37/result.json"
} | Evidence and limitsstrict-valid; ai-info 10K gate cleared Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 3701,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r3.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-37.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 37,
"original_host": "10.0.0.20",
"logged_model_load_delta_GiB": 6.75,
"model_load_report_rank": 0,
"logged_model_load_seconds": 106.384137,
"logged_available_KV_GiB": 22.79,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 373360,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 11.32,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "8db635ce1dee530b6536026905839b582cf7335807fb11229461c2333087d67b",
"source_archive": "historical-recovered/host10.0.0.20-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/vnext-repeat-20260630T045153Z/runtime/vnext-launch-runs/20260630T050006Z-2928686/gguf-dense27b-tp8",
"source_lines": {
"load": {
"number": 4596,
"text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-30 05:05:39 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 106.384137 seconds"
},
"KV": {
"number": 5170,
"text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-30 05:08:26 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB"
},
"tokens": {
"number": 5174,
"text": "(EngineCore_DP0 pid=200) INFO 06-30 05:08:26 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens"
},
"capacity": {
"number": 5175,
"text": "(EngineCore_DP0 pid=200) INFO 06-30 05:08:26 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x"
},
"engine": {
"number": 63,
"text": "(EngineCore_DP0 pid=200) INFO 06-30 05:03:28 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=6a9e13bd6fc8f0983b9b99948120bc37f49c13e9, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 22.79,
"logged_GPU_KV_tokens": 373360,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-37/result.json",
"retest_strict_tokens": 4263,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33744068608,
"1": 33634144256,
"2": 33684496384,
"3": 33634095104,
"4": 33684230144,
"5": 33633964032,
"6": 33684738048,
"7": 33634250752
},
"retest_sampled_peak_VRAM_GiB": 31.426612854003906,
"retest_memory_observation_count": 604,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-09T00:25:47.689313425Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:25:47 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 105.144041 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-09T00:28:42.161027038Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:28:42 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-09T00:28:42.250244890Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-09T00:28:42.250362411Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_memory_log_source": "historical-recovered/fresh-row-37/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.20 · Full vNext Release-Readiness Serving Replay / table 1 | HF FP16Qwen/Qwen3.6-27B | TP8MTP off | PASS4237 tokens | 71.395 | 66.876 | 14.888 / 42.91010K: 14.015 / 163.554Strict: 13.858 / 74.265Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.84 GiB (rank 0)Available KV: 22.64 GiB · engine report370,864 grouped KV tokens · 11.25× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-38 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-38 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row38-f33ad375e1-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-38/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-38/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-38/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-38/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-38/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-38/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/opt/local-models/qwen36-27b-hf,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-38 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-38 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 71.286 · valid historical-model sample10K 66.781 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3588,
"content_chars": 5883,
"content_has_think_close": false,
"elapsed_seconds": 64.89045448042452,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T06:32:14Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.01263996958732605,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 5883,
"qwen_answer_text_sha256": "3d0e920f50ef42aedafc11bbfb924b76222895f37114bb681663302f9b91f75c",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 5883,
"qwen_visible_text_sha256": "3d0e920f50ef42aedafc11bbfb924b76222895f37114bb681663302f9b91f75c",
"raw_has_think_close": false,
"reasoning_chars": 11124,
"text_chars": 17008,
"text_sha256": "fc08f3a103846a02d7b39738392e17893d685b555aca4052a1d533e1812c9962",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 50.96286449301988,
"vllm_decode_tps": 70.40420580148962,
"vllm_generation_tokens_delta": 3588.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 13.919675651937723,
"vllm_prefill_tps": 30.96336515140003,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 55.29318647448201
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 42.615525607950985,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T06:32:56Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008263365365564823,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9427,
"text_chars": 9427,
"text_sha256": "54cbb97b2b6cddbb147bbe1694b9bcc497925aeaf06256baf5c70d59ca44ec82",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 28.056015537120402,
"vllm_decode_tps": 71.28595995229033,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.55124822538346,
"vllm_prefill_tps": 29.61945211326653,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 46.93125267066635
},
"10K": {
"completion_tokens": 10000,
"content_chars": 38346,
"content_has_think_close": false,
"elapsed_seconds": 163.270943348296,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T06:35:40Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008678438141942024,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 38346,
"qwen_answer_text_sha256": "518a9ba4dd4d45e51d323b328845c047caa5881d47e582a24689ee5960efb863",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 38346,
"qwen_visible_text_sha256": "518a9ba4dd4d45e51d323b328845c047caa5881d47e582a24689ee5960efb863",
"raw_has_think_close": false,
"reasoning_chars": 11124,
"text_chars": 49471,
"text_sha256": "8c34f28f2fca20d1d2be75fb967201d2860e51eb043cda3b63d8047c14c5db74",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 149.74291642103344,
"vllm_decode_tps": 66.78112219934941,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 13.51911969203502,
"vllm_prefill_tps": 31.880774031014,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 61.247885232509546
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 35, 38, 41, 44. One new sample.PASS3,609 natural tokens2K 70.95110K 66.469 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-08T23:40:06.929336115Z",
"FinishedAt": "2026-11-08T23:58:45.910937408Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "a96218fee0c78b5145db76c43cdf273ff01d169be52e0662ddcc70106f936f79",
"retest_strict": "PASS",
"retest_strict_tokens": 3609,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 3609,
"decode_tps": 70.06009773846488,
"prefill_seconds": 12.782165881246328,
"client_total_seconds": 64.30384199228138,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 3609.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_sum": 12.782165881246328,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 51.512917002663016,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "ee97c35547770515d3bd63d45b2557687a8278bbeb38ce72abbaad2fee0ab250",
"summary_sha256": "6ae03f5709d05f854c0ff4b3e846b625d2d4a691aac551f35ae21d6a4b36c17e"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 70.95081153473755,
"prefill_seconds": 15.072314956225455,
"client_total_seconds": 43.269196966663,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 2000.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_sum": 15.072314956225455,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 28.188542974181473,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "d476546705501abd0468f2fc6527e9ace49f732079e53b75abf7bcfb98635f44",
"summary_sha256": "f70df1fbfc18c6af1f765a73663863bb438fbe1dd644049da3c9e283ae692d0e"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 66.46936099788869,
"prefill_seconds": 14.515854343771935,
"client_total_seconds": 164.97092120815068,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 10000.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_sum": 14.515854343771935,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 150.4452555263415,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "ee5df402413359190aaa7ed74f4a298f749233768156b8fc14f2ea4bef0e8017",
"summary_sha256": "1dfc4acf114048076b24fd3ed43f4a13dd3800ec8892ff4352974a55fd56aba2"
}
},
"retest_strict_TPS": 70.06009773846488,
"retest_strict_prefill_seconds": 12.782165881246328,
"retest_strict_total_seconds": 64.30384199228138,
"retest_2K_TPS": 70.95081153473755,
"retest_2K_prefill_seconds": 15.072314956225455,
"retest_2K_total_seconds": 43.269196966663,
"retest_10K_TPS": 66.46936099788869,
"retest_10K_prefill_seconds": 14.515854343771935,
"retest_10K_total_seconds": 164.97092120815068,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33776762880,
"1": 33666486272,
"2": 33716842496,
"3": 33666523136,
"4": 33716817920,
"5": 33666486272,
"6": 33716846592,
"7": 33666510848
},
"retest_sampled_peak_VRAM_GiB": 31.457061767578125,
"retest_memory_observation_count": 527,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 712,
"text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
"values": [
"6.84"
]
}
],
"available_KV_GiB": [
{
"line": 1267,
"text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
"values": [
"22.64"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 1273,
"text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
"values": [
"370,864"
]
}
],
"context_capacity": [
{
"line": 1274,
"text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
"values": [
"131,072",
"11.25"
]
}
]
},
"retest_logged_model_load_delta_GiB": 6.84,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 22.64,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 370864,
"retest_logged_context_capacity_x": 11.25,
"retest_memory_log_source": "historical-recovered/fresh-row-35/result.json"
} | Evidence and limitsstrict-valid; ai-info 10K gate cleared Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 4237,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r3.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-38.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 38,
"original_host": "10.0.0.20",
"logged_model_load_delta_GiB": 6.84,
"model_load_report_rank": 0,
"logged_model_load_seconds": 23.229114,
"logged_available_KV_GiB": 22.64,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 370864,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 11.25,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "9e63d9b84eed3eab33372a7e7d9ccd320c5f0004a5d35eecda83a3d621f402db",
"source_archive": "historical-recovered/host10.0.0.20-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/vnext-repeat-20260630T045153Z/runtime/vnext-launch-runs/20260630T050006Z-2928686/hf-dense27b-tp8",
"source_lines": {
"load": {
"number": 712,
"text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-30 05:39:43 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 23.229114 seconds"
},
"KV": {
"number": 1271,
"text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-30 05:41:25 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB"
},
"tokens": {
"number": 1273,
"text": "(EngineCore_DP0 pid=184) INFO 06-30 05:41:25 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens"
},
"capacity": {
"number": 1274,
"text": "(EngineCore_DP0 pid=184) INFO 06-30 05:41:25 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x"
},
"engine": {
"number": 55,
"text": "(EngineCore_DP0 pid=184) INFO 06-30 05:38:53 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-27b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 22.64,
"logged_GPU_KV_tokens": 370864,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-35/result.json",
"retest_strict_tokens": 3609,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33776762880,
"1": 33666486272,
"2": 33716842496,
"3": 33666523136,
"4": 33716817920,
"5": 33666486272,
"6": 33716846592,
"7": 33666510848
},
"retest_sampled_peak_VRAM_GiB": 31.457061767578125,
"retest_memory_observation_count": 527,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 712,
"text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
"values": [
"6.84"
]
}
],
"available_KV_GiB": [
{
"line": 1267,
"text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
"values": [
"22.64"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 1273,
"text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
"values": [
"370,864"
]
}
],
"context_capacity": [
{
"line": 1274,
"text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
"values": [
"131,072",
"11.25"
]
}
]
},
"retest_KV_budget_GiB": 22.64,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 370864,
"retest_memory_log_source": "historical-recovered/fresh-row-35/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.20 · Full vNext Release-Readiness Serving Replay / table 1 | HF FP16Qwen/Qwen3.6-35B-A3B | TP8MTP off | PASS3310 tokens | 116.542 | 109.672 | 0.810 / 17.98010K: 0.810 / 92.001Strict: 0.809 / 29.432Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 8.88 GiB (rank 0)Available KV: 19.91 GiB · engine report521,696 grouped KV tokens · 15.82× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-39 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-39 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row39-a344df5fa1-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-39/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-39/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-39/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-39/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-39/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-39/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-moe,dst=/opt/local-models/qwen36-35b-a3b-hf,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-moe "$MODEL_DIR/qwen36-moe" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-39 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-39 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 117.028 · valid historical-model sample10K 110.021 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3268,
"content_chars": 5228,
"content_has_think_close": false,
"elapsed_seconds": 28.94538085255772,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T07:12:14Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.008697621524333954,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 5228,
"qwen_answer_text_sha256": "facfcedef1b47165f2c0333e2266078fe5937046756c294e8c636f14b2e55856",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 5228,
"qwen_visible_text_sha256": "facfcedef1b47165f2c0333e2266078fe5937046756c294e8c636f14b2e55856",
"raw_has_think_close": false,
"reasoning_chars": 10118,
"text_chars": 15347,
"text_sha256": "c5ac798c99358ba9bfb1c1e5a6634c07750fbffbc3d4fa406686e880ae4b8fa6",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 28.128748228773475,
"vllm_decode_tps": 116.18007219592856,
"vllm_generation_tokens_delta": 3268.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8081940831616521,
"vllm_prefill_tps": 533.2877448371432,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 112.90229749080078
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 17.911016230471432,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T07:12:32Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008739661425352097,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9152,
"text_chars": 9152,
"text_sha256": "2e2cda4a7e91f091f07b9fb76f6795b4b46d34c06d67b86dd9a0f10cd7b071a0",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 17.08993177022785,
"vllm_decode_tps": 117.02796868295134,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.812127098441124,
"vllm_prefill_tps": 530.7051086305376,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 111.66312253111941
},
"10K": {
"completion_tokens": 10000,
"content_chars": 46155,
"content_has_think_close": false,
"elapsed_seconds": 91.70880929846317,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T07:14:04Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.0056197261437773705,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 46155,
"qwen_answer_text_sha256": "99f0313665a258aa889dc09836d4da910585d581ad15d09867b8075d647ac554",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 46155,
"qwen_visible_text_sha256": "99f0313665a258aa889dc09836d4da910585d581ad15d09867b8075d647ac554",
"raw_has_think_close": false,
"reasoning_chars": 10118,
"text_chars": 56274,
"text_sha256": "7024979e63cc90d99ec03dc2fe1d81b8cafbb572aa23120bb1f334686519c3f5",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 90.89162033330649,
"vllm_decode_tps": 110.02114345997177,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8079116530716419,
"vllm_prefill_tps": 533.4741717876681,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 109.04077892294234
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 36, 39, 42, 45. One new sample.PASS2,911 natural tokens2K 117.36710K 110.169 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-09T00:00:14.441833328Z",
"FinishedAt": "2026-11-09T00:20:09.78005595Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "d3e8150b92459eb9bfeb0628cfbc7f1ba14417d265985bd295006c8f223de819",
"retest_strict": "PASS",
"retest_strict_tokens": 2911,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 2911,
"decode_tps": 116.74599429536063,
"prefill_seconds": 0.8083759220317006,
"client_total_seconds": 25.750953457318246,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 0.8083759220317006,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 24.93447434809059,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2911.0
},
"request_sha256": "e31c712c0771301f2e4d592bc966448d3143f2660e1b4cab33a91704f19e9370",
"summary_sha256": "eebfaae9c73243c913cb60f1c6f4ec103cac551b4f8d5c5529c5364a6f3465ca"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 117.36665638338557,
"prefill_seconds": 0.8091103108599782,
"client_total_seconds": 17.857541773468256,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 0.8091103108599782,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 17.04061495512724,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2000.0
},
"request_sha256": "bca01af4a8338cd6274c1c9e0b281d8e59066ebae2ece3e3a636620066da3a03",
"summary_sha256": "2c37396cb2c1b23f8ae028e9beb941f96a022ffb8a5da29d3363d63003b69672"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 110.16866681199451,
"prefill_seconds": 0.8098680507391691,
"client_total_seconds": 91.58954442292452,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 0.8098680507391691,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 90.76991026010364,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 10000.0
},
"request_sha256": "46fd7808353544eb86016bdd1746bafb67def6864a3f5352647b66cb9149f95a",
"summary_sha256": "b44fdcf5c45447a4a3ba6eda737cc6adcf2f183f3569935f415e850a380531e5"
}
},
"retest_strict_TPS": 116.74599429536063,
"retest_strict_prefill_seconds": 0.8083759220317006,
"retest_strict_total_seconds": 25.750953457318246,
"retest_2K_TPS": 117.36665638338557,
"retest_2K_prefill_seconds": 0.8091103108599782,
"retest_2K_total_seconds": 17.857541773468256,
"retest_10K_TPS": 110.16866681199451,
"retest_10K_prefill_seconds": 0.8098680507391691,
"retest_10K_total_seconds": 91.58954442292452,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 34334547968,
"1": 34314465280,
"2": 34314543104,
"3": 34314469376,
"4": 34334240768,
"5": 34314461184,
"6": 34314457088,
"7": 34318581760
},
"retest_sampled_peak_VRAM_GiB": 31.976539611816406,
"retest_memory_observation_count": 565,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 193,
"text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
"values": [
"8.88"
]
}
],
"available_KV_GiB": [
{
"line": 313,
"text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
"values": [
"19.91"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 315,
"text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
"values": [
"521,696"
]
}
],
"context_capacity": [
{
"line": 316,
"text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
"values": [
"131,072",
"15.82"
]
}
]
},
"retest_logged_model_load_delta_GiB": 8.88,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 19.91,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 521696,
"retest_logged_context_capacity_x": 15.82,
"retest_memory_log_source": "historical-recovered/fresh-row-36/result.json"
} | Evidence and limitsstrict-valid Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 3310,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "995ad96eacd98c81ed38be0c5b274b04031597b0",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r3.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-39.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 39,
"original_host": "10.0.0.20",
"logged_model_load_delta_GiB": 8.88,
"model_load_report_rank": 0,
"logged_model_load_seconds": 115.559999,
"logged_available_KV_GiB": 19.91,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 521696,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 15.82,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "6e661b86c7832729f1ccff5efa87579d419dd3771f3ed7e6ea42e1105ddfb44a",
"source_archive": "historical-recovered/host10.0.0.20-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/vnext-repeat-20260630T045153Z/runtime/vnext-launch-runs/20260630T050006Z-2928686/hf-moe35b-tp8",
"source_lines": {
"load": {
"number": 193,
"text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-30 06:13:45 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 115.559999 seconds"
},
"KV": {
"number": 313,
"text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-30 06:15:07 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB"
},
"tokens": {
"number": 315,
"text": "(EngineCore_DP0 pid=171) INFO 06-30 06:15:07 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens"
},
"capacity": {
"number": 316,
"text": "(EngineCore_DP0 pid=171) INFO 06-30 06:15:07 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x"
},
"engine": {
"number": 33,
"text": "(EngineCore_DP0 pid=171) INFO 06-30 06:11:18 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-35b-a3b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-35b-a3b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.6-35B-A3B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 19.91,
"logged_GPU_KV_tokens": 521696,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-36/result.json",
"retest_strict_tokens": 2911,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 34334547968,
"1": 34314465280,
"2": 34314543104,
"3": 34314469376,
"4": 34334240768,
"5": 34314461184,
"6": 34314457088,
"7": 34318581760
},
"retest_sampled_peak_VRAM_GiB": 31.976539611816406,
"retest_memory_observation_count": 565,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 193,
"text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
"values": [
"8.88"
]
}
],
"available_KV_GiB": [
{
"line": 313,
"text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
"values": [
"19.91"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 315,
"text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
"values": [
"521,696"
]
}
],
"context_capacity": [
{
"line": 316,
"text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
"values": [
"131,072",
"15.82"
]
}
]
},
"retest_KV_budget_GiB": 19.91,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 521696,
"retest_memory_log_source": "historical-recovered/fresh-row-36/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.30 · Full vNext Release-Readiness Serving Replay / table 2 | GGUF F16Qwen/Qwen3.6-27B | TP8MTP off | PASS4315 tokens | 71.161 | 66.638 | 14.232 / 42.34610K: 13.878 / 163.951Strict: 14.445 / 76.208Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 22.79 GiB · engine report373,360 grouped KV tokens · 11.32× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-40 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-40 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row40-4ed2ea0727-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-40/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-40/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-40/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-40/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-40/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-40/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense" python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-40 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-40 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 70.949 · valid historical-model sample10K 66.461 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3907,
"content_chars": 6521,
"content_has_think_close": false,
"elapsed_seconds": 69.24951494298875,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T07:50:37Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.012738149613142014,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 6521,
"qwen_answer_text_sha256": "f71a7aa77008154118acaa8bce77c983cf68537ad16c8aaf250141d295344e11",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 6521,
"qwen_visible_text_sha256": "f71a7aa77008154118acaa8bce77c983cf68537ad16c8aaf250141d295344e11",
"raw_has_think_close": false,
"reasoning_chars": 12901,
"text_chars": 19423,
"text_sha256": "6a78b9fc249f729fd22f6baf6d2d6fe6bda8d6b3238969514e27d0790adb833a",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 55.90619322191924,
"vllm_decode_tps": 69.88492284729871,
"vllm_generation_tokens_delta": 3907.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 13.3340172059834,
"vllm_prefill_tps": 32.323342121277335,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 56.419167747478475
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 41.90379103273153,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T07:51:19Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.005990299396216869,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9874,
"text_chars": 9874,
"text_sha256": "ce16880512a707739f7b919b99fd9aec38787d685976d559c94585be4e3f91e7",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 28.189258734695613,
"vllm_decode_tps": 70.94901000494846,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 13.706564854830503,
"vllm_prefill_tps": 31.444786098108736,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 47.72837852398073
},
"10K": {
"completion_tokens": 10000,
"content_chars": 40712,
"content_has_think_close": false,
"elapsed_seconds": 164.8181800954044,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T07:54:04Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008725512772798538,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 40712,
"qwen_answer_text_sha256": "259e6e860384fbf4cd65398f1acb30ababd88f54788727ae13a546228ad57c72",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 40712,
"qwen_visible_text_sha256": "259e6e860384fbf4cd65398f1acb30ababd88f54788727ae13a546228ad57c72",
"raw_has_think_close": false,
"reasoning_chars": 12901,
"text_chars": 53614,
"text_sha256": "821c84d987e2c88b9fd53463cde3f56f1d66c21f7557970817da935d3f5067f6",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 150.4640983948484,
"vllm_decode_tps": 66.46103692960672,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.344401407055557,
"vllm_prefill_tps": 30.046565748502044,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 60.6729184499643
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 37, 40, 43. One new sample.PASS4,263 natural tokens2K 70.87610K 66.474 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-09T00:22:27.410864084Z",
"FinishedAt": "2026-11-09T00:43:40.972906165Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "ae0dba47e379e10ae120c82dbe767b915856ebc0a56771c0751407ceb9b17f98",
"retest_strict": "PASS",
"retest_strict_tokens": 4263,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 4263,
"decode_tps": 69.65298485444228,
"prefill_seconds": 13.264510760083795,
"client_total_seconds": 74.47932942770422,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 13.264510760083795,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 61.20340727549046,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 4263.0
},
"request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
"summary_sha256": "2a4ec12838f2054993cfc40a0ce46452682e1e06ac8c785eefd538bcc5266ae6"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 70.8759028403918,
"prefill_seconds": 15.0672946870327,
"client_total_seconds": 43.29480273462832,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 15.0672946870327,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 28.218335426412523,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2000.0
},
"request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
"summary_sha256": "304d5e8b0f2afd6fcf019e0b454f8917c037eece212906cfcf2c1686136c1ee7"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 66.47442948531119,
"prefill_seconds": 14.876342931762338,
"client_total_seconds": 165.31903470959514,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 14.876342931762338,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 150.43378450069577,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 10000.0
},
"request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
"summary_sha256": "895a7e44c83b5069acf38a1a85397205ac0d8a1cefc38010e259fa29cda426f9"
}
},
"retest_strict_TPS": 69.65298485444228,
"retest_strict_prefill_seconds": 13.264510760083795,
"retest_strict_total_seconds": 74.47932942770422,
"retest_2K_TPS": 70.8759028403918,
"retest_2K_prefill_seconds": 15.0672946870327,
"retest_2K_total_seconds": 43.29480273462832,
"retest_10K_TPS": 66.47442948531119,
"retest_10K_prefill_seconds": 14.876342931762338,
"retest_10K_total_seconds": 165.31903470959514,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33744068608,
"1": 33634144256,
"2": 33684496384,
"3": 33634095104,
"4": 33684230144,
"5": 33633964032,
"6": 33684738048,
"7": 33634250752
},
"retest_sampled_peak_VRAM_GiB": 31.426612854003906,
"retest_memory_observation_count": 604,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-09T00:25:47.689313425Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:25:47 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 105.144041 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-09T00:28:42.161027038Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:28:42 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-09T00:28:42.250244890Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-09T00:28:42.250362411Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_logged_model_load_delta_GiB": 6.75,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_logged_context_capacity_x": 11.32,
"retest_memory_log_source": "historical-recovered/fresh-row-37/result.json"
} | Evidence and limitsstrict-valid; ai-info 10K gate cleared Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 4315,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r3.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-40.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 40,
"original_host": "10.0.0.30",
"logged_model_load_delta_GiB": 6.75,
"model_load_report_rank": 0,
"logged_model_load_seconds": 115.568181,
"logged_available_KV_GiB": 22.79,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 373360,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 11.32,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "8e5636e5ce75ea4e080735843d07126b395cbb4f4b8ea2928b3f6e92369a61ae",
"source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/vnext-release-repro/public-staging-20260629/runtime-full-current-20260630T111241Z/vnext-launch-runs/20260630T111242Z-2834957/gguf-dense27b-tp8",
"source_lines": {
"load": {
"number": 4588,
"text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-30 11:18:48 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 115.568181 seconds"
},
"KV": {
"number": 5168,
"text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-30 11:21:37 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB"
},
"tokens": {
"number": 5174,
"text": "(EngineCore_DP0 pid=200) INFO 06-30 11:21:37 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens"
},
"capacity": {
"number": 5175,
"text": "(EngineCore_DP0 pid=200) INFO 06-30 11:21:37 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x"
},
"engine": {
"number": 63,
"text": "(EngineCore_DP0 pid=200) INFO 06-30 11:16:27 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=6a9e13bd6fc8f0983b9b99948120bc37f49c13e9, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 22.79,
"logged_GPU_KV_tokens": 373360,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-37/result.json",
"retest_strict_tokens": 4263,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33744068608,
"1": 33634144256,
"2": 33684496384,
"3": 33634095104,
"4": 33684230144,
"5": 33633964032,
"6": 33684738048,
"7": 33634250752
},
"retest_sampled_peak_VRAM_GiB": 31.426612854003906,
"retest_memory_observation_count": 604,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-09T00:25:47.689313425Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:25:47 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 105.144041 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-09T00:28:42.161027038Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:28:42 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-09T00:28:42.250244890Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-09T00:28:42.250362411Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_memory_log_source": "historical-recovered/fresh-row-37/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.30 · Full vNext Release-Readiness Serving Replay / table 2 | HF FP16Qwen/Qwen3.6-27B | TP8MTP off | PASS3518 tokens | 71.326 | 66.810 | 14.080 / 42.12810K: 11.670 / 161.357Strict: 13.255 / 63.184Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.84 GiB (rank 0)Available KV: 22.64 GiB · engine report370,864 grouped KV tokens · 11.25× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-41 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-41 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row41-ae19602e82-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-41/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-41/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-41/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-41/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-41/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-41/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/opt/local-models/qwen36-27b-hf,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-41 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-41 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 71.149 · valid historical-model sample10K 66.668 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3920,
"content_chars": 6472,
"content_has_think_close": false,
"elapsed_seconds": 70.2467966740951,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T08:20:47Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.014053838327527046,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 6472,
"qwen_answer_text_sha256": "d79b6fa83b0fa444add4ed1d95c2f6f62c5c0cc2099f90436c3859434f444eee",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 6472,
"qwen_visible_text_sha256": "d79b6fa83b0fa444add4ed1d95c2f6f62c5c0cc2099f90436c3859434f444eee",
"raw_has_think_close": false,
"reasoning_chars": 12896,
"text_chars": 19369,
"text_sha256": "9e353abcb76fb81d15044b5756245d235d27d4868bca1f06c1b821426adc19c9",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 55.93259923905134,
"vllm_decode_tps": 70.08435247656276,
"vllm_generation_tokens_delta": 3920.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.302407649345696,
"vllm_prefill_tps": 30.134786433647577,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 55.803256313402514
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 41.84833710733801,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T08:21:29Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008913260884582996,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9935,
"text_chars": 9935,
"text_sha256": "83610c81afefe692098576ea3890219b4f89e9edc8bb008987daa4e2e22f54b7",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 28.109985790215433,
"vllm_decode_tps": 71.14909324131224,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 13.728937095962465,
"vllm_prefill_tps": 31.393544670457594,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 47.79162418975316
},
"10K": {
"completion_tokens": 10000,
"content_chars": 39962,
"content_has_think_close": false,
"elapsed_seconds": 164.7553375950083,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T08:24:14Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008609971031546593,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 39962,
"qwen_answer_text_sha256": "d8c1f94bd6b8093da4fd146802c26246aae4ff3246e4039fbe101a47424784c6",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 39962,
"qwen_visible_text_sha256": "d8c1f94bd6b8093da4fd146802c26246aae4ff3246e4039fbe101a47424784c6",
"raw_has_think_close": false,
"reasoning_chars": 12896,
"text_chars": 52859,
"text_sha256": "008e0ab44ea086b2f133d8b151934ce3e8f74780a735fe7804fd9c4647331b54",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 149.99608603771776,
"vllm_decode_tps": 66.66840625084988,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.749934704042971,
"vllm_prefill_tps": 29.22046833752169,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 60.696060874102905
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 35, 38, 41, 44. One new sample.PASS3,609 natural tokens2K 70.95110K 66.469 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-08T23:40:06.929336115Z",
"FinishedAt": "2026-11-08T23:58:45.910937408Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "a96218fee0c78b5145db76c43cdf273ff01d169be52e0662ddcc70106f936f79",
"retest_strict": "PASS",
"retest_strict_tokens": 3609,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 3609,
"decode_tps": 70.06009773846488,
"prefill_seconds": 12.782165881246328,
"client_total_seconds": 64.30384199228138,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 3609.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_sum": 12.782165881246328,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 51.512917002663016,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "ee97c35547770515d3bd63d45b2557687a8278bbeb38ce72abbaad2fee0ab250",
"summary_sha256": "6ae03f5709d05f854c0ff4b3e846b625d2d4a691aac551f35ae21d6a4b36c17e"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 70.95081153473755,
"prefill_seconds": 15.072314956225455,
"client_total_seconds": 43.269196966663,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 2000.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_sum": 15.072314956225455,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 28.188542974181473,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "d476546705501abd0468f2fc6527e9ace49f732079e53b75abf7bcfb98635f44",
"summary_sha256": "f70df1fbfc18c6af1f765a73663863bb438fbe1dd644049da3c9e283ae692d0e"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 66.46936099788869,
"prefill_seconds": 14.515854343771935,
"client_total_seconds": 164.97092120815068,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 10000.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_sum": 14.515854343771935,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 150.4452555263415,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "ee5df402413359190aaa7ed74f4a298f749233768156b8fc14f2ea4bef0e8017",
"summary_sha256": "1dfc4acf114048076b24fd3ed43f4a13dd3800ec8892ff4352974a55fd56aba2"
}
},
"retest_strict_TPS": 70.06009773846488,
"retest_strict_prefill_seconds": 12.782165881246328,
"retest_strict_total_seconds": 64.30384199228138,
"retest_2K_TPS": 70.95081153473755,
"retest_2K_prefill_seconds": 15.072314956225455,
"retest_2K_total_seconds": 43.269196966663,
"retest_10K_TPS": 66.46936099788869,
"retest_10K_prefill_seconds": 14.515854343771935,
"retest_10K_total_seconds": 164.97092120815068,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33776762880,
"1": 33666486272,
"2": 33716842496,
"3": 33666523136,
"4": 33716817920,
"5": 33666486272,
"6": 33716846592,
"7": 33666510848
},
"retest_sampled_peak_VRAM_GiB": 31.457061767578125,
"retest_memory_observation_count": 527,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 712,
"text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
"values": [
"6.84"
]
}
],
"available_KV_GiB": [
{
"line": 1267,
"text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
"values": [
"22.64"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 1273,
"text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
"values": [
"370,864"
]
}
],
"context_capacity": [
{
"line": 1274,
"text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
"values": [
"131,072",
"11.25"
]
}
]
},
"retest_logged_model_load_delta_GiB": 6.84,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 22.64,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 370864,
"retest_logged_context_capacity_x": 11.25,
"retest_memory_log_source": "historical-recovered/fresh-row-35/result.json"
} | Evidence and limitsstrict-valid; ai-info 10K gate cleared Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 3518,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r3.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-41.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 41,
"original_host": "10.0.0.30",
"logged_model_load_delta_GiB": 6.84,
"model_load_report_rank": 0,
"logged_model_load_seconds": 24.431786,
"logged_available_KV_GiB": 22.64,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 370864,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 11.25,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "31b012b6fd4429cc642f68c73f43900399682da4ad3cc8254abae19f76ce393f",
"source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/vnext-release-repro/public-staging-20260629/runtime-full-current-20260630T111241Z/vnext-launch-runs/20260630T111242Z-2834957/hf-dense27b-tp8",
"source_lines": {
"load": {
"number": 712,
"text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-30 11:53:09 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 24.431786 seconds"
},
"KV": {
"number": 1270,
"text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-30 11:54:49 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB"
},
"tokens": {
"number": 1273,
"text": "(EngineCore_DP0 pid=184) INFO 06-30 11:54:49 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens"
},
"capacity": {
"number": 1274,
"text": "(EngineCore_DP0 pid=184) INFO 06-30 11:54:49 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x"
},
"engine": {
"number": 55,
"text": "(EngineCore_DP0 pid=184) INFO 06-30 11:52:18 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-27b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 22.64,
"logged_GPU_KV_tokens": 370864,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-35/result.json",
"retest_strict_tokens": 3609,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33776762880,
"1": 33666486272,
"2": 33716842496,
"3": 33666523136,
"4": 33716817920,
"5": 33666486272,
"6": 33716846592,
"7": 33666510848
},
"retest_sampled_peak_VRAM_GiB": 31.457061767578125,
"retest_memory_observation_count": 527,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 712,
"text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
"values": [
"6.84"
]
}
],
"available_KV_GiB": [
{
"line": 1267,
"text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
"values": [
"22.64"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 1273,
"text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
"values": [
"370,864"
]
}
],
"context_capacity": [
{
"line": 1274,
"text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
"values": [
"131,072",
"11.25"
]
}
]
},
"retest_KV_budget_GiB": 22.64,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 370864,
"retest_memory_log_source": "historical-recovered/fresh-row-35/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.30 · Full vNext Release-Readiness Serving Replay / table 2 | HF FP16Qwen/Qwen3.6-35B-A3B | TP8MTP off | PASS3657 tokens | 115.895 | 109.001 | 0.813 / 18.07810K: 0.811 / 92.562Strict: 0.811 / 32.717Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 8.88 GiB (rank 0)Available KV: 19.91 GiB · engine report521,696 grouped KV tokens · 15.82× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-42 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-42 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row42-acca00650b-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-42/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-42/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-42/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-42/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-42/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-42/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-moe,dst=/opt/local-models/qwen36-35b-a3b-hf,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-moe "$MODEL_DIR/qwen36-moe" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-42 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-42 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 117.029 · valid historical-model sample10K 110.049 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 4563,
"content_chars": 9559,
"content_has_think_close": false,
"elapsed_seconds": 40.47988819144666,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T09:03:15Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.01213598158210516,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 9559,
"qwen_answer_text_sha256": "c0bf1c311f22fd6f7b5cc4f84da6468132661431aef27df3dfbb257bd7c5ebf4",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 9559,
"qwen_visible_text_sha256": "c0bf1c311f22fd6f7b5cc4f84da6468132661431aef27df3dfbb257bd7c5ebf4",
"raw_has_think_close": false,
"reasoning_chars": 14042,
"text_chars": 23602,
"text_sha256": "66143a28ce4a77ff43265dad18efd33c26b176e6beae677a61343a6964631afc",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 39.66698031872511,
"vllm_decode_tps": 115.03270385938605,
"vllm_generation_tokens_delta": 4563.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8039803570136428,
"vllm_prefill_tps": 536.0827490872222,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 112.72264336352971
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 17.905847259797156,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T09:03:33Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.005839169956743717,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 10261,
"text_chars": 10261,
"text_sha256": "c4fec0daf03f83f76b3d525966214f007fa4c81c71478fe6125229c7da45b99b",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 17.089729534462094,
"vllm_decode_tps": 117.02935356390067,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8079051412642002,
"vllm_prefill_tps": 533.478471650244,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 111.69535688436655
},
"10K": {
"completion_tokens": 10000,
"content_chars": 42369,
"content_has_think_close": false,
"elapsed_seconds": 91.69410692714155,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T09:05:05Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.006687283515930176,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 42369,
"qwen_answer_text_sha256": "4b64ba6d64efec639f826c39f2d980a8d2fbc330e9c841ed671d422667c1738a",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 42369,
"qwen_visible_text_sha256": "4b64ba6d64efec639f826c39f2d980a8d2fbc330e9c841ed671d422667c1738a",
"raw_has_think_close": false,
"reasoning_chars": 14042,
"text_chars": 56412,
"text_sha256": "24d73cff4cbffbb79f61919dbac79b0311246bb8408bdcad000d2265afb850c0",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 90.86887793801725,
"vllm_decode_tps": 110.04867922789934,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8149959305301309,
"vllm_prefill_tps": 528.836996424813,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 109.0582626857996
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 36, 39, 42, 45. One new sample.PASS2,911 natural tokens2K 117.36710K 110.169 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-09T00:00:14.441833328Z",
"FinishedAt": "2026-11-09T00:20:09.78005595Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "d3e8150b92459eb9bfeb0628cfbc7f1ba14417d265985bd295006c8f223de819",
"retest_strict": "PASS",
"retest_strict_tokens": 2911,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 2911,
"decode_tps": 116.74599429536063,
"prefill_seconds": 0.8083759220317006,
"client_total_seconds": 25.750953457318246,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 0.8083759220317006,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 24.93447434809059,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2911.0
},
"request_sha256": "e31c712c0771301f2e4d592bc966448d3143f2660e1b4cab33a91704f19e9370",
"summary_sha256": "eebfaae9c73243c913cb60f1c6f4ec103cac551b4f8d5c5529c5364a6f3465ca"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 117.36665638338557,
"prefill_seconds": 0.8091103108599782,
"client_total_seconds": 17.857541773468256,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 0.8091103108599782,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 17.04061495512724,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2000.0
},
"request_sha256": "bca01af4a8338cd6274c1c9e0b281d8e59066ebae2ece3e3a636620066da3a03",
"summary_sha256": "2c37396cb2c1b23f8ae028e9beb941f96a022ffb8a5da29d3363d63003b69672"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 110.16866681199451,
"prefill_seconds": 0.8098680507391691,
"client_total_seconds": 91.58954442292452,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 0.8098680507391691,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 90.76991026010364,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 10000.0
},
"request_sha256": "46fd7808353544eb86016bdd1746bafb67def6864a3f5352647b66cb9149f95a",
"summary_sha256": "b44fdcf5c45447a4a3ba6eda737cc6adcf2f183f3569935f415e850a380531e5"
}
},
"retest_strict_TPS": 116.74599429536063,
"retest_strict_prefill_seconds": 0.8083759220317006,
"retest_strict_total_seconds": 25.750953457318246,
"retest_2K_TPS": 117.36665638338557,
"retest_2K_prefill_seconds": 0.8091103108599782,
"retest_2K_total_seconds": 17.857541773468256,
"retest_10K_TPS": 110.16866681199451,
"retest_10K_prefill_seconds": 0.8098680507391691,
"retest_10K_total_seconds": 91.58954442292452,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 34334547968,
"1": 34314465280,
"2": 34314543104,
"3": 34314469376,
"4": 34334240768,
"5": 34314461184,
"6": 34314457088,
"7": 34318581760
},
"retest_sampled_peak_VRAM_GiB": 31.976539611816406,
"retest_memory_observation_count": 565,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 193,
"text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
"values": [
"8.88"
]
}
],
"available_KV_GiB": [
{
"line": 313,
"text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
"values": [
"19.91"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 315,
"text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
"values": [
"521,696"
]
}
],
"context_capacity": [
{
"line": 316,
"text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
"values": [
"131,072",
"15.82"
]
}
]
},
"retest_logged_model_load_delta_GiB": 8.88,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 19.91,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 521696,
"retest_logged_context_capacity_x": 15.82,
"retest_memory_log_source": "historical-recovered/fresh-row-36/result.json"
} | Evidence and limitsstrict-valid Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 3657,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "995ad96eacd98c81ed38be0c5b274b04031597b0",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r3.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-42.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 42,
"original_host": "10.0.0.30",
"logged_model_load_delta_GiB": 8.88,
"model_load_report_rank": 0,
"logged_model_load_seconds": 110.381889,
"logged_available_KV_GiB": 19.91,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 521696,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 15.82,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "537cbbf045673c0c5d8d0ae112aca2b44ca8cf1d8a8055c940cdd6d0e50761ed",
"source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/vnext-release-repro/public-staging-20260629/runtime-full-current-20260630T111241Z/vnext-launch-runs/20260630T111242Z-2834957/hf-moe35b-tp8",
"source_lines": {
"load": {
"number": 193,
"text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-30 12:26:46 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 110.381889 seconds"
},
"KV": {
"number": 311,
"text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-30 12:28:16 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB"
},
"tokens": {
"number": 315,
"text": "(EngineCore_DP0 pid=171) INFO 06-30 12:28:16 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens"
},
"capacity": {
"number": 316,
"text": "(EngineCore_DP0 pid=171) INFO 06-30 12:28:16 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x"
},
"engine": {
"number": 33,
"text": "(EngineCore_DP0 pid=171) INFO 06-30 12:24:26 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-35b-a3b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-35b-a3b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.6-35B-A3B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 19.91,
"logged_GPU_KV_tokens": 521696,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-36/result.json",
"retest_strict_tokens": 2911,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 34334547968,
"1": 34314465280,
"2": 34314543104,
"3": 34314469376,
"4": 34334240768,
"5": 34314461184,
"6": 34314457088,
"7": 34318581760
},
"retest_sampled_peak_VRAM_GiB": 31.976539611816406,
"retest_memory_observation_count": 565,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 193,
"text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
"values": [
"8.88"
]
}
],
"available_KV_GiB": [
{
"line": 313,
"text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
"values": [
"19.91"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 315,
"text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
"values": [
"521,696"
]
}
],
"context_capacity": [
{
"line": 316,
"text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
"values": [
"131,072",
"15.82"
]
}
]
},
"retest_KV_budget_GiB": 19.91,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 521696,
"retest_memory_log_source": "historical-recovered/fresh-row-36/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.30 · Full vNext Release-Readiness Serving Replay / table 3 | GGUF F16Qwen/Qwen3.6-27B | TP8MTP off | PASS3742 tokens | 71.081 | 66.543 | 12.165 / 40.31110K: 12.343 / 162.632Strict: 14.023 / 67.416Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.75 GiB (rank 0)Available KV: 22.79 GiB · engine report373,360 grouped KV tokens · 11.32× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-43 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-43 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row43-5690e3a7ec-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-43/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-43/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-43/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-43/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-43/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-43/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-gguf/Qwen3.6-27B-F16.gguf,dst=/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-27b-text-config-eosfix,dst=/opt/local-models/qwen36-27b-text-config-eosfix,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense" python3 launch-tools-r2/model-acquisition/acquire_gguf.py "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-43 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-43 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 70.836 · valid historical-model sample10K 66.352 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3649,
"content_chars": 6475,
"content_has_think_close": false,
"elapsed_seconds": 66.57352643553168,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T09:41:13Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.012507816776633263,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 6475,
"qwen_answer_text_sha256": "925a28033b3d31f0cc40bce6a370158e0957324d09afef3a046da393cd2d5a6c",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 6475,
"qwen_visible_text_sha256": "925a28033b3d31f0cc40bce6a370158e0957324d09afef3a046da393cd2d5a6c",
"raw_has_think_close": false,
"reasoning_chars": 11161,
"text_chars": 17637,
"text_sha256": "f364c8a84832f94e723a28a513e06a4560a7ef6759d7a9117df82e9d93eadfa5",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 52.22083283774555,
"vllm_decode_tps": 69.87632716118766,
"vllm_generation_tokens_delta": 3649.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.343986123800278,
"vllm_prefill_tps": 30.047435648648786,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 54.811577444881344
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 43.17953444086015,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T09:41:56Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008289745077490807,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9518,
"text_chars": 9518,
"text_sha256": "2e522d4f025ae541c3176110880f955f89516600fa7c9c2e05d1f38f19146aee",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 28.23441171180457,
"vllm_decode_tps": 70.83554707689613,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.937016525305808,
"vllm_prefill_tps": 28.854490404413347,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 46.31823908938281
},
"10K": {
"completion_tokens": 10000,
"content_chars": 39271,
"content_has_think_close": false,
"elapsed_seconds": 164.38277336861938,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T09:44:41Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008772118017077446,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B-F16-GGUF-lmheadunquant",
"prompt_repeat": 32,
"qwen_answer_text_chars": 39271,
"qwen_answer_text_sha256": "b7d029235f4bef11afa9b73a697e4e253bf00a0aa9615344c274272fd62549e9",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 39271,
"qwen_visible_text_sha256": "b7d029235f4bef11afa9b73a697e4e253bf00a0aa9615344c274272fd62549e9",
"raw_has_think_close": false,
"reasoning_chars": 11161,
"text_chars": 50433,
"text_sha256": "75f602e176c6e4d050f56384f17751b47349c8769a376212726722bb6d890950",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 150.71104567032307,
"vllm_decode_tps": 66.35213733354865,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 13.661930229514837,
"vllm_prefill_tps": 31.547518744377726,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 60.833625051303564
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 37, 40, 43. One new sample.PASS4,263 natural tokens2K 70.87610K 66.474 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-09T00:22:27.410864084Z",
"FinishedAt": "2026-11-09T00:43:40.972906165Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "ae0dba47e379e10ae120c82dbe767b915856ebc0a56771c0751407ceb9b17f98",
"retest_strict": "PASS",
"retest_strict_tokens": 4263,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 4263,
"decode_tps": 69.65298485444228,
"prefill_seconds": 13.264510760083795,
"client_total_seconds": 74.47932942770422,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 13.264510760083795,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 61.20340727549046,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 4263.0
},
"request_sha256": "cee5b3e69c4536b401273fbab9093ac75f2e77c2992f89ab5e2c8f2cc20fcad8",
"summary_sha256": "2a4ec12838f2054993cfc40a0ce46452682e1e06ac8c785eefd538bcc5266ae6"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 70.8759028403918,
"prefill_seconds": 15.0672946870327,
"client_total_seconds": 43.29480273462832,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 15.0672946870327,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 28.218335426412523,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2000.0
},
"request_sha256": "fb3631f25efa0d86c0631caf9b534448d8eb1d0d295d20ccbddb57f4b2b8168a",
"summary_sha256": "304d5e8b0f2afd6fcf019e0b454f8917c037eece212906cfcf2c1686136c1ee7"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 66.47442948531119,
"prefill_seconds": 14.876342931762338,
"client_total_seconds": 165.31903470959514,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 14.876342931762338,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 150.43378450069577,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 10000.0
},
"request_sha256": "0d4c6bf01c385fa782ffef87f014c7a329ca06561609e32989c089837866415f",
"summary_sha256": "895a7e44c83b5069acf38a1a85397205ac0d8a1cefc38010e259fa29cda426f9"
}
},
"retest_strict_TPS": 69.65298485444228,
"retest_strict_prefill_seconds": 13.264510760083795,
"retest_strict_total_seconds": 74.47932942770422,
"retest_2K_TPS": 70.8759028403918,
"retest_2K_prefill_seconds": 15.0672946870327,
"retest_2K_total_seconds": 43.29480273462832,
"retest_10K_TPS": 66.47442948531119,
"retest_10K_prefill_seconds": 14.876342931762338,
"retest_10K_total_seconds": 165.31903470959514,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33744068608,
"1": 33634144256,
"2": 33684496384,
"3": 33634095104,
"4": 33684230144,
"5": 33633964032,
"6": 33684738048,
"7": 33634250752
},
"retest_sampled_peak_VRAM_GiB": 31.426612854003906,
"retest_memory_observation_count": 604,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-09T00:25:47.689313425Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:25:47 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 105.144041 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-09T00:28:42.161027038Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:28:42 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-09T00:28:42.250244890Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-09T00:28:42.250362411Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_logged_model_load_delta_GiB": 6.75,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_logged_context_capacity_x": 11.32,
"retest_memory_log_source": "historical-recovered/fresh-row-37/result.json"
} | Evidence and limitsstrict-valid; ai-info 10K gate cleared Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 3742,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r3.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-43.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 43,
"original_host": "10.0.0.30",
"logged_model_load_delta_GiB": 6.75,
"model_load_report_rank": 0,
"logged_model_load_seconds": 108.890125,
"logged_available_KV_GiB": 22.79,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 373360,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 11.32,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "a9babfca90bf4dc23965294f0abf27b401f6228b3748e1864c5814dd2d3dc3c4",
"source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/vnext-release-repro/public-staging-20260629/runtime-full-current-proof/vnext-launch-runs/20260630T130329Z-2951956/gguf-dense27b-tp8",
"source_lines": {
"load": {
"number": 4588,
"text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-30 13:09:21 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 108.890125 seconds"
},
"KV": {
"number": 5163,
"text": "(Worker pid=227) (Worker_TP0 pid=227) INFO 06-30 13:12:06 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB"
},
"tokens": {
"number": 5174,
"text": "(EngineCore_DP0 pid=200) INFO 06-30 13:12:08 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens"
},
"capacity": {
"number": 5175,
"text": "(EngineCore_DP0 pid=200) INFO 06-30 13:12:08 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x"
},
"engine": {
"number": 63,
"text": "(EngineCore_DP0 pid=200) INFO 06-30 13:07:07 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-f16-gguf/Qwen3.6-27B-F16.gguf', speculative_config=None, tokenizer='Qwen/Qwen3.6-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=6a9e13bd6fc8f0983b9b99948120bc37f49c13e9, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=gguf, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B-F16-GGUF-lmheadunquant, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': PosixPath('/runwork/debug_dump'), 'cache_dir': '/runwork/compile_cache', 'compile_cache_save_format': 'unpacked', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': True, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 22.79,
"logged_GPU_KV_tokens": 373360,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-37/result.json",
"retest_strict_tokens": 4263,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33744068608,
"1": 33634144256,
"2": 33684496384,
"3": 33634095104,
"4": 33684230144,
"5": 33633964032,
"6": 33684738048,
"7": 33634250752
},
"retest_sampled_peak_VRAM_GiB": 31.426612854003906,
"retest_memory_observation_count": 604,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 4589,
"text": "2026-11-09T00:25:47.689313425Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:25:47 [gpu_model_runner.py:4579] Model loading took 6.75 GiB memory and 105.144041 seconds",
"values": [
"6.75"
]
}
],
"available_KV_GiB": [
{
"line": 5172,
"text": "2026-11-09T00:28:42.161027038Z (Worker pid=227) (Worker_TP0 pid=227) INFO 11-09 00:28:42 [gpu_worker.py:453] Available KV cache memory: 22.79 GiB",
"values": [
"22.79"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 5175,
"text": "2026-11-09T00:28:42.250244890Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1314] GPU KV cache size: 373,360 tokens",
"values": [
"373,360"
]
}
],
"context_capacity": [
{
"line": 5176,
"text": "2026-11-09T00:28:42.250362411Z (EngineCore_DP0 pid=200) INFO 11-09 00:28:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.32x",
"values": [
"131,072",
"11.32"
]
}
]
},
"retest_KV_budget_GiB": 22.79,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 373360,
"retest_memory_log_source": "historical-recovered/fresh-row-37/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.30 · Full vNext Release-Readiness Serving Replay / table 3 | HF FP16Qwen/Qwen3.6-27B | TP8MTP off | PASS3742 tokens | 71.254 | 66.796 | 14.435 / 42.51210K: 13.400 / 163.117Strict: 14.229 / 67.462Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 6.84 GiB (rank 0)Available KV: 22.64 GiB · engine report370,864 grouped KV tokens · 11.25× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-44 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-44 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row44-fb2c570fcb-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-44/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-44/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-44/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-44/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-44/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-44/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-dense,dst=/opt/local-models/qwen36-27b-hf,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-dense "$MODEL_DIR/qwen36-dense" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-44 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-44 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 71.545 · valid historical-model sample10K 67.124 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3547,
"content_chars": 6457,
"content_has_think_close": false,
"elapsed_seconds": 64.16236412711442,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T10:11:23Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.013237979263067245,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 6457,
"qwen_answer_text_sha256": "3ad006f79090660185525516d2562a6210d7ac8433bbec5df068b059fa1ec551",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 6457,
"qwen_visible_text_sha256": "3ad006f79090660185525516d2562a6210d7ac8433bbec5df068b059fa1ec551",
"raw_has_think_close": false,
"reasoning_chars": 11482,
"text_chars": 17940,
"text_sha256": "fec7940db9e37cea727ec405d4b1b0ceaa254e277632e0b30b189fde4134c4c5",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 50.13813655637205,
"vllm_decode_tps": 70.74455182457739,
"vllm_generation_tokens_delta": 3547.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.016393848694861,
"vllm_prefill_tps": 30.7497067114829,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 55.28162885290367
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 42.49045191332698,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T10:12:05Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008615121245384216,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9954,
"text_chars": 9954,
"text_sha256": "901ca3921ce1cca130da42852733cff33e53103148af2ea7b9fdac37416bf1a1",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 27.954272541217506,
"vllm_decode_tps": 71.54541392737288,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.52750407718122,
"vllm_prefill_tps": 29.66786295224549,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 47.06939818102305
},
"10K": {
"completion_tokens": 10000,
"content_chars": 41009,
"content_has_think_close": false,
"elapsed_seconds": 163.0703025655821,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T10:14:48Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.00643538311123848,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen3.6-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 41009,
"qwen_answer_text_sha256": "baab8ed253ef12664fea03f14001f648e3b72046331a0e03aacf6420570d88e6",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 41009,
"qwen_visible_text_sha256": "baab8ed253ef12664fea03f14001f648e3b72046331a0e03aacf6420570d88e6",
"raw_has_think_close": false,
"reasoning_chars": 11482,
"text_chars": 52492,
"text_sha256": "1690d20b0f764e652f782470928e6a6525e42a044bd9c10a0fbd3bdb8daa4486",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 148.97759592998773,
"vllm_decode_tps": 67.1241869462004,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.08258059155196,
"vllm_prefill_tps": 30.605186116140807,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 61.323244285870466
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 35, 38, 41, 44. One new sample.PASS3,609 natural tokens2K 70.95110K 66.469 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-08T23:40:06.929336115Z",
"FinishedAt": "2026-11-08T23:58:45.910937408Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "a96218fee0c78b5145db76c43cdf273ff01d169be52e0662ddcc70106f936f79",
"retest_strict": "PASS",
"retest_strict_tokens": 3609,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 3609,
"decode_tps": 70.06009773846488,
"prefill_seconds": 12.782165881246328,
"client_total_seconds": 64.30384199228138,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 3609.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_sum": 12.782165881246328,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 51.512917002663016,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "ee97c35547770515d3bd63d45b2557687a8278bbeb38ce72abbaad2fee0ab250",
"summary_sha256": "6ae03f5709d05f854c0ff4b3e846b625d2d4a691aac551f35ae21d6a4b36c17e"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 70.95081153473755,
"prefill_seconds": 15.072314956225455,
"client_total_seconds": 43.269196966663,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 2000.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_sum": 15.072314956225455,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 28.188542974181473,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "d476546705501abd0468f2fc6527e9ace49f732079e53b75abf7bcfb98635f44",
"summary_sha256": "f70df1fbfc18c6af1f765a73663863bb438fbe1dd644049da3c9e283ae692d0e"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 66.46936099788869,
"prefill_seconds": 14.515854343771935,
"client_total_seconds": 164.97092120815068,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:generation_tokens_total": 10000.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_prefill_time_seconds_sum": 14.515854343771935,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:request_decode_time_seconds_sum": 150.4452555263415,
"vllm:request_decode_time_seconds_count": 1.0
},
"request_sha256": "ee5df402413359190aaa7ed74f4a298f749233768156b8fc14f2ea4bef0e8017",
"summary_sha256": "1dfc4acf114048076b24fd3ed43f4a13dd3800ec8892ff4352974a55fd56aba2"
}
},
"retest_strict_TPS": 70.06009773846488,
"retest_strict_prefill_seconds": 12.782165881246328,
"retest_strict_total_seconds": 64.30384199228138,
"retest_2K_TPS": 70.95081153473755,
"retest_2K_prefill_seconds": 15.072314956225455,
"retest_2K_total_seconds": 43.269196966663,
"retest_10K_TPS": 66.46936099788869,
"retest_10K_prefill_seconds": 14.515854343771935,
"retest_10K_total_seconds": 164.97092120815068,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33776762880,
"1": 33666486272,
"2": 33716842496,
"3": 33666523136,
"4": 33716817920,
"5": 33666486272,
"6": 33716846592,
"7": 33666510848
},
"retest_sampled_peak_VRAM_GiB": 31.457061767578125,
"retest_memory_observation_count": 527,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 712,
"text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
"values": [
"6.84"
]
}
],
"available_KV_GiB": [
{
"line": 1267,
"text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
"values": [
"22.64"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 1273,
"text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
"values": [
"370,864"
]
}
],
"context_capacity": [
{
"line": 1274,
"text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
"values": [
"131,072",
"11.25"
]
}
]
},
"retest_logged_model_load_delta_GiB": 6.84,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 22.64,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 370864,
"retest_logged_context_capacity_x": 11.25,
"retest_memory_log_source": "historical-recovered/fresh-row-35/result.json"
} | Evidence and limitsstrict-valid; ai-info 10K gate cleared Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 3742,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r3.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-44.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 44,
"original_host": "10.0.0.30",
"logged_model_load_delta_GiB": 6.84,
"model_load_report_rank": 0,
"logged_model_load_seconds": 23.129179,
"logged_available_KV_GiB": 22.64,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 370864,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 11.25,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "b5013067f3d140ab979a82911c5da3b6999e4ab982bb451aba02c56a7eb79a3a",
"source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/vnext-release-repro/public-staging-20260629/runtime-full-current-proof/vnext-launch-runs/20260630T130329Z-2951956/hf-dense27b-tp8",
"source_lines": {
"load": {
"number": 712,
"text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-30 13:43:37 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 23.129179 seconds"
},
"KV": {
"number": 1269,
"text": "(Worker pid=202) (Worker_TP0 pid=202) INFO 06-30 13:45:17 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB"
},
"tokens": {
"number": 1273,
"text": "(EngineCore_DP0 pid=184) INFO 06-30 13:45:17 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens"
},
"capacity": {
"number": 1274,
"text": "(EngineCore_DP0 pid=184) INFO 06-30 13:45:17 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x"
},
"engine": {
"number": 55,
"text": "(EngineCore_DP0 pid=184) INFO 06-30 13:42:48 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-27b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-27b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.6-27B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [4], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 22.64,
"logged_GPU_KV_tokens": 370864,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-35/result.json",
"retest_strict_tokens": 3609,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 33776762880,
"1": 33666486272,
"2": 33716842496,
"3": 33666523136,
"4": 33716817920,
"5": 33666486272,
"6": 33716846592,
"7": 33666510848
},
"retest_sampled_peak_VRAM_GiB": 31.457061767578125,
"retest_memory_observation_count": 527,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 712,
"text": "2026-11-08T23:42:07.039298012Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:42:07 [gpu_model_runner.py:4579] Model loading took 6.84 GiB memory and 22.869422 seconds",
"values": [
"6.84"
]
}
],
"available_KV_GiB": [
{
"line": 1267,
"text": "2026-11-08T23:43:52.218209587Z (Worker pid=202) (Worker_TP0 pid=202) INFO 11-08 23:43:52 [gpu_worker.py:453] Available KV cache memory: 22.64 GiB",
"values": [
"22.64"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 1273,
"text": "2026-11-08T23:43:52.553188353Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1314] GPU KV cache size: 370,864 tokens",
"values": [
"370,864"
]
}
],
"context_capacity": [
{
"line": 1274,
"text": "2026-11-08T23:43:52.553263624Z (EngineCore_DP0 pid=184) INFO 11-08 23:43:52 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.25x",
"values": [
"131,072",
"11.25"
]
}
]
},
"retest_KV_budget_GiB": 22.64,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 370864,
"retest_memory_log_source": "historical-recovered/fresh-row-35/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50Historical vNext Qwen3.6Host 10.0.0.30 · Full vNext Release-Readiness Serving Replay / table 3 | HF FP16Qwen/Qwen3.6-35B-A3B | TP8MTP off | PASS3719 tokens | 116.704 | 109.754 | 0.811 / 17.95610K: 0.806 / 91.928Strict: 0.811 / 33.017Recovered original raw result | — / —footprint not reported in the historical table / VRAM GiBReported card capacity: 31.98 GiB; usage not reportedLogged load delta: 8.88 GiB (rank 0)Available KV: 19.91 GiB · engine report521,696 grouped KV tokens · 15.82× reported context capacityOriginal memory log reports · rounded, not physical usage | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:8c380e9ca48943d8617de5a2e2eaf32a26dcc2c341e4b4f4f8c45294a72b8f1eRuntime image config: sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-45 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-45 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained historical portsThe launcher reads server and proxy ports from each hash-checked original benchmark environment. Row 29 retains 8078/19078; the other historical recipes retain 8001/19080. This repairs a missing plan field without changing the original ports, runtime, model or sampling. Retained lab Docker command · not portabledocker \ run \ -d \ --name \ historical-row45-ff4f548663-reproduction \ --network \ host \ --ipc \ host \ --privileged \ --device \ /dev/kfd \ --device \ /dev/dri \ --group-add \ video \ --cap-add \ SYS_PTRACE \ --cap-add \ IPC_LOCK \ --security-opt \ seccomp=unconfined \ --security-opt \ label=disable \ --ulimit \ memlock=-1:-1 \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-45/hf,dst=/root/.cache/huggingface \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-45/launch,dst=/runwork \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-45/cache/vllm,dst=/root/.cache/vllm \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-45/cache/triton,dst=/root/.triton \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-45/cache/inductor,dst=/tmp/torchinductor_root \ --mount \ type=bind,src=/home/ai/localaiservers-historical-20261108-r3/serve-row-45/overlay,dst=/opt/vllm_patch_bundle,readonly \ --mount \ type=bind,src=/usr/share/ollama/localaiservers-repro-models/qwen36-moe,dst=/opt/local-models/qwen36-35b-a3b-hf,readonly \ -e \ HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e \ LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/usr/local/lib:/opt/rocm/lib:/opt/venv/lib/python3.12/site-packages/torch/lib \ -e \ VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e \ HF_HUB_OFFLINE=1 \ --entrypoint \ /runwork/container_entrypoint.sh \ sha256:45decd88eb7c10c0408327438e07c2a655e45cc7534f8b662e5c4089a6b88568Historical host paths and device addresses are retained as evidence. Use the serving command above. Historical launch provenanceRecovered original scripts are supplied byte-for-byte. The prepared serving/benchmark commands use new paths and immutable image ID; these are not claimed as recovered fully resolved original Docker argv. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-model-collection python3 launch-tools-r2/model-acquisition/acquire_model.py qwen36-moe "$MODEL_DIR/qwen36-moe" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-historical-r6.tar.gz.sha256 mkdir -p launch-tools-historical-r6 tar -xzf launch-tools-historical-r6.tar.gz -C launch-tools-historical-r6 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-model-collection RUN_DIR=/absolute/path/to/new-runs/historical-row-45 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-historical-r6/portable/launch.py --config historical-row-45 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script launch-tools-r2.tar.gz · SHA-256 Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 116.069 · valid historical-model sample10K 109.160 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 3506,
"content_chars": 6574,
"content_has_think_close": false,
"elapsed_seconds": 31.27900091558695,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-10T10:53:44Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.011204360984265804,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 6574,
"qwen_answer_text_sha256": "4cc902e2dbd5bf26afa3078ea66d42bfcc2b84060bb1c08945302077e2c2901e",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 6574,
"qwen_visible_text_sha256": "4cc902e2dbd5bf26afa3078ea66d42bfcc2b84060bb1c08945302077e2c2901e",
"raw_has_think_close": false,
"reasoning_chars": 11741,
"text_chars": 18316,
"text_sha256": "74a69c47113596ca51a504d2471f356d634a7eba1608ff1fc76943610f355128",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 30.453829997219145,
"vllm_decode_tps": 115.12509265074857,
"vllm_generation_tokens_delta": 3506.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.815642450004816,
"vllm_prefill_tps": 528.4178134640431,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 112.08797907138046
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 18.055767958052456,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T10:54:03Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008598048239946365,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 10281,
"text_chars": 10281,
"text_sha256": "1952df1c990e0bd3a5ce8f3b98d46928f2d5a3c9c4516df616d4e9743568620b",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 17.23110121395439,
"vllm_decode_tps": 116.06919228008047,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8160559171810746,
"vllm_prefill_tps": 528.1500825198543,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 110.76792771409349
},
"10K": {
"completion_tokens": 10000,
"content_chars": 48989,
"content_has_think_close": false,
"elapsed_seconds": 92.42294359579682,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-10T10:55:35Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008693818934261799,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.6-35B-A3B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 48989,
"qwen_answer_text_sha256": "e8d83b5df9c7028076f75b0d2e0919b919b8ba383facde99c261cc4b7ebb1d39",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 48989,
"qwen_visible_text_sha256": "e8d83b5df9c7028076f75b0d2e0919b919b8ba383facde99c261cc4b7ebb1d39",
"raw_has_think_close": false,
"reasoning_chars": 11741,
"text_chars": 60731,
"text_sha256": "73352886b27a6286ec20b2c30a86c9440ba82d3da48a921453598027c9484fad",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 91.60833835601807,
"vllm_decode_tps": 109.16036879892894,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 0.8050807006657124,
"vllm_prefill_tps": 535.3500582533042,
"vllm_prompt_tokens_delta": 431.0,
"wall_completion_tps": 108.19824181032443
}
}Earlier exported-image retest · separate evidenceHistorical export retest completedActual CPU installation checks · Export importFresh result / replay · Sealed raw evidenceShared variant test: original rows 36, 39, 42, 45. One new sample.PASS2,911 natural tokens2K 117.36710K 110.169 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2026-11-09T00:00:14.441833328Z",
"FinishedAt": "2026-11-09T00:20:09.78005595Z"
},
"retest_historical_limits": [
"One fresh measurement per distinct runtime variant; grouped historical rows are not independent new samples.",
"Historical published measurements remain separate from this fresh run.",
"Installed source and mapped native identities establish admitted runtime, not a dynamic per-kernel invocation trace.",
"Fresh root-SSD caches and the current host state differ from the historical runs.",
"Physical VRAM is sampled, not a continuous peak; logged allocation budgets are separate from tensor-byte totals."
],
"retest_historical_current_Qwen38_official_eligible": false,
"retest_historical_verified_regular_files": 173,
"retest_historical_manifest_sha256": "d3e8150b92459eb9bfeb0628cfbc7f1ba14417d265985bd295006c8f223de819",
"retest_strict": "PASS",
"retest_strict_tokens": 2911,
"retest_historical_2K_eligible": true,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 2911,
"decode_tps": 116.74599429536063,
"prefill_seconds": 0.8083759220317006,
"client_total_seconds": 25.750953457318246,
"strict_pass": true,
"finish_reason": "stop",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 0.8083759220317006,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 24.93447434809059,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2911.0
},
"request_sha256": "e31c712c0771301f2e4d592bc966448d3143f2660e1b4cab33a91704f19e9370",
"summary_sha256": "eebfaae9c73243c913cb60f1c6f4ec103cac551b4f8d5c5529c5364a6f3465ca"
},
"2K": {
"completion_tokens": 2000,
"decode_tps": 117.36665638338557,
"prefill_seconds": 0.8091103108599782,
"client_total_seconds": 17.857541773468256,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 0.8091103108599782,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 17.04061495512724,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 2000.0
},
"request_sha256": "bca01af4a8338cd6274c1c9e0b281d8e59066ebae2ece3e3a636620066da3a03",
"summary_sha256": "2c37396cb2c1b23f8ae028e9beb941f96a022ffb8a5da29d3363d63003b69672"
},
"10K": {
"completion_tokens": 10000,
"decode_tps": 110.16866681199451,
"prefill_seconds": 0.8098680507391691,
"client_total_seconds": 91.58954442292452,
"strict_pass": null,
"finish_reason": "length",
"raw_counter_deltas": {
"vllm:request_prefill_time_seconds_sum": 0.8098680507391691,
"vllm:request_prefill_time_seconds_count": 1.0,
"vllm:prompt_tokens_total": 431.0,
"vllm:request_decode_time_seconds_sum": 90.76991026010364,
"vllm:request_decode_time_seconds_count": 1.0,
"vllm:generation_tokens_total": 10000.0
},
"request_sha256": "46fd7808353544eb86016bdd1746bafb67def6864a3f5352647b66cb9149f95a",
"summary_sha256": "b44fdcf5c45447a4a3ba6eda737cc6adcf2f183f3569935f415e850a380531e5"
}
},
"retest_strict_TPS": 116.74599429536063,
"retest_strict_prefill_seconds": 0.8083759220317006,
"retest_strict_total_seconds": 25.750953457318246,
"retest_2K_TPS": 117.36665638338557,
"retest_2K_prefill_seconds": 0.8091103108599782,
"retest_2K_total_seconds": 17.857541773468256,
"retest_10K_TPS": 110.16866681199451,
"retest_10K_prefill_seconds": 0.8098680507391691,
"retest_10K_total_seconds": 91.58954442292452,
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 34334547968,
"1": 34314465280,
"2": 34314543104,
"3": 34314469376,
"4": 34334240768,
"5": 34314461184,
"6": 34314457088,
"7": 34318581760
},
"retest_sampled_peak_VRAM_GiB": 31.976539611816406,
"retest_memory_observation_count": 565,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 193,
"text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
"values": [
"8.88"
]
}
],
"available_KV_GiB": [
{
"line": 313,
"text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
"values": [
"19.91"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 315,
"text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
"values": [
"521,696"
]
}
],
"context_capacity": [
{
"line": 316,
"text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
"values": [
"131,072",
"15.82"
]
}
]
},
"retest_logged_model_load_delta_GiB": 8.88,
"retest_historical_logged_load_scope": "Maximum of retained rounded worker model-load deltas; not exact tensor bytes or physical VRAM.",
"retest_KV_budget_GiB": 19.91,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 521696,
"retest_logged_context_capacity_x": 15.82,
"retest_memory_log_source": "historical-recovered/fresh-row-36/result.json"
} | Evidence and limitsstrict-valid Pinned published result tableRecovered original result / scorer replay Original raw evidence {
"strict_tokens": 3719,
"source": "https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro",
"source_revision": "16ae18fc0e44a8a035a3c75b61c24ae8f6a5e041",
"model_revision": "995ad96eacd98c81ed38be0c5b274b04031597b0",
"comparison_limit": "Older Qwen3.6, eight MI50s, different runtime/host; published report, not a new Qwen3.8 run. GGUF model revision denotes pinned source/tokenizer; exact GGUF payload is release-locked.",
"historical_reported_VRAM_capacity_bytes": 34342961152,
"historical_capacity_source": "Pinned release hardware table: 34,342,961,152 visible VRAM bytes per MI50 on both historical lanes; capacity is not measured usage.",
"historical_result_source_copy": "historical-published-results.md",
"historical_source_download": "historical-reproduction-r3.tar.gz",
"strict_verdict_source": "Recovered original uncapped raw response and unchanged scorer replay; published label retained separately.",
"historical_raw_source": "historical-recovered/row-45.json",
"historical_counter_limit": "Raw response tokens and retained summary-delta arithmetic agree. Original raw metric snapshots were not retained; no reconstruction is claimed.",
"historical_memory_reports": {
"row": 45,
"original_host": "10.0.0.30",
"logged_model_load_delta_GiB": 8.88,
"model_load_report_rank": 0,
"logged_model_load_seconds": 116.951902,
"logged_available_KV_GiB": 19.91,
"KV_memory_report_rank": 0,
"logged_GPU_KV_tokens": 521696,
"logged_context_tokens": 131072,
"logged_context_capacity_x": 15.82,
"engine_FP16_TP8_context131072_MTPoff": true,
"source_log_sha256": "5fb63afa234d5088372f07d9632c05c1e4329772bf04a0e3c8fa2fff842b58a7",
"source_archive": "historical-recovered/host10.0.0.30-original-evidence.tar.gz",
"original_launch_path": "/usr/share/ollama/vnext-release-repro/public-staging-20260629/runtime-full-current-proof/vnext-launch-runs/20260630T130329Z-2951956/hf-moe35b-tp8",
"source_lines": {
"load": {
"number": 193,
"text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-30 14:17:32 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 116.951902 seconds"
},
"KV": {
"number": 307,
"text": "(Worker pid=189) (Worker_TP0 pid=189) INFO 06-30 14:18:54 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB"
},
"tokens": {
"number": 315,
"text": "(EngineCore_DP0 pid=171) INFO 06-30 14:18:55 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens"
},
"capacity": {
"number": 316,
"text": "(EngineCore_DP0 pid=171) INFO 06-30 14:18:55 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x"
},
"engine": {
"number": 33,
"text": "(EngineCore_DP0 pid=171) INFO 06-30 14:15:05 [core.py:101] Initializing a V1 LLM engine (v0.0.0+gfx906) with config: model='/opt/local-models/qwen36-35b-a3b-hf', speculative_config=None, tokenizer='/opt/local-models/qwen36-35b-a3b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3.6-35B-A3B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}"
}
},
"limits": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable."
},
"logged_available_KV_GiB": 19.91,
"logged_GPU_KV_tokens": 521696,
"historical_memory_source": "historical-recovered/memory-reports.json",
"historical_memory_limit": "Rounded rank-specific engine load delta and available KV budget, not exact tensor-byte sums, physical VRAM usage or proof that every rank has the same footprint. Logged grouped KV tokens/concurrency do not demonstrate a full-context request. Original physical VRAM usage remains unavailable.",
"historical_binding_source": "historical-recovered/launch-bindings.json",
"historical_fresh_result_source": "historical-recovered/fresh-row-36/result.json",
"retest_strict_tokens": 2911,
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 34334547968,
"1": 34314465280,
"2": 34314543104,
"3": 34314469376,
"4": 34334240768,
"5": 34314461184,
"6": 34314457088,
"7": 34318581760
},
"retest_sampled_peak_VRAM_GiB": 31.976539611816406,
"retest_memory_observation_count": 565,
"retest_historical_memory_scope": "Physical device VRAM sampled every two seconds; may include allocations outside the test container. Sampled maximum is not an exact peak and not logical tensor bytes.",
"retest_historical_memory_log_observations": {
"model_load_delta_GiB": [
{
"line": 193,
"text": "2026-11-09T00:04:18.055901375Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:04:18 [gpu_model_runner.py:4579] Model loading took 8.88 GiB memory and 138.167252 seconds",
"values": [
"8.88"
]
}
],
"available_KV_GiB": [
{
"line": 313,
"text": "2026-11-09T00:05:42.193830378Z (Worker pid=189) (Worker_TP0 pid=189) INFO 11-09 00:05:42 [gpu_worker.py:453] Available KV cache memory: 19.91 GiB",
"values": [
"19.91"
]
}
],
"grouped_GPU_KV_tokens": [
{
"line": 315,
"text": "2026-11-09T00:05:42.197074952Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1314] GPU KV cache size: 521,696 tokens",
"values": [
"521,696"
]
}
],
"context_capacity": [
{
"line": 316,
"text": "2026-11-09T00:05:42.197193083Z (EngineCore_DP0 pid=171) INFO 11-09 00:05:42 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 15.82x",
"values": [
"131,072",
"15.82"
]
}
]
},
"retest_KV_budget_GiB": 19.91,
"retest_KV_budget_scope": "Minimum retained rounded automatic worker KV-availability report; not physical allocation or full-context execution.",
"retest_logged_GPU_KV_tokens": 521696,
"retest_memory_log_source": "historical-recovered/fresh-row-36/result.json",
"sampled_peak_VRAM_GiB": null
} |
| MI50New Qwen3.8 TP8 | 8-bit RTNQwen/Qwen3.8-27B | TP8MTP off | PASS17039 tokens | 59.395 | 56.660 | 15.686 / 49.36710K: 15.894 / 192.394Strict: 15.589 / 329.176 | 2.96 / 28.64packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:2ab2203bcd66050fcae7dbf0b0c8ddbb0958e08da57ce2a0c9d7adc6511e9052Runtime image config: sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp8-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-r2/portable/launch.py --config mi50-w8-tp8-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-w8-tp8-mtp0-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-15 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD128:/dev/dri/renderD128:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \ --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \ --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \ --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \ --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \ --user root \ --workdir /tmp \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \ -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \ -e MI50_RTN_TP=8 \ -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e HF_HUB_OFFLINE=1 \ -e MI50_RTN_BITS=8 \ -e TRANSFORMERS_OFFLINE=1 \ -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp8-mtp0/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp8-mtp0/cache/.triton,dst=/root/.triton \ --mount type=bind,src=/usr/share/ollama/vnext-offload/models/Qwen3.8-27B-1d4bf0f2ff60,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp8-mtp0/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp8-mtp0,dst=/runwork \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w8-tp8-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7Historical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w8-tp8-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-r2/portable/launch.py --config mi50-w8-tp8-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 58.721 · eligible10K 55.999 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 11995,
"content_chars": 19449,
"content_has_think_close": false,
"elapsed_seconds": 231.85842335037887,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T22:31:49Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.012737645767629147,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 19449,
"qwen_answer_text_sha256": "dbbbe18db3bceb6753e547e32d4d94d1551fdf9990d6039eec034a61a9913764",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 19449,
"qwen_visible_text_sha256": "dbbbe18db3bceb6753e547e32d4d94d1551fdf9990d6039eec034a61a9913764",
"raw_has_think_close": false,
"reasoning_chars": 37407,
"text_chars": 56857,
"text_sha256": "39538347312b140fcaf3edd5d34f8fff007518c4ba34cdfe7ceb6d41df4b81fb",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 216.69863117672503,
"vllm_decode_tps": 55.35337226111813,
"vllm_generation_tokens_delta": 11995.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.147749536670744,
"vllm_prefill_tps": 31.22576055637361,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 51.73415667488364
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 48.91990517452359,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T22:32:38Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.008235195651650429,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 9537,
"text_chars": 9537,
"text_sha256": "06336e353aafc8293a9164528c60db07e93500459fdef68e30cfa9d2a6a86cae",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 34.05907580256462,
"vllm_decode_tps": 58.72149942041004,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.851454346440732,
"vllm_prefill_tps": 31.848732721139744,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 40.88315365422164
},
"10K": {
"completion_tokens": 10000,
"content_chars": 9817,
"content_has_think_close": false,
"elapsed_seconds": 193.18140136823058,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T22:35:52Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.008163412101566792,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 9817,
"qwen_answer_text_sha256": "0c8abe5c6cf14820798890db3fbdc8867f8704107e61298d5fe5f1455d634785",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length"
],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 9817,
"qwen_visible_text_sha256": "0c8abe5c6cf14820798890db3fbdc8867f8704107e61298d5fe5f1455d634785",
"raw_has_think_close": false,
"reasoning_chars": 37407,
"text_chars": 47225,
"text_sha256": "1f9bb7602a5e39f2106774666188a115847e2d11885444f49d21544ae12b5e62",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 178.5761641394347,
"vllm_decode_tps": 55.99851496525518,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 14.595634530298412,
"vllm_prefill_tps": 32.406950106767944,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 51.764817571328265
}
}Earlier exported-image retest · separate evidenceRetest completedVerification · Raw retestPASS17,039 natural tokens2K 59.39510K 56.660 · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T16:44:04.393426541Z",
"FinishedAt": "2026-10-26T17:08:54.166149865Z"
},
"devices_after": [
{
"card": 0,
"render": 128,
"pci": "0000:06:00.0",
"actual_card": "card1",
"busy_percent": 0,
"vram_bytes": 17765081088,
"dpm": "auto"
},
{
"card": 1,
"render": 129,
"pci": "0000:09:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 20133625856,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:45:00.0",
"actual_card": "card3",
"busy_percent": 0,
"vram_bytes": 20133601280,
"dpm": "auto"
},
{
"card": 3,
"render": 131,
"pci": "0000:48:00.0",
"actual_card": "card4",
"busy_percent": 0,
"vram_bytes": 20133044224,
"dpm": "auto"
},
{
"card": 4,
"render": 132,
"pci": "0000:89:00.0",
"actual_card": "card5",
"busy_percent": 0,
"vram_bytes": 20133081088,
"dpm": "auto"
},
{
"card": 5,
"render": 133,
"pci": "0000:8c:00.0",
"actual_card": "card6",
"busy_percent": 0,
"vram_bytes": 20132954112,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:c5:00.0",
"actual_card": "card7",
"busy_percent": 0,
"vram_bytes": 20133105664,
"dpm": "auto"
},
{
"card": 7,
"render": 135,
"pci": "0000:c8:00.0",
"actual_card": "card8",
"busy_percent": 0,
"vram_bytes": 20081528832,
"dpm": "auto"
}
],
"finished": "2026-10-26T17:08:54.294353+00:00"
},
"retest_strict": "PASS",
"retest_official_2K_eligible": true,
"retest_2K_TPS": 59.39478915627555,
"retest_10K_TPS": 56.66006826564521,
"retest_strict_tokens": 17039,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 17039,
"elapsed_seconds": 329.1763630043715,
"qwen_gate_valid": true,
"vllm_decode_seconds_delta": 313.5765574136749,
"vllm_decode_tps": 54.337607825453276,
"vllm_generation_tokens_delta": 17039.0,
"vllm_prefill_seconds_delta": 15.588659266009927,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 49.36692126002163,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 33.67298762081191,
"vllm_decode_tps": 59.39478915627555,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 15.685871358029544,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 192.3944482197985,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 176.49113928200677,
"vllm_decode_tps": 56.66006826564521,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 15.894322982057929,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 15.685871358029544,
"retest_2K_total_seconds": 49.36692126002163,
"retest_10K_prefill_seconds": 15.894322982057929,
"retest_10K_total_seconds": 192.3944482197985,
"retest_strict_prefill_seconds": 15.588659266009927,
"retest_strict_total_seconds": 329.1763630043715,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": "Complete record: chart-details/row-46-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 0, \"fixed\": 0}",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30757179392,
"1": 30703603712,
"2": 30703562752,
"3": 30703595520,
"4": 30703566848,
"5": 30703587328,
"6": 30703579136,
"7": 30653231104
},
"retest_memory_observation_count": 24,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w8-tp8-mtp0/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 6.95,
"retest_logged_model_load_seconds": 17.205276,
"retest_logged_available_KV_GiB": 22.52,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 22.52,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 368800,
"retest_logged_context_capacity_x": 11.14,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-w8-tp8-mtp0",
"image": "sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7",
"container": "c61c4fe370d271c8db3d197080e59e6d5b1f4dacc57b36c94291766d55ea0e1f",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 6.95,
"retest_logged_model_load_seconds": 17.205276,
"retest_logged_available_KV_GiB": 22.52,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 22.52,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 368800,
"retest_logged_context_capacity_x": 11.14,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w8-tp8-mtp0/service-final.log",
"source_log_sha256": "dcee58767022051e10ffee4de33d2fdcc25ce21dcdc0a9f7092717ef867728eb",
"sealed_index_sha256": "21dc9b4b2d45cc245fe333cd4281f9c38eca3601458bec384de9e343d1c62f9a",
"source_lines": {
"model_load": {
"line": 196,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:45:52 [gpu_model_runner.py:4584] Model loading took 6.95 GiB memory and 17.205276 seconds"
},
"automatic_KV_budget": {
"line": 236,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:46:57 [gpu_worker.py:453] Available KV cache memory: 22.52 GiB"
},
"grouped_KV_tokens": {
"line": 243,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:46:57 [kv_cache_utils.py:1314] GPU KV cache size: 368,800 tokens"
},
"context_capacity": {
"line": 244,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:46:57 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.14x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w8-tp8-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.644855499267578
}Complete retest loaded admission | Evidence and limitsTP8 GPU engineering passed; full model test pending {
"retest_strict_tokens": 17039,
"retest_MTP_accounting_by_tier": {},
"strict_tokens": 17039,
"source_revision": "research-tp8-shapes-20261026",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"source": "retest-evidence/mi50-w8-tp8-mtp0/result.json",
"comparison_limit": "One first full-model sample; TP8-specific padding/dispatch compatibility and fresh root-SSD caches. No isolated TP attribution or repeatability. Failed strict makes fixed speeds diagnostic.",
"source_revision_type": "Preparation label, not a Git commit; exact source hashes are in the linked compatibility receipt.",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 30757179392,
"1": 30703603712,
"2": 30703562752,
"3": 30703595520,
"4": 30703566848,
"5": 30703587328,
"6": 30703579136,
"7": 30653231104
},
"retest_memory_observation_count": 24,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w8-tp8-mtp0/runtime-observations.jsonl",
"sampled_peak_VRAM_bytes_by_PCI": {
"0000:06:00.0": 30757179392,
"0000:09:00.0": 30703603712,
"0000:45:00.0": 30703562752,
"0000:48:00.0": 30703595520,
"0000:89:00.0": 30703566848,
"0000:8c:00.0": 30703587328,
"0000:c5:00.0": 30703579136,
"0000:c8:00.0": 30653231104
},
"retest_logged_available_KV_GiB": 22.52,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 22.52,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 368800,
"retest_memory_log_report": {
"profile": "mi50-w8-tp8-mtp0",
"image": "sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7",
"container": "c61c4fe370d271c8db3d197080e59e6d5b1f4dacc57b36c94291766d55ea0e1f",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 6.95,
"retest_logged_model_load_seconds": 17.205276,
"retest_logged_available_KV_GiB": 22.52,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 22.52,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 368800,
"retest_logged_context_capacity_x": 11.14,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w8-tp8-mtp0/service-final.log",
"source_log_sha256": "dcee58767022051e10ffee4de33d2fdcc25ce21dcdc0a9f7092717ef867728eb",
"sealed_index_sha256": "21dc9b4b2d45cc245fe333cd4281f9c38eca3601458bec384de9e343d1c62f9a",
"source_lines": {
"model_load": {
"line": 196,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:45:52 [gpu_model_runner.py:4584] Model loading took 6.95 GiB memory and 17.205276 seconds"
},
"automatic_KV_budget": {
"line": 236,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:46:57 [gpu_worker.py:453] Available KV cache memory: 22.52 GiB"
},
"grouped_KV_tokens": {
"line": 243,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:46:57 [kv_cache_utils.py:1314] GPU KV cache size: 368,800 tokens"
},
"context_capacity": {
"line": 244,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:46:57 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.14x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w8-tp8-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 28.644855499267578,
"sampled_peak_VRAM_GiB": 28.644855499267578
} |
| MI50New Qwen3.8 TP8 | 4-bit RTNQwen/Qwen3.8-27B | TP8MTP off | FAIL14034 tokens | 66.438* | 62.951 | 15.101 / 45.21210K: 15.105 / 173.970Strict: 14.373 / 243.245 | 1.48 / 27.28packed target projections / VRAM GiB | Docker Hub · pinned runtimeImmutable docker pulldocker pull joe2gaan/localaiservers@sha256:2ab2203bcd66050fcae7dbf0b0c8ddbb0958e08da57ce2a0c9d7adc6511e9052Runtime image config: sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7 | Serve on your host · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp8-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Verifies model hashes, exact registry image, selected GPUs and peer access. # Writes the resolved full Docker argv to $RUN_DIR/docker-argv.json and docker-run.sh. python3 launch-tools-r2/portable/launch.py --config mi50-w4-tp8-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --serveDownload serving scriptEdit only the documented host choices. The launcher records the exact resolved Docker command. Retained lab Docker command · not portabledocker run -d \ --name serve-mi50-w4-tp8-mtp0-20261028 \ --network host \ --ipc host \ --cpuset-cpus 0-15 \ --security-opt label=disable \ --shm-size 67108864 \ --device /dev/kfd:/dev/kfd:rwm \ --device /dev/dri/renderD128:/dev/dri/renderD128:rwm \ --device /dev/dri/renderD129:/dev/dri/renderD129:rwm \ --device /dev/dri/renderD130:/dev/dri/renderD130:rwm \ --device /dev/dri/renderD131:/dev/dri/renderD131:rwm \ --device /dev/dri/renderD132:/dev/dri/renderD132:rwm \ --device /dev/dri/renderD133:/dev/dri/renderD133:rwm \ --device /dev/dri/renderD134:/dev/dri/renderD134:rwm \ --device /dev/dri/renderD135:/dev/dri/renderD135:rwm \ --user root \ --workdir /tmp \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e VLLM_GFX906_PACKED_QKVZ_CONV=1 \ -e VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE=1 \ -e MI50_RTN_TP=8 \ -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e HF_HUB_OFFLINE=1 \ -e MI50_RTN_BITS=4 \ -e TRANSFORMERS_OFFLINE=1 \ -e VLLM_GFX906_PERSISTENT_AR_PREINIT_ON_GRAPH_CAPTURE=0 \ -e PATH=/opt/venv/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ -e ROCM_VERSION=7.2.1 \ -e AMDGPU_VERSION=not-installed \ -e PYTORCH_INDEX_URL=https://download.pytorch.org/whl/rocm7.2 \ -e TORCH_VERSION=2.11.0+rocm7.2 \ -e TORCHVISION_VERSION=0.26.0+rocm7.2 \ -e TORCHAUDIO_VERSION=2.11.0+rocm7.2 \ -e PYTORCH_TRITON_ROCM_VERSION=3.6.0 \ -e AMDSMI_VERSION=7.0.2 \ -e ROCM72_EXPERIMENT_SKIP_DPKG_LOCKS=1 \ -e VLLM_COMMIT=6a52668c14dc9cb94a270d94bfec735fe627ed0c \ -e VLLM_PACKAGE_VERSION=0.0.0+gfx906 \ -e VLLM_VERSION_OVERRIDE=0.0.0+gfx906 \ -e SOURCE_DATE_EPOCH=1764000000 \ -e VLLM_TARGET_DEVICE=rocm \ -e PYTORCH_ROCM_ARCH=gfx906 \ -e GPU_ARCHS=gfx906 \ -e AMDGPU_TARGETS=gfx906 \ -e CMAKE_HIP_ARCHITECTURES=gfx906 \ -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \ -e FLASH_ATTENTION_TRITON_AMD_REF=TRUE \ -e LD_LIBRARY_PATH=/opt/gfx906:/rccl-overlay/install/lib:/opt/rocm/lib:/opt/rocm/lib64:/usr/local/lib: \ -e PYTHONPATH=/opt/qwen36-python \ -e PYTHONUNBUFFERED=1 \ -e PYTHONHASHSEED=0 \ -e PIP_NO_CACHE_DIR=1 \ -e PIP_NO_COMPILE=1 \ -e PIP_DISABLE_PIP_VERSION_CHECK=1 \ -e PYTHONDONTWRITEBYTECODE=1 \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -e TZ=Etc/UTC \ -e VLLM_PATCH_BUNDLE=/opt/vllm_patch_bundle \ -e V756F105_CUSTOM_AR_LIBRARY=/opt/localaiservers/v756f105r3/lib/v756f105r3_custom_ar.so \ -e VLLM_GFX906_F105R3_M1_STAGED_AR=1 \ -e VLLM_GFX906_FORCE_PCIE_CUSTOM_AR=1 \ -e VLLM_GFX906_POSTED_GEMMA=1 \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp8-mtp0/cache/vllm,dst=/root/.cache/vllm \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp8-mtp0/cache/.triton,dst=/root/.triton \ --mount type=bind,src=/usr/share/ollama/vnext-offload/models/Qwen3.8-27B-1d4bf0f2ff60,dst=/opt/local-models/Qwen3.8-27B,readonly \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp8-mtp0/cache/huggingface,dst=/root/.cache/huggingface \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp8-mtp0,dst=/runwork \ --mount type=bind,src=/home/ai/qwen38-serving-20261028/mi50-w4-tp8-mtp0/cache/torchinductor,dst=/tmp/torchinductor_root \ --entrypoint /opt/vnext/profile/container_entrypoint.sh \ sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7Historical host paths and device addresses are retained as evidence. Use the serving command above. Pinned model acquisition · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint python3 launch-tools-r2/model-acquisition/acquire_model.py qwen38-dense "$MODEL_DIR" | Run the complete benchmark · copy#!/usr/bin/env bash set -euo pipefail # Run in the downloaded release directory on your Linux Docker host. sha256sum -c launch-tools-r2.tar.gz.sha256 mkdir -p launch-tools-r2 tar -xzf launch-tools-r2.tar.gz -C launch-tools-r2 # Edit these host choices; numerical/runtime settings remain pinned. MODEL_DIR=/absolute/path/to/pinned-Qwen3.8-checkpoint RUN_DIR=/absolute/path/to/new-runs/mi50-w4-tp8-mtp0 GPU_RENDER_NODES=renderD128,renderD129,renderD130,renderD131,renderD132,renderD133,renderD134,renderD135 # Starts a fresh cohort, runs the original ladder, and stops only that new cohort. python3 launch-tools-r2/portable/launch.py --config mi50-w4-tp8-mtp0 --model "$MODEL_DIR" --work-dir "$RUN_DIR" --renders "$GPU_RENDER_NODES" --benchmarkDownload benchmark script Use the archive named in each command. Fresh run/cache directory required; benchmark mode stops its own service. | All four commands verifiedTest host: 10.0.0.20acquire: verified · pull: verified · serve: verified · benchmark: verifiedStrict PASS2K 66.644 · eligible10K 63.164 · diagnosticFresh tier results and counters{
"strict": {
"completion_tokens": 10166,
"content_chars": 6892,
"content_has_think_close": false,
"elapsed_seconds": 176.45749805495143,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "stop",
"finished_at_utc": "2026-11-09T23:04:03Z",
"force_length": false,
"has_reasoning_field": true,
"max_tokens": null,
"metrics_settle_elapsed_seconds": 0.008709434419870377,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 6892,
"qwen_answer_text_sha256": "a53b93bd66e91cf5575d86b684573e1e556920dbc610ff1a106bf3d18e67e57d",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [],
"qwen_gate_parser_split_answer": true,
"qwen_gate_valid": true,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 6892,
"qwen_visible_text_sha256": "a53b93bd66e91cf5575d86b684573e1e556920dbc610ff1a106bf3d18e67e57d",
"raw_has_think_close": false,
"reasoning_chars": 37872,
"text_chars": 44765,
"text_sha256": "673608635cffe4379f984e99f0e44039895dc4e93c9285fe75c20544c2e233a1",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 160.9127823319286,
"vllm_decode_tps": 63.177081725115656,
"vllm_generation_tokens_delta": 10166.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.530331411398947,
"vllm_prefill_tps": 30.456529707590633,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 57.61160682916495
},
"2K": {
"completion_tokens": 2000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 45.10825101658702,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T23:04:48Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 2000,
"metrics_settle_elapsed_seconds": 0.006011459976434708,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 10262,
"text_chars": 10262,
"text_sha256": "baa1fafcb2abb3014a4dae9a3ac255ba88f4bb44538598a342507151ca6eac69",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 30.0101278424263,
"vllm_decode_tps": 66.64416794561383,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 15.089444040320814,
"vllm_prefill_tps": 31.346416656311987,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 44.337786434339215
},
"10K": {
"completion_tokens": 10000,
"content_chars": 0,
"content_has_think_close": false,
"elapsed_seconds": 174.96906033623964,
"endpoint": "http://127.0.0.1:19080/v1",
"finish_reason": "length",
"finished_at_utc": "2026-11-09T23:07:43Z",
"force_length": true,
"has_reasoning_field": true,
"max_tokens": 10000,
"metrics_settle_elapsed_seconds": 0.00648938212543726,
"metrics_settle_timeout_seconds": 30.0,
"metrics_settled": true,
"metrics_url": "http://127.0.0.1:8001/metrics",
"model": "Qwen/Qwen3.8-27B",
"prompt_repeat": 32,
"qwen_answer_text_chars": 0,
"qwen_answer_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"qwen_gate_close_in_content": false,
"qwen_gate_close_in_raw": false,
"qwen_gate_invalid_reasons": [
"client_max_tokens_cap",
"finish_reason_length",
"missing_think_close_or_parser_answer"
],
"qwen_gate_parser_split_answer": false,
"qwen_gate_valid": false,
"qwen_visible_parse_complete": true,
"qwen_visible_text_chars": 0,
"qwen_visible_text_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"raw_has_think_close": false,
"reasoning_chars": 18023,
"text_chars": 18023,
"text_sha256": "9552134caca7dee33fcf050196c55d90b694ce6c7dfe160f2d8697da6cf218bb",
"vllm_decode_request_count_delta": 1.0,
"vllm_decode_seconds_delta": 158.31779999658465,
"vllm_decode_tps": 63.164091468020196,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_request_count_delta": 1.0,
"vllm_prefill_seconds_delta": 16.642732575535774,
"vllm_prefill_tps": 28.42081358053504,
"vllm_prompt_tokens_delta": 473.0,
"wall_completion_tps": 57.15296167666963
}
}Earlier exported-image retest · separate evidenceRetest completed; strict failed; fixed speeds diagnostic onlyVerification · Raw retestFAIL14,034 natural tokens2K 66.438*10K 62.951* · diagnostic Retest latency / counters / KV{
"retest_full_ladder_completed": true,
"retest_closure": {
"stop_command_exit": 0,
"container_state": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-26T16:21:11.087699539Z",
"FinishedAt": "2026-10-26T16:43:25.499908697Z"
},
"devices_after": [
{
"card": 0,
"render": 128,
"pci": "0000:06:00.0",
"actual_card": "card1",
"busy_percent": 0,
"vram_bytes": 15957090304,
"dpm": "auto"
},
{
"card": 1,
"render": 129,
"pci": "0000:09:00.0",
"actual_card": "card2",
"busy_percent": 0,
"vram_bytes": 17210978304,
"dpm": "auto"
},
{
"card": 2,
"render": 130,
"pci": "0000:45:00.0",
"actual_card": "card3",
"busy_percent": 0,
"vram_bytes": 17213030400,
"dpm": "auto"
},
{
"card": 3,
"render": 131,
"pci": "0000:48:00.0",
"actual_card": "card4",
"busy_percent": 0,
"vram_bytes": 17212928000,
"dpm": "auto"
},
{
"card": 4,
"render": 132,
"pci": "0000:89:00.0",
"actual_card": "card5",
"busy_percent": 0,
"vram_bytes": 18720759808,
"dpm": "auto"
},
{
"card": 5,
"render": 133,
"pci": "0000:8c:00.0",
"actual_card": "card6",
"busy_percent": 0,
"vram_bytes": 18721447936,
"dpm": "auto"
},
{
"card": 6,
"render": 134,
"pci": "0000:c5:00.0",
"actual_card": "card7",
"busy_percent": 0,
"vram_bytes": 17212538880,
"dpm": "auto"
},
{
"card": 7,
"render": 135,
"pci": "0000:c8:00.0",
"actual_card": "card8",
"busy_percent": 0,
"vram_bytes": 17162596352,
"dpm": "auto"
}
],
"finished": "2026-10-26T16:43:25.625278+00:00"
},
"retest_strict": "FAIL",
"retest_official_2K_eligible": false,
"retest_2K_TPS": 66.43828553575366,
"retest_10K_TPS": 62.950553642009105,
"retest_strict_tokens": 14034,
"retest_tier_metrics": {
"strict": {
"completion_tokens": 14034,
"elapsed_seconds": 243.24463842017576,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 228.86323131993413,
"vllm_decode_tps": 61.32046602270283,
"vllm_generation_tokens_delta": 14034.0,
"vllm_prefill_seconds_delta": 14.372599754948169,
"vllm_prompt_tokens_delta": 473.0
},
"2K": {
"completion_tokens": 2000,
"elapsed_seconds": 45.21246994892135,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 30.103124785237014,
"vllm_decode_tps": 66.43828553575366,
"vllm_generation_tokens_delta": 2000.0,
"vllm_prefill_seconds_delta": 15.10064309882,
"vllm_prompt_tokens_delta": 473.0
},
"10K": {
"completion_tokens": 10000,
"elapsed_seconds": 173.97005397314206,
"qwen_gate_valid": false,
"vllm_decode_seconds_delta": 158.8548379871063,
"vllm_decode_tps": 62.950553642009105,
"vllm_generation_tokens_delta": 10000.0,
"vllm_prefill_seconds_delta": 15.105305118951946,
"vllm_prompt_tokens_delta": 473.0
}
},
"retest_2K_prefill_seconds": 15.10064309882,
"retest_2K_total_seconds": 45.21246994892135,
"retest_10K_prefill_seconds": 15.105305118951946,
"retest_10K_total_seconds": 173.97005397314206,
"retest_strict_prefill_seconds": 14.372599754948169,
"retest_strict_total_seconds": 243.24463842017576,
"retest_MTP_accounting_by_tier": {},
"retest_loaded_admission": "Complete record: chart-details/row-47-retest-loaded-admission.json",
"retest_audit_exit_codes": "{\"strict\": 1, \"fixed\": 1}",
"retest_diagnostic_counter_audit": "retest-evidence/mi50-w4-tp8-mtp0/local-diagnostic-fixed-audit.json",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 29295050752,
"1": 29241323520,
"2": 29241430016,
"3": 29241384960,
"4": 29241339904,
"5": 29241348096,
"6": 29241323520,
"7": 29191086080
},
"retest_memory_observation_count": 17,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w4-tp8-mtp0/runtime-observations.jsonl",
"retest_logged_model_load_delta_GiB": 6.94,
"retest_logged_model_load_seconds": 16.603594,
"retest_logged_available_KV_GiB": 22.54,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 22.54,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 368800,
"retest_logged_context_capacity_x": 11.15,
"retest_logged_context_tokens": 131072,
"retest_memory_log_report": {
"profile": "mi50-w4-tp8-mtp0",
"image": "sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7",
"container": "5e1221480e5b4d81ce3a4f5f513c0a5aff435c64a9b4f7378fa2e683791bd435",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 6.94,
"retest_logged_model_load_seconds": 16.603594,
"retest_logged_available_KV_GiB": 22.54,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 22.54,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 368800,
"retest_logged_context_capacity_x": 11.15,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w4-tp8-mtp0/service-final.log",
"source_log_sha256": "3d3066110fccc8799355e5183fb3c1bfc2e9bb090c058a66fc4a945252e9f2fe",
"sealed_index_sha256": "65edaba46ff0169318729d56558f624b0a3c954f850ea69f7848b5e0aa401997",
"source_lines": {
"model_load": {
"line": 196,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:22:57 [gpu_model_runner.py:4584] Model loading took 6.94 GiB memory and 16.603594 seconds"
},
"automatic_KV_budget": {
"line": 237,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:24:03 [gpu_worker.py:453] Available KV cache memory: 22.54 GiB"
},
"grouped_KV_tokens": {
"line": 243,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:24:03 [kv_cache_utils.py:1314] GPU KV cache size: 368,800 tokens"
},
"context_capacity": {
"line": 244,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:24:03 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.15x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w4-tp8-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 27.28314208984375
}Complete retest loaded admission | Evidence and limitsTP8 GPU engineering passed; full model test pending {
"retest_strict_tokens": 14034,
"retest_MTP_accounting_by_tier": {},
"retest_diagnostic_counter_audit": "retest-evidence/mi50-w4-tp8-mtp0/local-diagnostic-fixed-audit.json",
"strict_tokens": 14034,
"source_revision": "research-tp8-shapes-20261026",
"model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
"source": "retest-evidence/mi50-w4-tp8-mtp0/result.json",
"comparison_limit": "One first full-model sample; TP8-specific padding/dispatch compatibility and fresh root-SSD caches. No isolated TP attribution or repeatability. Failed strict makes fixed speeds diagnostic.",
"source_revision_type": "Preparation label, not a Git commit; exact source hashes are in the linked compatibility receipt.",
"retest_sampled_peak_VRAM_bytes_by_rank": {
"0": 29295050752,
"1": 29241323520,
"2": 29241430016,
"3": 29241384960,
"4": 29241339904,
"5": 29241348096,
"6": 29241323520,
"7": 29191086080
},
"retest_memory_observation_count": 17,
"retest_memory_scope": "Sampled physical VRAM of exact imported-image cohort at approximately one-minute intervals; may miss peaks.",
"retest_memory_observation_source": "retest-evidence/mi50-w4-tp8-mtp0/runtime-observations.jsonl",
"sampled_peak_VRAM_bytes_by_PCI": {
"0000:06:00.0": 29295050752,
"0000:09:00.0": 29241323520,
"0000:45:00.0": 29241430016,
"0000:48:00.0": 29241384960,
"0000:89:00.0": 29241339904,
"0000:8c:00.0": 29241348096,
"0000:c5:00.0": 29241323520,
"0000:c8:00.0": 29191086080
},
"retest_logged_available_KV_GiB": 22.54,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 22.54,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 368800,
"retest_memory_log_report": {
"profile": "mi50-w4-tp8-mtp0",
"image": "sha256:89bf56371852ba69e2dc3375fb1f667fe11f2a34a04537beb6ab75da4173c5c7",
"container": "5e1221480e5b4d81ce3a4f5f513c0a5aff435c64a9b4f7378fa2e683791bd435",
"model_load_report_rank": 0,
"retest_logged_model_load_delta_GiB": 6.94,
"retest_logged_model_load_seconds": 16.603594,
"retest_logged_available_KV_GiB": 22.54,
"retest_explicit_KV_budget_bytes": null,
"retest_KV_budget_GiB": 22.54,
"retest_KV_budget_scope": "Automatic available-memory report; rounded",
"retest_logged_GPU_KV_tokens": 368800,
"retest_logged_context_capacity_x": 11.15,
"retest_logged_context_tokens": 131072,
"source_log": "retest-evidence/mi50-w4-tp8-mtp0/service-final.log",
"source_log_sha256": "3d3066110fccc8799355e5183fb3c1bfc2e9bb090c058a66fc4a945252e9f2fe",
"sealed_index_sha256": "65edaba46ff0169318729d56558f624b0a3c954f850ea69f7848b5e0aa401997",
"source_lines": {
"model_load": {
"line": 196,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:22:57 [gpu_model_runner.py:4584] Model loading took 6.94 GiB memory and 16.603594 seconds"
},
"automatic_KV_budget": {
"line": 237,
"text": "(Worker pid=196) (Worker_TP0 pid=196) INFO 10-26 16:24:03 [gpu_worker.py:453] Available KV cache memory: 22.54 GiB"
},
"grouped_KV_tokens": {
"line": 243,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:24:03 [kv_cache_utils.py:1314] GPU KV cache size: 368,800 tokens"
},
"context_capacity": {
"line": 244,
"text": "(EngineCore_DP0 pid=178) INFO 10-26 16:24:03 [kv_cache_utils.py:1319] Maximum concurrency for 131,072 tokens per request: 11.15x"
}
},
"limits": "Model-load delta and automatic available-KV values are rounded engine reports, not measured physical VRAM or exact tensor-byte totals. Explicit budgets are configuration values confirmed in the loaded engine, not profiled availability. Grouped KV tokens and concurrency use the engine hybrid-cache accounting; they are not proof of a completed full-context request or an exact request ceiling."
},
"retest_memory_log_source": "retest-evidence/mi50-w4-tp8-mtp0/memory-log-report.json",
"retest_sampled_peak_VRAM_GiB": 27.28314208984375,
"sampled_peak_VRAM_GiB": 27.28314208984375
} |
| V620Unmeasured TP8 | FP16Qwen/Qwen3.8-27B | TP8MTP off | NOT RUNNot reported | — | — | — / —10K: — / —Strict: — / — | — / —FP16 parameter extents / VRAM GiB | —Image identityNot built | No validated eight-V620 serving configuration | No validated eight-V620 profile available | Hardware unavailable; no result invented | Evidence and limitsEight V620s unavailable on recorded hardware {
"sampled_peak_VRAM_GiB": null
} |
| V620Unmeasured TP8 | 8-bit RTNQwen/Qwen3.8-27B | TP8MTP off | NOT RUNNot reported | — | — | — / —10K: — / —Strict: — / — | — / —packed target projections / VRAM GiB | —Image identityNot built | No validated eight-V620 serving configuration | No validated eight-V620 profile available | Hardware unavailable; no result invented | Evidence and limitsEight V620s unavailable on recorded hardware {
"sampled_peak_VRAM_GiB": null
} |
| V620Unmeasured TP8 | 4-bit RTNQwen/Qwen3.8-27B | TP8MTP off | NOT RUNNot reported | — | — | — / —10K: — / —Strict: — / — | — / —packed target projections / VRAM GiB | —Image identityNot built | No validated eight-V620 serving configuration | No validated eight-V620 profile available | Hardware unavailable; no result invented | Evidence and limitsEight V620s unavailable on recorded hardware {
"sampled_peak_VRAM_GiB": null
} |
Portable commands use your selected devices and a fresh output directory. The checkpoint is read-only and checked against its full manifest. To repeat a command, choose a new output directory. Registry images share common layers; exact image IDs remain distinct.
Memory distinguishes packed projection tensors from sampled physical VRAM; these are not interchangeable. One recorded speed sample per entry does not establish repeatability or isolated TP/MTP causality. Generated: 2026-11-10T11:06:19.167713+00:00.