# ROCm MTP Memory Accounting and Context Optimization Available from: https://store.piffa.net/lm/bug/ This document is supposed to be ingested by a LLM in order to re apply or customize this patch. ## Current Status Native MTP speculative decoding creates an additional llama context alongside the target context. This patch improves accounting for this context, fits fixed multi-GPU layouts per device, makes target pipeline scheduling explicit. The current cumulative production patch is: ```text /home/eaman/llama/bug/latest_rocm_improvement_c6824a9e.patch ``` The standalone mainline MTP Compact Rollback patch is: ```text /home/eaman/llama/bug/mtp_compact_rollback_c6824a9e.patch ``` The cumulative patch contains the original Eaman MTP fitting and pipeline changes, MTP Compact Rollback with opt-in adaptive draft sizing, HIP quantized-KV VEC control, and the ported rdna-boosts blocks 09/10 (meta-buffer compute headroom and K-quant MMVQ VDR decode kernels). HIP VEC forcing now defaults to off; normal backend dispatch is used unless `--hip-fa-force-vec on` is supplied. MTP Compact Rollback is no longer an isolated prototype and its public control is `--spec-mtp-cr-depth`. It provides: 1. Complete MTP context and compute accounting. 2. MTP measurement at the target's fitted context with iterative refinement. 3. Per-device fitting when GPU placement is explicitly fixed. 4. Non-pipeline scheduling for the MTP context. 5. Explicit `--pipeline-parallel auto|on|off` control for the target context. 6. Compact target recurrent rollback depth with checkpoint restore and replay. 7. Exact per-device checkpoint fit reservation and startup-pinned allocation. 8. Opt-in HIP VEC forcing for supported quantized-KV attention shapes. 9. Opt-in adaptive MTP draft sizing with CR-aware acceptance accounting. 10. Ordered MTP multi-ubatch execution to prevent overlapping updates to the shared draft KV cache. 11. K-quant MMVQ VDR decode kernels (Q4_K/Q5_K vdr4, Q6_K vdr2, Q8_0 vdr4 on RDNA3_0/RDNA4) ported from the rdna-boosts series, with the RDNA3_5 MMVQ parameter table split off from RDNA2. 12. Meta-backend compute-container headroom sized for recurrent n_rs_seq snapshot views (16 -> 128). The best previously validated ROCm configuration used F16 MTP KV and disabled target pipeline parallelism: ```text --pipeline-parallel off --cache-type-k-draft f16 --cache-type-v-draft f16 ``` The merged VEC update initially reached **147,712 context tokens** with Q4/Q4 MTP KV on the tested Qwen3.6 27B ROCm system, compared with 121,088 for the previous F16/off build: +26,624 tokens (+21.99%). Startup and short generation succeeded; long-run stability and throughput comparisons remain pending. This is a historical cross-build result, not the current matched-cache comparison described below. --- ## Test System | Component | Configuration | |---|---| | Operating system | Debian Sid | | Target model | ThinkingCap Qwen3.6-27B Q6_K_L | | Model layout | 64 target layers plus 1 NextN/MTP layer | | GPU 0 | AMD Radeon RX 6800, 16 GiB | | GPU 1 | AMD Radeon RX 6700 XT, 12 GiB | | Backend | HIP/ROCm build with Vulkan support | | Split mode | Layer | | Tensor split | 0.6,0.4 | | GPU layers | 99, explicitly fixed | | Target KV cache | K=q8_0, V=q5_1 | | MTP KV cache | K=F16, V=F16 | | Maximum MTP draft length | 3 | | Fit target | 50 MiB per device | | Batch / microbatch | 1024 / 384 | | Parallel slots | 1 | | Pipeline mode | Off for target and MTP | | Base source | build 909 (`7bd8282`) | | Working branch | `eaman` | Model: ```text /home/eaman/lm/models/bottlecapai/ThinkingCap-Qwen3.6-27B-Q6_K_L.ggu.gguf on-line: https://store.piffa.net/lm/bug/llama_scripts/ ``` Current launcher: ```text /home/eaman/launch/dense.sh ``` --- ## Issues Addressed ### MTP owns a separate runtime context Native MTP initializes another `llama_context` for the draft path: ```cpp llama_context * ctx_dft = llama_init_from_model(model_tgt, cparams); ``` This context owns context-dependent memory in addition to the target context: - MTP KV memory; - a GPU compute buffer; - a host compute buffer; - backend-specific temporary workspace. The complete MTP allocation must be included in the memory reservation used by the target fitter. ### MTP memory depends on the fitted context The model is trained for 262,144 tokens, but available VRAM requires a smaller runtime context. Measuring MTP at the trained context overstates its runtime allocation. Measuring it only once at a preliminary fitted value also remains conservative after the target fit changes. The dependency is: ```text target context -> MTP memory -> target reservation -> fitted target context ``` The MTP allocation must be measured at the fitted target context and refined. ### Pipeline scheduling has a high memory cost on this topology Pipeline parallelism created four scheduler copies for the target context and duplicated a large compute allocation on both GPUs. With one active server slot and these asymmetric GPUs, the extra workspace and synchronization did not provide a measured performance advantage. The MTP graph also did not benefit from pipeline scheduling because its useful GPU work was placed on one device. The solution therefore keeps pipeline policy explicit: - `auto` preserves the normal llama.cpp decision; - `on` requests pipeline scheduling and warns if unavailable; - `off` disables it; - MTP remains non-pipeline. ### Fixed GPU placement was fitted with a generic allowance The tested command explicitly fixes placement: ```text -ngl 99 --split-mode layer --tensor-split 0.6,0.4 ``` For this case, reserving memory for possible layer redistribution is unnecessary because the fitter cannot change `n_gpu_layers`. The useful constraint is the memory limit of each individual device. The optimized fitter interpolates a context limit for every selected GPU and uses the lowest result. Automatic GPU-layer placement retains the existing conservative behavior. --- ## Solution Implemented by `sol4.patch` ### 1. Use the fitted target context for runtime MTP File: `common/speculative.cpp` ```cpp if (spec_mtp) { cparams.ctx_type = LLAMA_CONTEXT_TYPE_MTP; cparams.pipeline_parallel_type = LLAMA_PIPELINE_PARALLEL_TYPE_DISABLED; cparams.n_ctx = llama_n_ctx(ctx_tgt); } ``` The runtime MTP context now uses the actual fitted target context instead of reserving context-dependent workspace for the full trained context. ### 2. Measure the complete MTP allocation File: `tools/server/server-context.cpp` ```cpp const size_t bytes = (measure_model_bytes ? dmd[j].model : 0) + dmd[j].context + dmd[j].compute; ``` Both MTP context storage and its independent compute workspace are added to the per-device target reservation. ### 3. Refine the MTP reservation File: `tools/server/server-context.cpp` The server performs two refinement passes: ```text 1. Fit the target with the current MTP reservation. 2. Measure MTP at the newly fitted context. 3. Replace the reservation with the refined per-device values. 4. Repeat once and perform the final target fit. ``` This resolves most of the fitting dependency without an open-ended loop. ### 4. Fit fixed multi-GPU layouts per device File: `common/fit.cpp` When `n_gpu_layers` is explicitly fixed, the fitter calculates the context supported by each GPU after its configured margin, rounds it to the required 256-token boundary, and selects the limiting device. ### 5. Keep MTP non-pipeline Files: ```text common/speculative.cpp src/llama-context.cpp tools/server/server-context.cpp ``` MTP fitting and runtime initialization both use the same non-pipeline scheduler policy. This prevents estimation and runtime allocation from selecting different compute-buffer layouts. ### 6. Add explicit target pipeline control Files: ```text include/llama.h src/llama.cpp common/common.h common/common.cpp common/arg.cpp src/llama-context.cpp ``` The public context parameters and common command-line parser expose: ```text --pipeline-parallel auto|on|off ``` The default is `auto`, so applying the patch does not silently change target scheduling. The optimized ROCm launcher explicitly selects `off`. --- ## ROCm MTP Cache Finding For this model and HIP flash-attention path, quantized MTP KV did not minimize total GPU memory. Q4_0 K and V required a large F16 conversion workspace in the selected ROCm flash-attention implementation. Measured MTP allocations before target pipeline optimization were: | Draft KV | Context | ROCm1 KV | ROCm1 compute | MTP GPU total | |---|---:|---:|---:|---:| | Q4_0 | 99,072 | 108.84 MiB | 519.84 MiB | 628.68 MiB | | F16 | 104,704 | 409.00 MiB | 127.70 MiB | 536.70 MiB | F16 used more persistent KV memory but reduced the conversion workspace enough to save approximately 92 MiB of total MTP GPU memory. F16 is therefore the best validated ROCm MTP cache for this model: ```text --cache-type-k-draft f16 --cache-type-v-draft f16 ``` This is a runtime recommendation, not hard-coded patch behavior. Vulkan and other models must be measured separately. --- ## Final Validation ### Fit sequence With F16 MTP KV and target pipeline mode off: ```text Target-only fit: 121,856 Initial MTP estimate: 616.26 MiB First fitted context: 118,784 Refined MTP estimate: 602.01 MiB Second fitted context: 119,808 Refined MTP estimate: 606.76 MiB Final context: 119,552 ``` The runtime target and MTP contexts both initialized at 119,552 tokens. ### Runtime allocation Target context: ```text ROCm0 KV buffer: 2116.09 MiB ROCm1 KV buffer: 1269.66 MiB ROCm0 compute buffer: 614.84 MiB ROCm1 compute buffer: 614.84 MiB Host compute buffer: 102.84 MiB Scheduler copies: 1 ``` MTP context: ```text ROCm1 KV buffer: 467.00 MiB ROCm1 compute buffer: 138.57 MiB Host compute buffer: 102.58 MiB Scheduler copies: 1 ``` The validation log contained no `cudaMalloc` failure and no scheduler fallback. ### Pipeline auto versus off | Metric | F16 / auto | F16 / off | Change | |---|---:|---:|---:| | Final context | 104,704 | **119,552** | **+14,848 (+14.18%)** | | Target compute per GPU | 806.86 MiB | 614.84 MiB | -192.02 MiB | | Target host compute | 322.87 MiB | 102.84 MiB | -220.03 MiB | | Target scheduler copies | 4 | 1 | -3 | ### Initial performance sample One old-prompt sample produced: | Metric | Comparison run | `sol3`, F16/off | Change | |---|---:|---:|---:| | Prompt processing | 206.31 t/s | 251.09 t/s | +21.71% | | Token generation | 32.49 t/s | 36.05 t/s | +10.96% | | Draft acceptance | 90.640% | 88.349% | -2.291 points | This is encouraging but not a controlled benchmark. The generated lengths, build configuration, and draft-cache configuration were not identical. Stability testing and repeated same-build measurements remain necessary. --- ## Source Files Modified | Source file | Purpose | |---|---| | `common/arg.cpp` | Parse the target pipeline mode | | `common/common.cpp` | Transfer pipeline mode to llama context parameters | | `common/common.h` | Store the common pipeline mode parameter | | `common/fit.cpp` | Fit fixed GPU placement against per-device limits | | `common/speculative.cpp` | Use fitted `n_ctx` and non-pipeline mode for runtime MTP | | `include/llama.h` | Define the public pipeline mode API | | `src/llama-context.cpp` | Apply pipeline policy to scheduler creation | | `src/llama.cpp` | Provide the pipeline mode name helper | | `ggml/src/ggml-cuda/fattn.cu` | Dispatch supported HIP quantized-KV attention shapes to VEC | | `tools/server/server-context.cpp` | Measure and refine complete MTP memory | --- ## Important Artifacts | Artifact | Description | |---|---| | `/home/eaman/llama/bug/sol4.patch` | Current complete source patch | | `/home/eaman/llama/bug/rocm_improvement.patch` | Duplicate copy of `sol4.patch` | | `/home/eaman/llama/bug/logs/pipeline_off_f16.log` | Final F16/off validation and prompt sample, 119,552 context | | `/home/eaman/llama/bug/logs/sol2_f16.log` | F16/auto comparison log, 104,704 context | | `/home/eaman/llama/bug/logs/q6_current_full.log` | Q4_0/auto comparison log, 99,072 context | | `/home/eaman/models/think.sh_sol` | Current optimized ROCm launcher | | `/home/eaman/llama/llama.cpp/build/bin` | Source-tree build output | | `/home/eaman/llama/bin_vulkan` | Deployed HIP and Vulkan build updated by the user | | `/home/eaman/llama/bug/old/possible_improvements.md` | Archived, superseded improvement notes | | `/home/eaman/llama/bug/dual_gpu_context_balancing_guide.md` | Reusable manual dual-GPU placement and tensor-override procedure | Patch base and checksum: ```text Historical base commit: 7bd8282 Historical solution commit: e37aae4 Historical complete patch SHA-256: afc4216b15e833712c9c9228b41a1134d6f703614f6aec334c73e53f2ef46839 Current eaman tip commit: a4e47992 Merged mainline commit: 8144f319 Recovery branch: eaman-pre-latest-8144f31 Current standalone patch SHA-256: dc266cba88936c9e6db596dca897e4b7470c51cc9f3597c93026d967512ed6f5 ``` --- ## Deployment Notes The current build contains both HIP and Vulkan backends: ```text GGML_HIP=ON GGML_VULKAN=ON ``` The complete build output, including all llama and GGML shared libraries, must be deployed together. The new `a4e47992` build is in `/home/eaman/llama/llama.cpp/build/bin`; `/home/eaman/llama/bin_vulkan` was not updated by this refresh. ROCm recommended settings for this model: ```text --pipeline-parallel off --cache-type-k-draft f16 --cache-type-v-draft f16 ``` Vulkan should initially retain `--pipeline-parallel auto` and compare Q4_0 with F16 MTP KV because the ROCm conversion-workspace result does not automatically apply to Vulkan. --- ## Historical d2f8305 IQ4_XS Context Comparison On 2026-08-10, `sol4.patch` was applied with `git apply --3way` to current mainline commit `d2f8305`. The patched and unpatched builds used Release mode, HIP and Vulkan backends, and HIP architectures `gfx1030;gfx1031`. Both builds compiled and linked `llama-server` successfully. The standalone patch rebased directly to this mainline commit is: ```text /home/eaman/llama/bug/latest_rocm_improvement_d2f8305.patch SHA-256: d215611605f0d1a3299613f8e34e4bf65ce7cdbd5e36a1383f2f6d4fdf61c319 ``` It applies cleanly without three-way fallback to commit `d2f8305` and carries header metadata for llama.cpp version 996 and ROCm 7.14/Vulkan validation. The single-GPU IQ4_XS launcher settings were taken from `/home/eaman/models/tester_iq4_single_gpu.sh`: target KV Q5_1/Q5_1, draft KV Q4_0/Q4_0, draft maximum 2, batch 1024, microbatch 128, one slot, fixed 99 GPU layers, and a 60 MiB fit target. Patched runs explicitly used `--pipeline-parallel off`; unpatched mainline used its default scheduling because that option is supplied by the patch. | Backend | Previous stock | Current stock | Previous patched | Current patched | Current patch gain | |---|---:|---:|---:|---:|---:| | ROCm | 19,456 | 22,272 | 76,032 | **76,544** | **+54,272** | | Vulkan | 68,352 | 69,120 | 78,592 | **79,616** | **+10,496** | Every current run completed fitting, initialized the model, and reached the server listening state. The patched results remain close to the previous build: +512 tokens on ROCm and +1,024 on Vulkan. This confirms that context allocation is consistent across the mainline update and that the patch remains necessary, especially for ROCm. ### ROCm Q4_0 versus F16 MTP cache comparison On 2026-08-10, the deployed patched binary and libraries from `/home/eaman/llama/bin_vulkan` were tested on ROCm0 with the single-GPU IQ4_XS launcher settings. The two runs differed only in the MTP draft K/V cache types. Both completed fitting, loaded the model, and reached the server listening state. | MTP draft KV | Fitted context | |---|---:| | Q4_0/Q4_0 | **76,544** | | F16/F16 | 68,864 | Q4_0/Q4_0 gained 7,680 context tokens, or 11.15% relative to F16/F16. On this model and supported HIP VEC shape, quantized MTP KV therefore has a meaningful context-capacity advantage. This startup test does not establish throughput, draft acceptance, delayed-OOM behavior, or long-context stability. ### Current Q6_K_L dual-GPU cache comparison Also on 2026-08-10, `/home/eaman/models/think.sh_test` was reproduced with the deployed patched binary. The command used ROCm0 and ROCm1, fixed layer placement with a 0.6/0.4 tensor split, a 20 MiB fit target, draft maximum 3, and pipeline mode off. Only the two MTP draft-cache types changed. | MTP draft KV | Fitted context | MTP KV on ROCm1 | MTP compute on ROCm1 | Refined MTP total | |---|---:|---:|---:|---:| | F16/F16 | **153,856** | 601.00 MiB | 163.70 MiB | 764.70 MiB | | Q4_0/Q4_0 | **153,856** | 169.03 MiB | 163.96 MiB | 333.00 MiB | Q4_0 reduced the MTP allocation on ROCm1 by 431.70 MiB but produced no fitted context increase. The fixed layout is constrained by the per-device fit and the MTP allocation resides on ROCm1; in this run, the saving became ROCm1 headroom rather than changing the limiting 256-token-rounded context. Thus Q4_0 has no context-capacity advantage over F16 for this Q6 dual-GPU configuration, although it still materially reduces memory allocated on ROCm1. #### Converting the ROCm1 headroom into context Changing the whole-layer split from 0.60/0.40 to 0.59/0.41 moved approximately 300 MiB of model weights to ROCm1 but overshot the balance: fitted context fell from 153,856 to 145,152. Layer placement is too coarse for this topology. A target tensor-buffer override provided finer placement while retaining the 0.60/0.40 layer boundary. Moving the three FFN weights of boundary layer 39 to ROCm1 used: ```text -ot '^blk\.39\.ffn_(up|gate|down)\.weight$=ROCm1' ``` This moved 209.18 MiB of model storage from ROCm0 to ROCm1 and raised the fitted context to **165,632**, a gain of 11,776 tokens (+7.65%) over Q4_0 without the override. Startup completed and the server reached the listening state. This configuration still requires controlled throughput, acceptance, delayed-OOM, and long-context stability validation because fine-grained placement can add cross-device traffic. #### Applying the helper to `think.sh_small` On 2026-08-10, the procedure in `dual_gpu_context_balancing_guide.md` was also applied to `/home/eaman/models/think.sh_small`, using the Q6_K model and deployed ROCm build 996 (`d2f8305`). With the original 0.60/0.40 layer split, F16/F16 and Q4_0/Q4_0 both fitted **189,440** context. Q4_0 nevertheless reduced the refined MTP allocation on ROCm1 from 929.76 MiB to 398.15 MiB, a 531.61 MiB saving that the coarse placement could not convert into context. The bounded placement search produced: | Q4_0 placement | Context | Change | Target graph splits | |---|---:|---:|---:| | Original 0.60/0.40 | 189,440 | baseline | 35 | | Whole-layer 0.59/0.41 | 187,392 | -2,048 | 35 | | Boundary `ffn_up` | 193,280 | +3,840 | 37 | | Boundary FFN triplet | 201,216 | +11,776 | 35 | | FFN triplet plus attention Q/output | **205,312** | **+15,872 (+8.38%)** | 39 | The max-context override is: ```text -ot '^blk\.39\.(ffn_(up|gate|down)|attn_(q|output))\.weight$=ROCm1' ``` It moved 283.01 MiB from ROCm0 to ROCm1 and left the modeled device headroom within approximately 23 MiB. The server reached the listening state, `/health` returned OK, and a short real request completed at 27.09 generated tokens/s with 11/11 MTP draft tokens accepted. This single short request is only a functional smoke test. The FFN-only override remains an attractive lower-complexity profile: it retains the baseline 35 target graph splits while still gaining 11,776 tokens. Comparative throughput, delayed-OOM, and long-context stability remain pending. Logs: ```text logs/iq4_backend_patch_comparison_20260810.log logs/mtp_cache_q4_vs_f16_rocm_20260810.log logs/q6_mtp_cache_q4_vs_f16_rocm_20260810.log logs/q6_rocm_q4_tensor_rebalance_20260810.log logs/q6_k_small_rocm_tensor_balance_20260810.log ``` #### Q8_0/Q8_0 target-KV calibration for `think.sh_test` On 2026-08-11, the same manual placement procedure was repeated for `/home/eaman/models/think.sh_test` after its target V cache changed from Q5_1 to Q8_0. The matched trials used Q8_0/Q8_0 target KV, Q4_0/Q4_0 MTP KV, the 0.60/0.40 layer split, and a 20 MiB fit target. They began with 16,310 MiB free on ROCm0 and 12,248 MiB free on ROCm1. | Placement | Context | Model MiB ROCm0/ROCm1 | Target scheduler splits | |---|---:|---:|---:| | No override | 126,720 | 12,569.81 / 9,094.91 | 3 | | Boundary `ffn_up` | 130,048 | 12,500.09 / 9,164.64 | 5 | | Boundary FFN triplet | **136,448** | 12,360.63 / 9,304.09 | 3 | | FFN triplet plus attention Q/output | 139,008 | 12,279.54 / 9,385.19 | 7 | The selected Q8_0 override remains the boundary FFN triplet: ```text -ot '^blk\.39\.ffn_(up|gate|down)\.weight$=ROCm1' ``` It moved 209.18 MiB from ROCm0 to ROCm1 and gained 9,728 tokens (+7.68%) over the matched baseline while retaining 3 target scheduler graph splits. The larger override gained another 2,560 tokens but raised those splits to 7, so it was not selected without a controlled throughput comparison. A clean repeat of the final launcher, with 32 MiB more initial ROCm0 headroom, fitted 137,984 and reached the listening state. This is startup-capacity evidence only; throughput, acceptance, near-context, delayed-OOM, and long-context testing remain pending. Detailed measurements: `logs/q6_kl_q8_target_tensor_balance_20260811.log`. --- ## TODO and Development Path ### Phase 0: Current stability and performance validation - [x] Build HIP and Vulkan backends successfully. - [x] Initialize the 27B Q6 model with F16 MTP KV and pipeline mode off. - [x] Confirm 119,552 target and MTP context without allocation fallback. - [x] Run an initial real prompt and confirm normal MTP generation. - [x] Exercise the current build with the established old-prompt collection (user-confirmed for build 1180). - [x] Test prompts that grow close to the fitted context limit. - [x] Run several long generations and watch for delayed OOM, corruption, or scheduler errors. - [x] Record repeated prompt-cache and generation runs for prompt speed, generation speed, and acceptance. - [x] Perform a controlled same-build F16 `auto` versus `off` comparison at fixed context and seed. - [x] Smoke-test representative models on Vulkan with pipeline mode `auto`. - [x] Compare Q4_0 and F16 MTP KV on Vulkan before choosing a Vulkan default. The selected production Q8_0/Q8_0 target-KV, Q4_0 MTP, VEC, and FFN-triplet placement profile completed repeated 32K-token prompt runs and accumulated approximately 129K slot tokens without truncation, allocation failure, or logged corruption. Long generations reached approximately 8K output tokens. The final runs used a stabilized GPU clock/voltage profile; an earlier apparent decode failure coincided with excessive junction temperature and is not treated as a patch or context-limit failure. These results establish functional long-context stability for this profile, but not a formal throughput benchmark. The matched baseline-versus-override performance comparison remains open. #### Build 1180 controlled MoE pipeline and Vulkan comparison On 2026-08-20, build 1180 (`5850e07c`) was tested with the dual-GPU Qwen3.6-35B-A3B-UD-Q5_K_S MoE model at fixed 32,768 context, target KV Q8_0/Q8_0, tensor split `0.572,0.428`, MTP draft maximum 4, a fixed 50-token prompt and seed 424242, and 512 generated tokens. ROCm used `GGML_CUDA_DISABLE_GRAPHS=1`; all runs used backend sampling disabled. | Backend | MTP KV | Pipeline | Prompt t/s | Generation t/s | Draft acceptance | |---|---|---|---:|---:|---:| | ROCm | F16/F16 | off | 81.12 | 69.67 | 96.283% (259/269) | | ROCm | F16/F16 | auto | 80.28 | 70.38 | 96.283% (259/269) | | Vulkan | Q4_0/Q4_0 | auto | 159.40 | 41.45 | 97.133% (271/279) | | Vulkan | F16/F16 | auto | 159.47 | 39.00 | 96.198% (253/263) | Verbose startup evidence confirmed that ROCm `auto` enabled pipeline parallelism for the target context while MTP remained disabled. Against `off`, `auto` changed generation by +1.02% and prompt processing by -1.03%, too small for a winner from one matched request. On Vulkan, Q4_0 improved generation by 6.27% over F16 with effectively identical prompt speed and 0.935 percentage points higher acceptance in this sample. Q4_0 is the provisional Vulkan choice for this model, pending repeated runs and a matched fitted-context comparison. Raw logs are `logs/controlled_*_20260820.log`; the fixed request is `logs/controlled_comparison_request_20260820.json`. ### Phase 1: Recurrent-state snapshot memory The design investigation reached a validated boundary on 2026-08-20. An opt-in prototype was implemented and tested on 2026-08-21 in the isolated llama.cpp branch `eaman-rs-snapshot-depth` (deleted 2026-09-06), with its linked worktree at `/home/eaman/llama/llama-rs-snapshot-depth` (removed 2026-09-06). Production branch `eaman` remains unchanged, and omitting the new option preserves the existing `n_rs_seq = draft.n_max` behavior. The dense Q6 target context allocates 598.50 MiB of recurrent state for three rollback snapshots: ```text n_rs_seq = 3 ROCm0 recurrent state = 374.06 MiB ROCm1 recurrent state = 224.44 MiB ``` Planned work: - [x] Compare the active MoE model with `--spec-draft-n-max 4` and 5 for context, acceptance, and tokens/second. - [x] Determine the exact base-state and per-snapshot allocation. - [x] Implement configurable GPU rollback depth with recomputation when rejection exceeds the stored depth on an isolated feature branch. - [ ] Deferred: investigate lower-precision rollback snapshots only if the simpler rollback-depth design is insufficient. - [x] Validate the existing rollback and rejection path while keeping the existing behavior as the default. - [x] Fall back to host storage when an on-device recurrent checkpoint has a fragmented layout. - [x] Add parser and fitter-input coverage for default devices, split-mode none, asymmetric targets, and reduced/full depth. - [x] Validate two concurrent slots with independent on-device checkpoints on ROCm. - [ ] Reduce redundant startup measurement work with a load-scoped scratch-fit cache; this is an efficiency follow-up, not a correctness requirement. - [ ] Complete repeated dense-Q6 and MoE benchmarks plus long-prompt and delayed-OOM testing before considering integration into `eaman`. The recorded full-fit contexts are 203,776 with MTP `n-max 4` and 195,584 with `n-max 5`, an 8,192-token capacity advantage for 4. In `/home/eaman/launch/bench/moe_large`, the initial results use `n-max 4`; later results switch to 5 only where that setting is explicitly labeled. The file records multiple model/configuration stages, so timings across those stage boundaries are not treated as one matched pair. The user concluded that 4 has the better overall tradeoff while 5 provides a small speculative improvement that may be more useful at long context; the active launcher therefore retains 5 for now. #### Build 1180 recurrent-state allocation and rollback validation On 2026-08-20, the single-GPU Qwopus3.5 9B Q6_K MTP model was tested on the RX 6800 with build 1180 (`5850e07c`). Controlled MTP-enabled startup runs changed only `--spec-draft-n-max`: | `n_rs_seq` | Total recurrent state | R | S | |---:|---:|---:|---:| | 0 | 50.25 MiB | 2.25 MiB | 48.00 MiB | | 1 | 100.50 MiB | 4.50 MiB | 96.00 MiB | | 4 | 251.25 MiB | 11.25 MiB | 240.00 MiB | The measured formula is `50.25 MiB * (1 + n_rs_seq)` with zero deviation in all three runs. Raw logs are `logs/qwopus9b_rs_n{0,1,4}_build1180_20260820.log`. The dedicated `test-recurrent-state-rollback` binary reached its final dirty-context case, which means its clean full restore and `PARTIAL_ONLY` recurrent restore comparisons passed. The dirty full checkpoint restore then failed at replay position 6 with a token-0 logit mismatch (`7.30374` versus `6.80103`). This is separate from the server MTP path, which uses `PARTIAL_ONLY` target checkpoints. The failing test log is `logs/qwopus9b_recurrent_rollback_build1180_20260820.log`. The operational server path was validated with a fixed 62-token prompt, greedy sampling, seed 1234, and 128 generated tokens. MTP `n-max 4` generated 181 draft tokens and accepted 81 (44.751%), exercising 100 rejected drafts. Its output was byte-identical to a target-only run with the same target settings. The logs are `logs/qwopus9b_mtp_rejection_n4_build1180_20260820.log` and `logs/qwopus9b_target_only_baseline_build1180_20260820.log`. The minimum operational MTP setting was then tested with the same request. MTP `n-max 1` allocated 100.50 MiB of target recurrent state, generated 72 draft tokens, accepted 54 (75.000%), and exercised 18 rejected drafts. Its 128-token output was also byte-identical to the target-only result. The log is `logs/qwopus9b_mtp_rejection_n1_build1180_20260820.log`. MTP `n-max 0` is valid for allocation-only startup but is not an operational server baseline. Its first request aborted at `GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max)` after both target and draft contexts selected `n_outputs_max = 1`; see `logs/qwopus9b_mtp_rejection_n0_build1180_20260820.log`. Use target-only mode as the zero-draft generation baseline. The existing `n_rs_seq = draft.n_max` behavior is validated for real rejection rollback at both the minimum operational value 1 and the active value 4, and remains the default. A lower independent rollback limit cannot be introduced as a memory-only clamp: recurrent partial rollback currently fails when the requested rollback exceeds `n_rs_seq`. A safe design must either bound the verification batch to the available snapshots or provide a recomputation fallback. #### Isolated Phase 1 prototype (2026-08-21) The feature branch adds `--spec-mtp-rs-depth N` and the environment equivalent `LLAMA_ARG_SPEC_MTP_RS_DEPTH`. The accepted range is `1..--spec-draft-n-max`, independent of command-line argument order. When omitted, MTP retains the full draft maximum; EAGLE3, DFlash, and DSpark continue to request full depth even if combined with MTP. Deep rejection restores the existing pre-verification `PARTIAL_ONLY` checkpoint and replays the previously sampled token, accepted draft prefix, and replacement token. The existing direct recurrent rollback remains the fast path when the rejected suffix fits in the retained depth. The server reports `draft_replay_count` and `draft_replay_n` in response timings and prints an MTP replay summary. Argument tests cover the default, both valid argument orders, zero, negative values, and depth above the draft maximum. The deterministic rollback matrix uses draft maximum 5, depth 1, and rejection after every accepted-prefix length from 0 through 4. On HIP with 20 target layers offloaded, all replacement and continuation logits were bit-identical to full-depth rollback. On Vulkan, every greedy replacement and continuation token matched; the largest logged absolute logit drift was 0.0189342. The executable subsequently reaches the pre-existing unrelated dirty full-checkpoint case; Vulkan reports its known dirty-context mismatch, while the two-context HIP run lacks VRAM for that final 2.7 GiB legacy context. The combined build compiled and linked HIP, Vulkan, `llama-server`, `test-recurrent-state-rollback`, and `test-arg-parser`. The CPU/Vulkan argument test reaches all new cases, then fails only at the existing network download test in the restricted environment. Matched RX 6800 ROCm measurements used the Qwopus3.5 9B Q6_K model, target Q8_0/Q8_0 KV, MTP Q4_0/Q4_0 KV, draft maximum 5, and backend draft sampling disabled: | Run | RS buffer | Generation | Draft accepted | Deep replay | |---|---:|---:|---:|---:| | depth 5, 192 tokens | 301.50 MiB | 64.75 t/s | 140/246 | none | | depth 1, 192 tokens | 100.50 MiB | 54.03 t/s | 129/202 | 20 events / 69 tokens | | depth 5, 512 tokens | 301.50 MiB | 60.24 t/s | 366/718 | none | | depth 1, 512 tokens | 100.50 MiB | 47.08 t/s | 311/616 | 75 events / 231 tokens | The initial host-checkpoint depth-1 implementation saved 201.00 MiB, or 66.7% of the target recurrent-state buffer. The matched 192-token output was byte-identical. The 512-token depth-1 and depth-5 outputs diverged; the depth-5 result was byte-identical on repeat, but a target-only 512-token run also differed from both speculative results. Long-output byte identity is therefore not treated as a sufficient oracle for this existing MTP path. The feature remains opt-in and isolated pending repeated dense/MoE workloads and longer stability testing. Relevant logs use the `logs/phase1_rs_*_20260821` prefix. #### Hardcoded on-device checkpoint proof of concept Feature commit `cccc8c78` changes only recurrent speculative fallback checkpoints to `LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY | LLAMA_STATE_SEQ_FLAGS_ON_DEVICE`; other checkpoint types remain host-backed. This remains intentionally hardcoded for the experiment: there is no additional command-line flag. Feature commit `be535675` subsequently added automatic fit accounting for the lazy device allocation. The deterministic depth-1 rejection matrix passed at every accepted-prefix length on both Vulkan and HIP. HIP replacement and continuation logits remained bit-identical to full-depth rollback. The later pre-existing dirty full-checkpoint case remains separate; Vulkan reaches its known mismatch, while the HIP two-context test lacks VRAM for the final 2.7 GiB legacy context. On the same fixed 192-token RX 6800 9B request, device depth 1 generated identical content with the same 202 drafts, 129 accepted tokens, and 20 replay events / 69 replayed tokens. It improved generation from 54.03 to 60.39 t/s versus the host-checkpoint implementation (+11.76%) and remained 6.74% below the 64.75 t/s full-depth result. The device checkpoint allocated exactly one additional 50.25 MiB state group. Effective target recurrent storage was therefore 150.75 MiB versus 301.50 MiB at depth 5, retaining a 150.75 MiB (50%) saving for this draft-maximum-5 configuration. Logs are `logs/phase1_rs_depth1_device_rocm_9b_trace_20260821.{log,json}`. The fixed-context dense draft-maximum-3 run completed at 136,448 tokens with `--fit off`. It generated 11,761 tokens at 26.50 t/s, accepted 8,009 of 8,733 draft tokens (91.710%), averaged 3.55 accepted tokens, and used 98 replay events / 288 replayed tokens. This is 10.60% faster than the earlier 23.96 t/s host-checkpoint depth-1 run and 5.12% below the 27.93 t/s normal depth-3 baseline. The device checkpoint therefore recovers most of the host-copy slowdown in this long dense workload as well as in the 9B test. The supplied timing record is `logs/phase1_rs_depth1_device_dense_fixed_20260821.log`. For this dense model, one checkpoint group consumes 149.625 MiB across the two GPUs. Depth 1 plus that checkpoint uses 448.875 MiB versus 598.50 MiB at depth 3, retaining a net saving of 149.625 MiB (25% of recurrent allocation). Feature commit `be535675` adds automatic fit accounting for the lazy on-device checkpoint. When MTP uses a recurrent depth below its draft maximum, server startup measures the target context at `n_rs_seq` and `n_rs_seq + 1`, takes only the per-device context-memory delta, and adds that one-group reservation to the fitter margin in the measured model-device order. Device-count/order changes, a missing fit target, decreasing memory, a zero total, or a measurement exception fail model loading instead of permitting a delayed OOM. Vulkan and combined HIP/Vulkan server builds both compiled and linked successfully with `-j 12`. The successful dense `--fit-target 20` run measured and reserved 93.52 MiB on ROCm0 plus 56.11 MiB on ROCm1, 149.62 MiB total, and fitted 149,504 context tokens. Final allocation reached the listening state with 114/124 MiB reported free on ROCm0/ROCm1. A deterministic 192-token request triggered the lazy device checkpoint allocation at the same 93.516/56.109 MiB split and completed without OOM at 23.15 t/s. It accepted 103/112 draft tokens (91.964%) and exercised one deep-rejection replay event / three replayed tokens. The server then shut down cleanly and both GPUs released their allocations. Logs are `logs/phase1_rs_device_fit_dense_success_20260821.{log,json}`; the earlier occupied-port attempt remains `logs/phase1_rs_device_fit_dense_20260821.log` for historical context. #### Fragmented-checkpoint and multi-slot hardening Feature commit `732dd501` converts fragmented on-device recurrent checkpoint saves from an abort into a catchable failure. The common checkpoint wrapper retries with host storage and records the actual storage mode for restore. A forced fragmented whole-cache fixture validates this fallback. Normal positive per-slot recurrent allocation remains contiguous, so the fallback is defensive for unusual, malformed, or future layouts rather than an expected `-np > 1` path. The same commit expands parser and fitter-input tests across fit disabled, default margins, split-mode none, asymmetric targets, reduced/full depth, combined EAGLE3, and dynamic two-device argument order. Vulkan rollback testing still passes every accepted-prefix position from 0 through 4; only the later known dirty full-checkpoint case differs. Clean build 1184 (`732dd501`) compiled and linked `llama-server`, `test-arg-parser`, and the rollback test in both the Vulkan-only and combined HIP/Vulkan configurations. A real build-1184 ROCm `-np 2` run used two concurrent Qwopus3.5 9B requests. Each slot allocated an independent 50.250 MiB device checkpoint. Both completed 192 generated tokens without OOM or host fallback. Slot 0 achieved 35.99 t/s, accepted 118/222 drafts, and replayed 90 tokens in 27 events; slot 1 achieved 28.73 t/s, accepted 91/283 drafts, and replayed 123 tokens in 44 events. The raw capture lacks the original startup fitter section, so 100.50 MiB is the measured total lazy allocation and only an inferred fitter reservation, not a captured reservation line. See `logs/phase1_rs_hardening_np2_rocm_9b_summary_20260821.log` and its referenced raw log/responses. An audit of the extra startup measurements found no safe local pass deletion: the D, D+1, MTP-context, and refinement measurements have distinct inputs. Reducing repeated baseline/minimum-context work requires an explicit load-scoped scratch-fit cache with a complete cache key; the final fit should remain fresh. This optimization is deferred because it affects startup time, not runtime correctness or memory safety. Rebasing is also deferred until the production `eaman` base advances; the feature branch is already based on the current `eaman` tip. #### Exported `a3b1eff` test patches (2026-08-22) Two experimental standalone exports now support controlled testing without changing either source branch: - `latest_rs_cumulative_a3b1eff.patch` applies directly to `a3b1eff` and contains the complete Eaman ROCm/MTP/VEC/MoE work plus recurrent-state rollback through `732dd501`. Clean application produced tracked-tree hash `80b60d13ccc56bd3ea975c54ecb2461dc373bbc0`, exactly matching the feature branch tip. SHA-256: `870cae983d962cb250d9e9b347cd3d51b58bcc12a5b7d07063fee81f3e4876a4`. - `mtp_rs_only_a3b1eff.patch` applies directly to pristine `a3b1eff` and contains only the four-commit recurrent rollback delta (`e04c6d9b`, `cccc8c78`, `be535675`, and `732dd501`). It excludes the cumulative Eaman fitter, pipeline-control, HIP VEC, and MoE scratch-placement changes. Clean application and `git diff --check` passed; CPU/default build 1171 linked `llama-server`, `test-arg-parser`, and `test-recurrent-state-rollback`, and `test-arg-parser` passed. The model-dependent rollback executable and isolated GPU runtime were not rerun. SHA-256: `2aa8149d703d730e57b0b8dd4acef62bbef80e5e0baac00b9d55c7bfeb19c3b8`. Both files are pinned test artifacts for the exact `a3b1eff` base. Any upstream submission should first rebase the RS-only change onto a freshly fetched upstream `master`, then rebuild and repeat the focused GPU/model validation. #### Original deferred-work report and resume point The objective was to keep a useful MTP draft maximum while reducing the GPU memory occupied by target recurrent rollback snapshots. The allocation and correctness questions are resolved, but the implementation is not a small allocator change. Measured facts: - Recurrent memory contains one live state plus `n_rs_seq` rollback snapshots and scales exactly with `1 + n_rs_seq`. - The dense Q6 configuration with `n_rs_seq = 3` uses 598.50 MiB: 374.06 MiB on ROCm0 and 224.44 MiB on ROCm1. One state group is 149.625 MiB total, split as approximately 93.515 MiB on ROCm0 and 56.110 MiB on ROCm1. - Reducing stored depth from 3 to 1 while retaining draft maximum 3 would save approximately 299.25 MiB total. The measured dense-Q6 placement slope estimates approximately 8,704 more context tokens, moving the normal 136,448-137,984 fit to roughly 145,152-146,688. - At draft maximum 5, reducing stored depth from 5 to 1 would save approximately 598.50 MiB total and an estimated 17,408 tokens compared with unmodified `n-max 5`. Relative to the usual unmodified `n-max 3` launcher, the expected final context gain remains approximately 8,704 because unmodified `n-max 5` first spends two additional snapshots. - These are fit estimates, not measured final contexts. Actual fitting is rounded to 256 tokens and varies with initial free VRAM and per-device placement. Correctness boundary: - If five draft tokens are verified together and token 3 is rejected, the correct target state is the state after the two accepted tokens. A depth-1 snapshot cannot restore that state directly from the end of the verification batch. - Clamping the rollback distance to the stored depth is incorrect and can corrupt subsequent output. - Restricting verification batches to the stored depth is simple but imposes a regular speed penalty and removes much of the benefit of larger draft maxima. - The preferred design is a recomputation fallback: retain the current fast path while rejection fits within the stored snapshots; otherwise restore the pre-verification `PARTIAL_ONLY` checkpoint and replay the accepted prefix. Accepted output tokens are preserved, but their recurrent state is recomputed. - The expected average slowdown may be small at the observed 95%-99% production acceptance, but this must be measured. Deep rejection frequency, not only aggregate acceptance, determines the fallback cost. Relevant source map: - `common/common.h`: `need_n_rs_seq()` currently returns `draft.n_max` for MTP and related draft implementations. - `src/llama-memory-recurrent.cpp`: tensor widening uses `1 + n_rs_seq`; `seq_rm()` rejects rollback distances greater than `n_rs_seq`. - `tools/server/server-context.cpp`: the hardcoded proof of concept uses `PARTIAL_ONLY | ON_DEVICE` for recurrent speculative target checkpoints around verification and rollback. - `tests/test-recurrent-state-rollback.cpp`: existing full, partial, and dirty-context checkpoint coverage. Known separate edge cases: - MTP `n-max 0` is not a useful operational setting. It starts and measures allocations but aborts on its first server request. Normal MTP use starts at 1; target-only mode is the zero-draft baseline. - The dedicated rollback test passes clean full restore and the server-relevant `PARTIAL_ONLY` restore, then fails when a full checkpoint is loaded over a dirty hybrid context. Do not use that full dirty-restore path as the proposed fallback without resolving it. The current server MTP path does not use it. Resume checklist: 1. Define an opt-in rollback-depth parameter with range `1..draft.n_max`; preserve `draft.n_max` as the default so existing behavior does not change. 2. Specify the recomputation state machine before editing code: checkpoint position, accepted-prefix replay, replacement token handling, target attention-cache rollback, draft-context synchronization, and statistics accounting. 3. Add deterministic tests for `n-max 5` with depth 1 and rejection at every draft position. Compare tokens and logits against full-depth behavior across repeated speculative rounds and prompt-checkpoint restoration. 4. Build HIP and Vulkan, then repeat the 9B single-GPU rejection test before testing the dense Q6 dual-GPU fit and throughput. 5. Measure context, tokens per second, deep-rejection frequency, replay count, and acceptance on matched dense Q6 and MoE workloads. Keep the feature opt-in unless the memory gain clearly outweighs the measured slowdown. The prototype portion is complete. Focused dense/MoE performance validation and long-context confidence testing remain before integration. ### Phase 2: MTP placement and backend overhead - [x] Measure fine-grained target placement for the boundary FFN tensors; the FFN-triplet override is the preferred lower-complexity candidate because it preserves the baseline graph-split count. - [x] Retain the FFN-triplet override as the production choice. The attention-inclusive override adds only 2,560 context tokens while increasing target graph splits from 3 to 7, so no further comparison is planned unless priorities change. - [ ] Measure free VRAM and MTP tensor placement on both GPUs after long runs. - [ ] Investigate whether selected NextN tensors or MTP workspace can use the less constrained GPU. - [x] Test `--no-spec-draft-backend-sampling` to quantify the ROCm sampling fallback overhead. - [ ] Separate context-capacity changes from performance-only backend changes. #### Build 1180 ROCm draft-backend sampling comparison On 2026-08-20, the single-GPU Qwopus3.5 9B Q6_K MTP model was tested on the RX 6800 with explicit `--spec-draft-backend-sampling` and `--no-spec-draft-backend-sampling`. Each mode received one 64-token warm-up followed by the same three 512-token requests with seeds 4101-4103, MTP `n-max 4`, Q4_0/Q4_0 draft KV, and identical temperature, top-k, top-p, and penalty settings. | Draft backend sampling | Mean generation t/s | Median generation t/s | Draft acceptance | |---|---:|---:|---:| | enabled | 66.98 | 65.43 | 95.187% (890/935) | | disabled | 66.13 | 65.22 | 95.187% (890/935) | Disabling the option changed mean throughput by -1.27% and median throughput by -0.33%, which is below run-to-run noise. All three matched outputs were byte-identical. With backend sampling requested, ROCm reported that `ROCm0` does not support the `TOP_K` operation required by the draft top-k sampler; disabling the option removed that warning and the extra sampler graph reservation, but produced no measurable generation benefit in this workload. Keep the active launcher setting unchanged where it is already disabled, but do not claim a general ROCm speedup from this 9B result. Raw logs are `logs/qwopus9b_spec_backend_sampling_{on,off}_build1180_20260820.log`. ### Phase 3: Native quantized HIP flash attention This remains the most valuable long-term backend improvement: - [ ] Identify the exact HIP flash-attention kernels selected for target and MTP graphs. - [ ] Measure F16 K/V conversion allocations independently. - [ ] Prototype direct q8_0 K support. - [ ] Add q5_1 V and q4_0 MTP support if the first kernel is successful. - [ ] Validate long-context numerical accuracy and performance against F16. Native quantized kernels could remove the main reason F16 MTP KV can use less total ROCm memory than Q4_0. The cache choice is currently model-, backend-, and topology-dependent: Q4_0 wins on the tested single-GPU IQ4_XS case, while the dual-GPU Q6_K_L case shows equal fitted context with substantially lower Q4_0 MTP allocation. Do not promote F16 or Q4_0 to a universal default until the corresponding runtime and Vulkan comparisons are complete. --- ## 2026-08-15 Mainline Refresh The dedicated `latest` branch was fast-forwarded to upstream `master` commit `5f754ea`. The previously validated `77918ca` patch was committed and merged into the durable integration branch first, then `latest` was merged as `d00e998`. That integration branch was originally named `sol` and is now named `eaman`. Recovery branch `eaman-pre-latest-5f754ea` preserves pre-merge tip `e5c647b`. Older recovery branch names `sol-pre-latest-d2f8305` and `sol-pre-latest-77918ca` were pruned after their tips were verified as ancestors of the integration branch; their commits remain in `eaman` history. The persistent Release build enabled HIP, Vulkan, all quantized flash-attention instances, and HIP architectures `gfx1030;gfx1031`. HIP and Vulkan libraries compiled, and the final `llama-server` linked successfully. The fresh binary reports llama.cpp build 1093 at commit `d00e998`. A separate CPU/default `llama-server` build also completed in a temporary detached worktree. The standalone patch is: ```text /home/eaman/llama/bug/latest_rocm_improvement_5f754ea.patch SHA-256: 5b653a6aaa50fd30cfc05cb9cf50b7a75adc491395c285b587f7e5b237b847c9 ``` After the 2026-08-18 MoE integration, it applies cleanly without three-way fallback to `5f754ea`. Its applied tree is `0f28cb047c374d20ac95e9912167eec13ad0caf1`, exactly matching committed `eaman` tip `773364c`. The temporary validation worktree was removed. The earlier mainline refresh itself performed no runtime benchmark; the later MoE runtime results are recorded in their dedicated section below. ## 2026-08-20 Mainline Refresh The dedicated `latest` branch was fast-forwarded to upstream `master` commit `a3b1eff`. The previous validated Eaman state at `773364c` was preserved as `eaman-pre-latest-a3b1eff`, then `latest` was merged into `eaman` without conflicts as `5850e07`. The persistent Release build used HIP architectures `gfx1030;gfx1031` with `GGML_HIP=ON`, `GGML_VULKAN=ON`, `GGML_CUDA_FA_ALL_QUANTS=ON`, and `LLAMA_BUILD_SERVER=ON`. The HIP and Vulkan targets compiled, the final `llama-server` linked, and the fresh binary reports build 1180 at commit `5850e07c` on ROCm `7.14.60850-0000000` / Vulkan. The standalone patch is: ```text /home/eaman/llama/bug/latest_rocm_improvement_a3b1eff.patch SHA-256: 55812a942789e7e8e234cb01c87bc00db75208e3caad4feb408e6b4d70ac275f ``` It applies cleanly without three-way fallback to `a3b1eff`. Its applied tracked tree is `1eeaabc8556036d82687956e582cefccfb890d63`, exactly matching committed `eaman` tip `5850e07`. The temporary detached validation worktree was removed. After verifying ancestry, the older recovery branch name `eaman-pre-latest-5f754ea` was pruned; `eaman-pre-latest-a3b1eff` is retained. No runtime benchmark or binary deployment was performed by this refresh. ## 2026-08-23 Mainline Refresh The dedicated `latest` branch already matched fetched upstream `master` commit `8144f319`. Production tip `5850e07` was preserved as `eaman-pre-latest-8144f31`, then `latest` was merged into `eaman` as `a4e47992`. The sole conflict was the old server-side MTP estimator/refinement block. It was removed in favor of upstream's joint target/draft fitter, which measures the extra context again whenever the fitted target context changes and does not publish provisional placement into `params_base`. This makes the old MoE scratch-copy workaround unnecessary. The residual Eaman fit adapts fixed-layout per-device interpolation to upstream's total `n_streams` context accounting and forces the fitted MTP extra context to use the same disabled pipeline mode as runtime MTP. The persistent Release build used HIP architectures `gfx1030;gfx1031` with `GGML_HIP=ON`, `GGML_VULKAN=ON`, `GGML_CUDA_FA_ALL_QUANTS=ON`, and `LLAMA_BUILD_SERVER=ON`. HIP, Vulkan, all quantized flash-attention instances, and the final `llama-server` target built successfully. The fresh binary reports build 1243 at `a4e47992` on ROCm `7.14.60850-0000000` / Vulkan, and `test-arg-parser` passed. Dense Q6_K_L joint fitting measured the MTP extra context at 262,144 and 4,096, selected 136,704 context through the fixed-layout per-device path, and reached listening without target-only fallback. This retained the production launcher's manual layer-39 FFN tensor override to ROCm1. At the same reported starting free VRAM (16,310/12,248 MiB), the historical matched placement trial fitted 136,448; the separate 137,984 startup had 32 MiB more free on ROCm0, and its rendering/session mode was not recorded. A 64-token request completed at 27.03 t/s with 41/44 drafts accepted (93.18%). MoE Q5_K_S joint fitting measured the MTP extra context at 262,144, 4,096, and the selected 233,216 context without falling back to target-only fitting. Final automatic fine placement retained tensor-level overrides: ROCm0 received 25 layers with one partial `LAYER_FRACTION_UP` layer and ROCm1 received 17 full layers. It reached listening and a 64-token request completed at 76.28 t/s with 43/47 drafts accepted (91.49%). Target and MTP pipeline modes were disabled in both runs. Runtime `auto` and `on` were not repeated. These are startup and short generation checks, not long-context stability or formal throughput benchmarks. Logs are `logs/mainline_8144f31_dense_mtp_startup_20260823.log`, `logs/mainline_8144f31_dense_mtp_request_20260823.log`, and `logs/mainline_8144f31_moe_mtp_validation_20260823.log`, with adjacent response JSON files. The standalone patch is: ```text /home/eaman/llama/bug/latest_rocm_improvement_8144f31.patch SHA-256: dc266cba88936c9e6db596dca897e4b7470c51cc9f3597c93026d967512ed6f5 ``` It applies cleanly without three-way fallback to `8144f319`. Its applied tracked-tree ID is `673cdbe49ed18722a1bee022ac8d4a0cfedaf69a`, exactly matching committed `eaman` tip `a4e47992`. The temporary validation worktree was removed. The source `master` branch remains untouched at `77918caf`. No production `a4e47992` binary was deployed during that refresh; the later experimental build-1249 deployment is recorded in the next section. ### Rebased recurrent rollback validation The isolated `eaman-rs-snapshot-depth` branch (deleted 2026-09-06) was rebased onto production `a4e47992` without merging it into production. Recovery branch `eaman-rs-snapshot-depth-pre-8144f31` (deleted 2026-09-06) preserved old tip `732dd501`. The tip was `fff1fc7c`; tracked-tree ID is `7a5a49628b0f05507391edcf2cef0b187d804ba7`. CPU/default and combined HIP/Vulkan Release builds passed at build 1249. `ggml-hip`, `ggml-vulkan`, all quantized `fattn-vec` instances, and the final server linked. `test-arg-parser` passed, and registered generated-model tests passed for Qwen3.5 and Nemotron-H, including fragmented device-save host fallback and depth-1 recomputation at every rejection position. Matched real-model results: | Model/configuration | Full depth | Depth 1 | Difference | |---|---:|---:|---:| | Dense Q6_K_L, n-max 3, target 20 | 136,704 | 149,504 | +12,800 (+9.36%) | | MoE Q5_K_S, n-max 4, target 50 | 233,216 | 262,144 | +28,928 (+12.4%) | The dense seeded 192-token responses were byte-identical, accepted 105/105 drafts, and ran at 24.79 versus 24.12 t/s for full/depth-1. Dense depth 1 directly reserved 93.52 MiB on ROCm0 plus 56.11 MiB on ROCm1. The MoE depth-1 run reserved 43.97/18.84 MiB, reached the model's 262,144-token ceiling, and completed 192 tokens at 71.30 t/s with 111/119 accepted drafts plus 2 replay events / 5 replayed tokens. At fixed 233,216 context, seeded MoE full/depth-1 and target-only outputs all diverged at the same late position; this isolates the observed long-output non-identity from the rollback path and is consistent with the existing real-GPU reproducibility limitation. A directly captured one-GPU `-np 2` startup reserved the incremental 100.50 MiB checkpoint group before joint fitting. The final two-sequence target RS buffer was 201.00 MiB because the ordinary 100.50 MiB recurrent group is already part of the baseline. Both slots retained the full 262,144 context. Two explicitly concurrent 192-token requests launched before either finished, completed without OOM or cross-talk, and replayed 1 event / 2 tokens and 7 events / 21 tokens. A separate 1,024-token 9B depth-1 run completed at 62.94 t/s, accepted 523/612 drafts, and exercised 24 replay events / 69 tokens. #### Long-prompt ROCm MoE prefill diagnostics (2026-08-23) A deterministic synthetic prompt was verified as exactly 60,550 tokens with the server `/tokenize` endpoint. Matched Qwen3.6-35B-A3B Q5_K_S runs used both ROCm devices, target Q8_0/Q8_0 KV, MTP Q4_0/Q4_0 KV, batch 2,250, target pipeline off, MTP draft maximum 5, RS depth 1, disabled HIP graphs, and fixed 224,256 context. The microbatch sweep measured: | Microbatch | Prompt time | Prompt throughput | Result | |---:|---:|---:|---| | 256 | 158.09 s | 383.02 t/s | passed | | 512 | 157.14 s | **385.33 t/s** | passed | | 750 | 162.65 s | 372.27 t/s | passed | | 1,024 | - | - | `CUBLAS_STATUS_INTERNAL_ERROR` before timing | Microbatch 512 improved only 3.51% over 750, so launch tuning does not explain the much larger long-prompt Vulkan/ROCm difference. An unrestricted microbatch-512 fit selected the model ceiling of 262,144 but the first request OOMed in `launch_fattn<256,2,1>`. In contrast, microbatch 256 completed at that ceiling. The 262,144 result is therefore not a generally safe production limit for the active batch-2,250, draft-maximum-5, target-70 profile; runtime safety depends on microbatch/transient workspace as well as the fitted persistent allocation. A second matched comparison used microbatch 512 and fixed 120,064 context. Target Q8_0/Q8_0 KV completed at 385.13 t/s; target F16/F16 KV completed at 835.97 t/s, a 2.17x improvement (+117.1%). Q8_0 with flash attention disabled failed during recurrent-state pre-fit context creation. This isolates the main ROCm long-prefill bottleneck to the patched quantized-target-KV flash-attention policy. The Eaman HIP change deliberately forces every supported quantized-KV shape onto VEC, including large prompt batches, because VEC dequantizes in registers and avoids the large F16 temporary buffers used by the upstream HIP tile/MFMA path. Upstream already selects VEC for the usual small-query generation case; the Eaman dispatch primarily extends it to wide queries as a context-capacity and RDNA2-safety tradeoff. F16 target KV follows the faster large-query MMA path without quantized conversion, explaining the diagnostic speed result. It is not yet a production recommendation because its larger persistent KV memory reduces available context and its maximum fitted context was not measured here. Mainline quantized-KV prefill may be faster, but can instead pay the original large conversion-workspace cost; no matched current-MoE mainline throughput run was performed. Experimental tip `fff1fc7c` adds `--hip-fa-force-vec on|off` (environment equivalent `LLAMA_ARG_HIP_FA_FORCE_VEC`), defaulting to `on` to preserve the Eaman patch behavior. `off` bypasses only the unconditional supported quantized-KV HIP VEC override and resumes upstream dispatch; upstream may still choose VEC for its normal small-query cases. The value is stored on every flash-attention graph node, so fitter measurements and runtime use the same policy. This provides the requested opt-in test of Q8_0 KV with upstream's fast large-query kernel and its larger temporary-workspace/context tradeoff. The subsequent production-style Qwen3.6-35B-A3B Q5_K_S test established the intended middle ground. With target Q8_0/Q8_0 KV, batch 2,250, microbatch 512, target 70, and otherwise matching settings, forced VEC fitted 224,256 context and processed the deterministic 60,550-token prompt at 361.95 t/s. Setting `--hip-fa-force-vec off` fitted 163,072 context and processed it at 973.90 t/s: 61,184 fewer context tokens (-27.3%) for 2.69x prompt throughput. The `off` run also generated 3,383 tokens at 76.21 t/s, accepted 2,756/3,236 drafts (85.17%), exercised 27 MTP replay events / 208 replayed tokens, and completed without a checkpoint-allocation failure. This is a strong single-run stability result and the preferred tested MoE compromise when prompt speed matters more than maximum context; it is not yet a repeated stress benchmark. The source record is `/home/eaman/launch/bench/moe_optimization`. Logs are `logs/rocm_ubatch_*_c224256_20260823_build1247.log`, the separate auto-context `logs/rocm_ubatch_{256,512}_20260823_build1247.log`, and `logs/rocm_prefill_{q8_faon,f16_faon,q8_faoff}_c120k_ub512_20260823_build1247.log`. All test servers were stopped and both GPUs returned to their baseline residual allocations. The production-style F16 target-KV launcher `/home/eaman/launch/moe.sh_opti` then exposed a repeatable error in the experimental RS reservation. With batch 2,250, microbatch 512, target-70 fitting, draft maximum 5, RS depth 1, and 256 context checkpoints, the old fitter selected 151,552. Twice, the identical 60,550-token prompt completed prefill (the second run reached 791.99 t/s in its progress report), then the server OOMed on ROCm1 while creating the first on-device `PARTIAL_ONLY` recurrent replay checkpoint. The requested 23.03 MiB buffer came from `llama_io_write_device`; allocation failed and the destructor path asserted at `ggml-backend.cpp:194`. This was not an ordinary server prompt checkpoint, attention execution, or HIP graph-update crash. The RS fitter had used the per-device context-memory delta from `n_rs_seq = D` to `D + 1` as a proxy for the checkpoint. That proxy reported 43.97 MiB on ROCm0 and 18.84 MiB on ROCm1, but the separately allocated checkpoint has its own aligned tensor layout. Feature commit `37c29f3c` now measures that exact backend allocation, including one independent checkpoint per configured slot, and reserves it in the actual device layout. The exact split is 37.69 MiB on ROCm0 and 25.12 MiB on ROCm1; the old plane deltas remain diagnostics only. Commit `fff1fc7c` closes the remaining timing gap by allocating and pinning the real checkpoint buffers for every slot immediately after speculative-context initialization, before evaluation can consume the fitted headroom. If startup allocation fails, or a later device save unexpectedly falls back, the server uses host checkpoints for the rest of that run rather than retrying a device allocation every speculative round. The final build-1249 production-equivalent `--fit-target 70` startup retained 151,552 context, successfully preallocated 37.69/25.12 MiB, logged the early reservation, and reached listening without OOM. The previously recorded 140,288 startup used `--fit-target 200`, so it was not a target-70 before/after result and must not be used to claim that exact sizing reduced context. CPU and HIP/Vulkan focused tests, including the new two-slot preallocation regression, passed. A later Q8_0 target-KV run with VEC forcing disabled completed the full 60,550-token prompt plus 3,383 generated tokens while exercising recurrent replay, providing delayed-OOM coverage for the preferred MoE profile. The exact historical F16 target-KV configuration has not been repeated. The standalone issue record is `rs_checkpoint_fit_reservation_oom.md`. The complete combined HIP/Vulkan build 1249 output from experimental tip `fff1fc7c` was subsequently deployed to `/home/eaman/llama/bin_vulkan`. The server and all shared libraries were copied together, the deployed directory matched the build directory with `diff -qr`, and the deployed `llama-server` reported build 1249 / `fff1fc7c`. The immediately preceding deployment is recoverable from `/tmp/bin_vulkan_pre_1249` until `/tmp` is cleared. The new experimental artifacts are: ```text mtp_rs_only_8144f31.patch SHA-256: 902eaa2424444fc3afc414b8ca68f83c0dbf5bc4681fb8f95c06ffd466feea59 latest_rs_cumulative_8144f31.patch SHA-256: 339317ad294c33ba6e9c28f2a64cc839c69d899140202586c41de7abd8554209 ``` Both apply cleanly without `--3way` to exact base `8144f319`. The RS-only application produces Eaman-free synthetic tree `6c57941ffeb96205441d3398d9868f5e1249d24e`; the cumulative application produces `7a5a49628b0f05507391edcf2cef0b187d804ba7`, exactly matching the RS branch. Temporary validation worktrees were removed. The feature remains experimental and opt-in; startup scratch-fit caching and near-limit long-prompt/delayed-OOM validation remain follow-up work. ## Current Patch and Validation Status - `latest` (`8144f319`) was merged into `eaman` as `a4e47992`. - Upstream's joint fitter replaces the old server estimator/refinement and its MoE scratch-placement workaround; dense and MoE MTP joint fitting both passed. - The current `eaman` tracked-tree hash is `673cdbe49ed18722a1bee022ac8d4a0cfedaf69a`, exactly matching the standalone patch applied to `8144f319`. - Recovery branch `eaman-pre-latest-8144f31` preserves pre-merge tip `5850e07`. - `sol4.patch` remains the historical complete patch based on `7bd8282`. - The persistent HIP/Vulkan build compiled both backends and linked build 1243. - The real source `master` branch was not modified; temporary validation worktrees were removed. - Dense and MoE production short generation passed. The isolated RS branch also passed matched dense/MoE, concurrent two-slot, and 1,024-token replay tests. Near-limit long-prompt, delayed-OOM, controlled throughput, and production runtime pipeline `auto`/`on` testing remain pending. ## Scope and Limitations - Production allocation is validated for the stated dense 27B and MoE 35B models and ROCm topology. - Multiple short requests and one 1,024-token RS request completed, but near-context-limit stability testing is still in progress. - The performance improvement is based on one non-controlled comparison and must not yet be treated as a formal benchmark. - The patch preserves `auto` as the default target pipeline policy. - F16 is the best validated MTP cache for this ROCm model; it is not assumed to be optimal on Vulkan or other architectures. - The per-device fitting path applies when GPU layers are explicitly fixed. Automatic placement retains the existing conservative path. --- ## 2026-08-18 Qwen3.6-35B-A3B MoE MTP fitter investigation The Qwen3.6-35B-A3B Q5_K_S model exposed an ordering bug in the server's MTP pre-fit path. The preliminary target fit wrote its generated tensor split and tensor buffer overrides directly into `params_base`. Refinement and final fit passes then treated that provisional placement as user-supplied and failed with `model_params::tensor_split already set by user`. Context reduction survived the exception, but the final target loaded with the wrong MoE placement. The fix committed as `773364c` (also available independently in `mtp_fit_scratch_placement.patch`) runs every MTP estimation fit against a scratch copy of `common_params`. Only the final target fit publishes generated placement. This keeps the dense-model fitting logic unchanged and lets the final MoE fit account for the device-local MTP load. Test system and launch-relevant settings: - ROCm0 RX 6800: 16,368 MiB total, 16,342 MiB initially free. - ROCm1 RX 6700 XT: 12,272 MiB total, 12,248 MiB initially free. - Target KV Q8_0/Q8_0, MTP KV Q4_0/Q4_0, batch 1024, one slot, target pipeline parallelism off, no warmup. - Model: Qwen3.6-35B-A3B-UD-Q5_K_S. Measured results: | Configuration | Runtime context | Prompt/generation result | |---|---:|---| | Unpatched auto `--fit-target 50` | 235,008 | 12,309-token prompt OOM on ROCm1 | | Explicit `-c 228000 --tensor-split 0.572,0.428` | 228,096 | 32,327 prompt tokens + 16 generated tokens passed | | Explicit `-c 232000 --tensor-split 0.572,0.428` | 232,192 | 12,308-token prompt OOM on ROCm0 | | Patched auto `--fit-target 150` | 220,672 | 32,327 prompt tokens + 16 generated tokens passed | | Patched auto `--fit-target 100` | 227,840 | 32,327 prompt tokens + 16 generated tokens passed | | Patched auto `--fit-target 50` | 235,008 | 32,327 prompt tokens + 16 generated tokens passed; separate 12,308 + 64 test also passed | The patched target-50 fit completed all four fit passes successfully and loaded 13,797.79 MiB of model buffers on ROCm0 plus 10,031.37 MiB on ROCm1. The unpatched run froze an earlier 13,620.73/10,208.43 MiB placement and OOMed. This also shows why a simple tensor split is insufficient: `0.572,0.428` crossed a discrete layer boundary and loaded 14,153.92/9,675.23 MiB; the fitter uses tensor overrides to find an intermediate, MTP-aware placement. The HIP/Vulkan `llama-server` compiled and linked successfully in the detached temporary worktree. The patch is committed on source branch `eaman` as `773364c`, is included in `latest_rocm_improvement_5f754ea.patch`, and the validated full `build/bin` output is deployed to `/home/eaman/llama/bin_vulkan`. The previous deployment is retained at `/home/eaman/llama/bin_vulkan.pre-moe-fit-20260818`. The active `/home/eaman/launch/moe_tester.sh` now uses no explicit context or placement, `--fit-target 50`, and MTP n-max 4; its previous version is retained beside it as `moe_tester.sh.pre-moe-fit-20260818`. Logs are under `logs/moe_*`. ### Delayed ROCm graph-update crash and MoE-only workaround A production run later reached 96,484 cached tokens, restored/reused that prompt, and generated at least 6,632 more tokens before receiving SIGSEGV. The core identified LWP 44707 as the faulting thread. Its stack was `hipGraphExecUpdate` -> `ggml_cuda_graph_evaluate_and_capture` -> `ggml_backend_cuda_graph_compute` -> `llama_decode`. The signal was `SEGV_MAPERR`; no OOM frame or prompt-checkpoint frame was present. This is a separate delayed ROCm graph-executable update failure, not the original MTP auto-fit placement bug. The runtime environment variable `GGML_CUDA_DISABLE_GRAPHS=1` bypasses the failing HIP graph capture/update path without changing the binary, fitter, generated tensor split/overrides, MTP, or other launchers. It is exported only by `/home/eaman/launch/moe_tester.sh`, so dense-model processes retain graphs. With that variable and the unchanged patched `--fit-target 50` configuration, the fitter again selected 235,008 context and the same 13,797.79/10,031.37 MiB model placement. Validation ingested 32,318 tokens and generated 4,718 tokens to EOS at 41.73 t/s. A second request restored the 81.180 MiB prompt checkpoint and completed a forced 7,000-token generation at 47.31 t/s. The combined 11,718 generated tokens completed without a crash. This is a controlled regression test of the observed failing path, not yet a full 235k-context stability claim. A subsequent production run with the same MoE-only graph disable restored a 94,446-token checkpoint, ingested 49,401 new tokens into a 143,851-token prompt, and generated 1,698 tokens at 30.26 t/s before EOS. Prompt ingestion averaged 102.94 t/s and the slot released cleanly at 145,550 tokens. The immediately preceding request on the same process generated 10,385 tokens at 40.86 t/s and released cleanly at 104,837 slot tokens. This exceeds the original delayed failure's generation exposure and validates checkpoint restore plus long-prompt ingestion in production, while still not claiming the full 235k allocation. The final target-50 stress request went substantially further. It restored a 197,327-token checkpoint, processed a 203,372-token prompt (6,045 prompt-eval tokens at 58.80 t/s), and generated 11,145 tokens at 25.78 t/s. Draft acceptance was 99.252% (8,623/8,688), with mean accepted length 4.70. The slot released cleanly at 214,518 tokens, 20,490 below the fitted 235,008 limit, with no OOM or segfault. This is the strongest long-context production validation of the MoE-only HIP-graph workaround so far. Evidence: - `logs/moe_hip_graph_exec_update_segv_coredump_info.log` - `logs/moe_patched_auto_fit50_no_hip_graphs_success.log` - `logs/moe_no_hip_graphs_production_145k_success.log` - `logs/moe_no_hip_graphs_production_214k_success.log` --- ## 2026-08-25 MTP Compact Rollback integration and mainline refresh Local `latest` was fast-forwarded to upstream `3737e413` without changing the source `master` branch. The previous production tip `a4e47992` is preserved as `eaman-pre-latest-3737e41`. Mainline was merged into production as `19594a4a`, standalone MTP Compact Rollback was merged as `f3365fe1`, and cumulative tip `5c366ffc` makes `--hip-fa-force-vec` explicitly opt-in and default-off. The standalone branch is `eaman-mtp-compact-rollback` at `06c1414f`. During its rebase, the new upstream multi-sequence split-replay test was retained beside the MTP CR recomputation tests. DeepSeek-V4's compact layout has no recurrent device rows to preallocate, so its test follows the server's supported host checkpoint fallback instead of treating that layout as an allocation failure. Validation results: - Standalone CPU/default build 1266 linked `llama-server`. - Cumulative CPU/default build completed and linked `llama-server`. - Cumulative build 1279 compiled `ggml-hip`, including `fattn.cu` and the VEC instances, compiled `ggml-vulkan`, and linked the final application. - The focused parser and recurrent rollback suite passed 5/5 on the standalone CPU build, 5/5 on the cumulative CPU build, and 5/5 on the combined HIP/Vulkan build. Qwen, Nemotron-H, and DeepSeek-V4 fixtures all passed. - A build-1279 ROCm1 startup selected 241,408 fitted context and reserved the exact 50.25 MiB device checkpoint before evaluation. A fixed-32K repeat then completed a deterministic 1,024-token request at MTP CR depth 1 with 11 deep replay events / 25 replayed tokens. The isolated test server was stopped and the separately running user Vulkan process was not disturbed. - The complete build-1279 `build/bin` directory was staged, compared with `diff -qr`, and atomically deployed to `/home/eaman/llama/bin_vulkan`. Its server reports build 1279 / `5c366ffc`. The prior loaded deployment remains at `/home/eaman/llama/bin_vulkan_previous`, and the existing Vulkan process continued running. Active and test launchers were migrated mechanically from the retired `--spec-mtp-rs-depth` spelling to `--spec-mtp-cr-depth`. - Adaptive MTP was merged into the standalone branch by fast-forwarding `eaman-mtp-compact-rollback` to `6cb89357`, then merged into cumulative `eaman` at `527be7e6`. - Cumulative HIP/Vulkan build 1282 linked `llama-server`, and its focused parser test passed. The matching standalone combined binary remains build 1268 at `/home/eaman/llama/llama-mtp-cr-adaptive/build-adaptive-gpu/bin/llama-server`; the directory name is historical, but its commit exactly matches the current standalone branch. - Both regenerated patches applied cleanly to detached `3737e413` without `--3way`. Standalone applied tree `15b31de45eeefbf85df27aa4e27d6f5e78d60fd0` exactly matched `6cb89357`; cumulative applied tree `64439f0db2b9afe42ddf35e1e0bdf3c57f8464e4` exactly matched `527be7e6`. Artifacts: ```text mtp_compact_rollback_3737e41.patch SHA-256: 759ad384f22859f084aa4a1c11252e63a423568f7e31cb9f623d4b473bd526ba latest_rocm_improvement_3737e41.patch SHA-256: 266efe7114dc74f0b947e98d5a355c2827a353317e9ea2cb6f6b746a4d0f4425 ``` The existing dense/MoE, two-slot, and delayed-OOM evidence was produced on the preceding base. Build 1279 now also has direct 1,024-token replay coverage. A near-limit long-prompt confirmation remains useful additional stress coverage; startup scratch-fit caching is an optional efficiency improvement, not a correctness prerequisite. --- ## 2026-08-25/26 adaptive MTP integration Adaptive speculative draft sizing was isolated from `LaurentZuijdwijk/llama.cpp` commit `ca261699` and ported onto the standalone MTP Compact Rollback tip `06c1414f`. No Vulkan, FP4, ROCmFPx, benchmark, or other unrelated fork changes were imported. After isolated validation, commit `6cb89357` became the current standalone `eaman-mtp-compact-rollback` tip. Cumulative `eaman` includes it through merge `527be7e6`. The former `eaman-mtp-cr-adaptive` branch and linked worktree were deleted 2026-09-06. The opt-in control is: ```text --spec-draft-adaptive ``` It defaults to off. `--spec-draft-n-max` remains the ceiling. The stabilized controller starts each sequence at that known-good ceiling. Two consecutive ceiling-reaching drafts that accept fewer than three tokens step the ceiling down by one; full acceptance steps it back up by one. Drafts stopped early by `--spec-draft-p-min`, a slot cap, or decode failure do not alter the ceiling. `--spec-draft-n-min` can still impose a floor, but the validated launcher leaves it at its normal zero value (the controller internally clamps its floor to one). The port includes corrections needed for the current server and MTP CR path: - adaptive state resets when a sequence begins or a server slot is reused; - the controller records the draft length actually emitted and distinguishes a reached adaptive ceiling from confidence truncation; - effective limits remain within the configured minimum and maximum; - after a deep MTP CR restore/replay, adaptation observes the original target acceptance count exactly once instead of the replay batch's extra replacement token, while physical MTP state accumulation still receives the replay count. CPU/default and combined HIP/Vulkan builds completed. The parser suite passed on both configurations, both GPU backends enumerated, and the final combined binary reports build 1268 / `6cb89357`: ```text /home/eaman/llama/llama-mtp-cr-adaptive/build-adaptive-gpu/bin/llama-server SHA-256: 0e353fcb1f7f6c49b472ece38c43e5596093b743a802522ac3df451dc4384406 ``` The adaptive binary has not replaced `/home/eaman/llama/bin_vulkan`. The five focused parser/recurrent-state tests pass. A matched 1,024-token Vulkan run with `n-max=7`, `p-min=0.75`, and `n-min=0` produced identical fixed/adaptive draft and replay counts (785 generated, 581 accepted, 48 MTP CR replays / 171 tokens) at 30.76 and 30.73 t/s respectively. The original slowdown was reproduced by setting `n-min=3`: fixed and adaptive then produced the same altered counters and 26.59 / 26.60 t/s. Therefore the regression was caused by the launcher's minimum-draft setting, not adaptive-controller overhead. The test launcher now uses `--spec-draft-adaptive --spec-draft-p-min 0.75 --spec-draft-n-max 7` and does not force `--spec-draft-n-min`. The broader user runtime matrix confirmed a workload-dependent trade-off. With build 1268 and identical `p-min=0.75`, `n-max=7` settings, adaptive MTP cost about 2% on long code (27.46 to 26.83 t/s at 68,864 context; 26.52 to 26.08 t/s at 144,384 context). Creative prose improved from 14.38 to 14.92 t/s at 68,864 context and from 10.49 to 14.20 t/s at 144,384 context. Adaptive therefore remains default-off and opt-in, while being included in both canonical patches. --- ## 2026-08-31 mainline and dual-ROCm refresh Local `latest` was fast-forwarded from `3737e413` to upstream `662a0b0121a53c23b825a71e64ab6eff59b7f4d8` without modifying source `master`. Recovery branches preserve the preceding validated tips: ```text eaman-pre-latest-662a0b01 527be7e6 eaman-mtp-compact-rollback-pre-662a0b01 6cb89357 ``` The standalone merge is `cdc3d024` with tree `89b4b2266f368b5deaeb79cd398db88ef0e6de47`. The cumulative merge is `9ee6674f` with tree `31dfa8e080bb3c3657979c59675661473ae73a90`. Upstream changed the same speculative-decoding and server control paths, so the merge adaptations retained both the Eaman adaptive/rollback state and new upstream synthetic-acceptance controls. The four conflicts were `common/speculative.cpp`, `tests/test-arg-parser.cpp`, `tools/server/server-context.cpp`, and `tools/server/tests/utils.py`. Both exact branches were configured for `gfx1030;gfx1031` with HIP, Vulkan, all quantized Flash Attention instances, and `llama-server`. Validation was repeated with absolute, version-pinned package paths for two installed ROCm trees: - ROCm core-7.14 / HIP `7.14.60850-0000000`; - ROCm core-10.0 / HIP `7.15.26333-0000000`. The unqualified system alternatives had moved to core-10.0. The first 7.14 link attempt therefore mixed the 7.14 compiler with an unversioned `/opt/rocm/lib` dependency and stopped on a nonexistent 7.14 SONAME under that 10.0 directory. Reconfiguration pinned the compiler, `hipconfig`, HIP, hipBLAS, rocBLAS, HSA, amd_comgr, and AMD device-library packages to core-7.14; the stale dependency disappeared and the build completed. The core-10.0 builds used separate clean build directories and similarly pinned 10.0 paths. For both branches under both ROCm trees, HIP (including `fattn.cu` and all `fattn-vec` instances) compiled, Vulkan compiled, the final server linked, and `llama-server --version` enumerated the RX 6800 and RX 6700 XT. The parser passed and all four registered generated-model/recurrent tests passed, including the new upstream DSV4 case. The standalone binary reports build 1362 / `cdc3d024`; the cumulative binary reports build 1376 / `9ee6674f`. The generated patches apply cleanly to detached `662a0b01` without `--3way`. Their applied tracked trees exactly match the source branch trees. Validation worktrees were removed. Recovery branch `eaman-pre-latest-3737e41` was verified as an ancestor of the new cumulative tip and pruned with `git branch -d`. `eaman-mtp-compact-rollback-pre-3737e41` was not an ancestor of the refreshed standalone tip, so its name was retained for manual history as required (deleted 2026-09-06 as superseded). The new retained recovery branches are `eaman-pre-latest-662a0b01` and `eaman-mtp-compact-rollback-pre-662a0b01` (deleted 2026-09-06). ```text old/patches/mtp_compact_rollback_662a0b01.patch SHA-256: 4ff2fcaa1ab12002430d0a999761ba0b5fbd59afb7e23faa6743307928f29ea6 old/patches/latest_rocm_improvement_662a0b01.patch SHA-256: 9cf18849b195d0da34a6e648702d67c9359ee0b2a1cdc67956eebe6a90ea8178 ``` The refreshed binaries have not been deployed to `/home/eaman/llama/bin_vulkan`. Build 1282 / `527be7e6` deliberately remains deployed for daily use; build 1376 is retained as the validated refresh build pending any later matched performance comparison. The prior runtime and long-context evidence remains historical. The former open-ended near-limit long-prompt and delayed-OOM follow-up has been retired from the current refresh acceptance criteria and preserved under `old/`. --- ## 2026-09-03 mainline refresh and MTP multi-ubatch integration Before refreshing mainline, the ordered MTP multi-ubatch change was committed as `886fda24`. Its focused test verifies that a split MTP batch synchronizes and accounts the complete batch while a single ubatch retains asynchronous execution. The user runtime validation then reached 124,055 context tokens and generated another 2,160 tokens at 16.08 t/s. MTP draft acceptance was 0.90974; the log showed no OOM, NaN, decode failure, or hang. Around 78K context, prompt processing was effectively unchanged from the older comparison build (about 230.86 versus 231.06 tokens/s), though this was not a fully controlled cross-ROCm benchmark. Local `latest` was fast-forwarded by 69 commits from `662a0b01` to upstream `0df017d6dd246edb1d06577976c759cf7a3c50d6` without modifying source `master`. The cumulative merge is `7f07026c36ab2ea6a99947e6d0fd2d2b96838ff6` with tree `e664f43209816389de32cbe3794ad14d326960b4`. Recovery branch `eaman-pre-latest-0df017d6` preserves `886fda24`. Three merge conflicts were resolved. The MTP synchronization test was retained while adopting upstream's restored numeric `-v` behavior. More importantly, new sparse flash attention uses FLASH_ATTN_EXT op-parameter slot 4 for `n_kv_max`, which collided with the Eaman HIP VEC flag. Sparse FA remains in slot 4 and the VEC flag now uses slot 5; graph construction sets both values. Notable upstream GPU changes in this refresh include: - ROCm radix TOP_K support for long rows (`f8dbcd61`), relevant to the prior ROCm1 TOP_K fallback warning; - concurrent streams per split for multi-GPU CUDA/HIP scheduling (`0ba6499c`); - sparse FA for DSV4/GLM, Vulkan FA dequant dispatch fixes, and fused MoE weighted expert reduction. The existing combined build was configured for `gfx1030;gfx1031` with HIP, Vulkan, all quantized FA instances, and `llama-server`. Although the shell's unqualified `hipconfig` points at core-10.0, `CMakeCache.txt` confirms this build used `/opt/rocm/core-7.14/lib/llvm/bin/clang++`, HIP `7.14.60850-0000000`. HIP including `fattn.cu`, Vulkan, and the final server all compiled and linked. The resulting server is build 1447 / `7f07026c`; both ROCm devices and both Vulkan devices enumerated. Validation passed: - `test-arg-parser`; - `test-mtp-ubatch-sync`; - `test-generate-models`; - dense, Nemotron-H, and DSV4 recurrent-state rollback tests. The focused `test-backend-ops test -o TOP_K` suite also passed all 525 cases on ROCm0 (RX 6800/gfx1030) and ROCm1 (RX 6700 XT/gfx1031), directly validating the new ROCm radix TOP_K path on both target devices. The cumulative patch applies cleanly without `--3way` to a detached `0df017d6`. The applied binary diff and `latest..eaman` diff both hash to `1b02f255d42ff90b7724e429cf0cf5629504eac69281f699e0c71b12d2b3c075`, confirming exact tracked-tree equality. The temporary worktree was removed. The obsolete cumulative recovery `eaman-pre-latest-662a0b01` was verified as an ancestor and pruned; standalone recovery branches were left unchanged. ```text latest_rocm_improvement_0df017d6.patch SHA-256: ff848e448498fd8fdcb9c8817e975959d8ce22fc6a5c5be4ff4da4fe75e90e20 ``` The standalone export was still `mtp_compact_rollback_662a0b01.patch` at this point; it was rebased immediately afterward as documented below. Build 1447 has not been deployed, and `/home/eaman/llama/bin_vulkan` remains build 1282 / `527be7e6`. --- ## 2026-09-03 standalone MTP Compact Rollback refresh Before the standalone refresh completed, upstream advanced again from `0df017d6` to `de8656bd94f1163188125542534e4bcbc9f9fb1f`. Local `latest` was fast-forwarded to that exact commit without modifying source `master`. The standalone MTP Compact Rollback branch merged it cleanly as `f1b4a96f5ea066fba2fca35585e01fcd664852f0`, with tree `fd6ced5f833f0ff5532f014f52251c2e1a5770f5`. Recovery branch `eaman-mtp-compact-rollback-pre-de8656bd` preserves the preceding standalone merge `cdc3d024`. The `latest..eaman-mtp-compact-rollback` review confirmed that the standalone delta contains Compact Rollback, on-device checkpoint reservation and fitting, adaptive MTP, and their tests. It excludes the cumulative pipeline-parallel, HIP VEC, and MoE-specific changes. No merge adaptation was required. The CPU/default build completed. The pinned ROCm core-7.14 / HIP `7.14.60850-0000000` combined build targeted `gfx1030;gfx1031` with HIP, Vulkan, all quantized FA instances, and `llama-server`. HIP compiled, Vulkan compiled, and the final server linked as build 1434 / `f1b4a96f`. Both ROCm GPUs and both Vulkan devices enumerated. The parser, generated-model, dense recurrent rollback, Nemotron-H rollback, and DSV4 rollback tests all passed. The standalone server exposes `--spec-mtp-cr-depth` and `--spec-draft-adaptive`; cumulative-only `--pipeline-parallel` and `--hip-fa-force-vec` controls are absent as intended. The patch applies cleanly without `--3way` to detached `de8656bd`. The applied binary diff and branch diff both hash to `e8d721eec620f50949d82f9a366461c2f5df7d304957a85491dbd906e1d71749`, confirming exact tracked-tree equality. The validation worktree was removed. ```text mtp_compact_rollback_de8656bd.patch SHA-256: ff1eec3c4d0fc32812322a5a11fa4d78ceba3100456eaeb24af9300934162106 ``` The previous standalone patch was archived as `old/patches/mtp_compact_rollback_662a0b01.patch`. No binary was deployed. --- ## 2026-09-09 rdna-boosts blocks 09/10 port (K-quant decode kernels) The RDNA3 / RX 7900 XTX optimization fork (`nasone32/llama.cpp-RDNA3-7900xtx-opt`, fork base `9cffdcc80`) was evaluated as a source of decode-path improvements for the `gfx1030;gfx1031` system (RX 6800 + RX 6700 XT). Its base is 53 upstream commits older than `73a43d1`; the port was done by cherry-pick onto `eaman` in the worktree branch `eaman-rdna3-port` (kept for further candidate ports). Two blocks from the rdna-boosts series (original author: Stew Forster, stew675) were selected and ported: - **Block 09** (`91de35039`): meta-backend compute-container headroom 16 -> 128 in `ggml-backend-meta.cpp`. The commit motivates the change with recurrent models creating ~2*(n_rs_seq+1) views per recurrent layer for conv-state snapshot copies — directly related to the MTP Compact Rollback checkpoint subsystem. Applied with zero conflicts. - **Block 10** (`efa4e8641`): K-quant MMVQ VDR boost. New `vec_dot_q4_K_q8_1_vdr4`, `vec_dot_q5_K_q8_1_vdr4`, and `vec_dot_q6_K_q8_1_vdr2` kernels double the work per shared block-metadata load; Q4_K/Q5_K/Q6_K dispatch is unconditional across RDNA, Q8_0 vdr4 is gated to RDNA3_0/RDNA4. Also splits the RDNA3_5 MMVQ parameter table off from RDNA2, hoists the row scale pair in `mmq-vec-dot.cuh`, guards CUDA-graph capture against `GGML_CUDA_OP_TIMING`, and adds 27B decode-shape perf cases to `test-backend-ops`. All four CUDA files auto-merged over the 53-commit upstream delta; the single conflict (both sides adding perf test cases at the same location in `test-backend-ops.cpp`) was resolved by keeping both additions. Build 1492 (worktree `eaman-rdna3-port`, ROCm core-10.0 / HIP 7.15.26333, Vulkan, `gfx1030;gfx1031`, `GGML_CUDA_FA_ALL_QUANTS`) compiled and linked; the only log error was a cosmetic UI tarball download failure. The CPU/Release build also passed 100%. A/B validation on Qwen3.8-27B-UD-Q6_K_M, layer split 0.6/0.4, seeded synthetic MTP acceptance (`--spec-synth-len 4.5`, request seed fixed), 2048 generated tokens, build 1282 (`/tmp/llama.std`) vs 1492 (`/tmp/llama.rd`): | Metric | 1282 | 1492 | |---|---|---| | Eval 2048 tok | 84.36 s (24.27 t/s) | 80.15 s (25.54 t/s, **+5.2%**) | | Draft acceptance (slot) | 0.7329 | 0.7342 (flat) | | acc rate/pos | (0.995, 0.848, 0.730, 0.620, 0.431) | (0.986, 0.806, 0.696, 0.597, 0.405) | | Draft gen duration | 26.0 ms/draft | 25.3 ms/draft | | Verify duration | 0.534 ms | 0.534 ms | | Warnings / VRAM / shutdown | clean | clean, identical VRAM | The small per-position accept-rate shift is the expected synthetic-seed phase effect (different draft call counts consume different phases of the seeded stream); the overall acceptance is flat. The +5.2% TG is the Q6_K vdr2 kernel working on gfx1030/gfx1031; prefill is neutral as expected (mmvq is the decode path). Also evaluated and **dropped**: the fork's Q8-wire AllReduce compression (`e06dcf630` + `7c5bb5cb9`). In the daily layer-split configuration there is no AllReduce at all; cross-PCIe traffic is only the residual handoff at the ~17 device boundaries (~357 MB per 1024-token prefill batch, a fraction of a percent of prefill time at 200-300 t/s). The Q8 wire only becomes a first- class optimization under `--split-mode row`, which is not used. The fork's topology (6800 CPU-direct x16 Gen4, 6700 XT via PCH x16 Gen4, no P2P) was confirmed as the basis for this decision. The remaining fork candidates were all evaluated and closed (round detail in `rdna3_port_plan.md`): the wave32-native TOP_K trio measured flat in the seeded A/B and was not merged (2026-09-09); mmvdq (`670512936`) measured +0.7% on Q5_K_M, below the merge bar, not merged (2026-09-10); the adaptive MTP controller (`10579a736`) was compared line-by-line against the Eaman `--spec-draft-adaptive` controller and found no idea that beats it for this config — compared, no port (2026-09-11). The MoE expert cache and MMQ MoE tiling are N/A on this layer-split, RDNA2, dense-daily setup. The rdna3 port plan is exhausted for the current workload; revisit only on a config change (MoE expert offload, GDN models, or new fork commits). Blocks 04 (RDNA4 WMMA), 05/06/07, and 11 are not candidates for this system. Blocks 09/10 were merged into `eaman` as `9ee65ec3`. The cumulative patch was regenerated and validated (clean apply to detached `73a43d1`, applied diff exactly equal to the `eaman` tree `489357af`): ```text latest_rocm_improvement_73a43d1.patch SHA-256: 1c67b3d62ab618a8cec4771618b5e9e8f8ad05d034dd66f907944bee42d1cab9 ``` The previous cumulative export (eaman `9570cc08`, tested version 1490, SHA-256 `8149508206fa581f8af17811d7379c3ca0e2de7184065c5f72ae6ab1730caabb`) is now historical. No binary was deployed; `/home/eaman/llama/bin_vulkan` remains build 1282 / `527be7e6`, and the canonical `build/bin` was not rebuilt at this step. ## Adaptive p_min draft gate — abandoned (2026-09-11) An experimental adaptive p_min confidence-gate controller (draft-mtp only) was developed as uncommitted working-tree changes on `eaman` tip `9ee65ec3` and built/tested in `build-min-p` (build 1493, ROCm core-10.0, HIP + Vulkan): `--spec-draft-p-min-adaptive` moved the `--spec-draft-p-min` gate within a `--spec-draft-p-min-range` band (tested [0.45, 0.78]) from measured per-round draft acceptance (EMA, dead band, step 0.05). It touched `common/arg.cpp`, `common/common.h`, and `common/speculative.cpp` (172 insertions, 1 deletion). The gate correctly tracked the content phase, but the A/B results on Qwen3.8-27B-UD-Q6_K_M were not good enough to keep. At shallow context it was ~+1% versus static 0.75 (the gate pinned at the 0.45 floor ~94% of samples); at deep context (~130k, session-KV continuation) it was clearly worse: | metric | static 0.75 | adaptive 0.45–0.78 | |---|---|---| | TG speed | 14.3–14.8 t/s | 12.3–12.6 t/s (≈14% slower) | | draft acceptance | 0.73–0.77 | 0.61–0.62 | | MTP-replay rate | 15–17% | 28–35% (≈2×) | The 0.45 floor is tuned for the shallow-repetitive regime; at deep ctx the replay cost rises and the draft model is less confident, so deep drafts are mostly rejected. **Static `--spec-draft-p-min 0.75` is the retained configuration.** The planned follow-up (ctx-depth-interpolated band) is dropped with the experiment. The exact diff was exported and the working tree reverted to the committed `eaman` state: ```text old/patches/spec_draft_p_min_adaptive_eaman_9ee65ec3.patch base eaman 9ee65ec3 (mainline base 73a43d1) SHA-256: dfb9b1b9e25910b9bb4d5a7fed885065f575c19cb42c52e5e8da60bba93485f5 ``` The patch was validated in a temporary detached worktree: `git apply --check` passed on clean `9ee65ec3` and the applied diff round-tripped byte-identical to the saved patch. The temporary worktree was removed; no binary was deployed for this experiment. Historical artifacts for this experiment: - Plan, implementation notes, and full A/B record: `old/adaptive_pmin_plan.md` (§9 has the measured results). - A/B logs: `logs/qwen27b_q6_static_pmin0.75_session1_20260910.log`, `logs/qwen27b_q6_adaptive_pmin_0.45-0.78_20260910.log`, `logs/qwen27b_q6_static_pmin0.75_deepctx_20260910.log`. ## 2026-09-11 deployment state correction `/home/eaman/llama/bin_vulkan` was checked and is the latest cumulative build: `version 1493, commit 9ee65ec3` (deployed 2026-09-10 19:37), i.e. the current `eaman` tip — the same tree as the canonical `latest_rocm_improvement_73a43d1.patch` (tree `489357af`). `llama-server --help` confirms the Eaman flags `--spec-mtp-cr-depth`, `--hip-fa-force-vec`, and `--spec-draft-adaptive`. The earlier doc statements that the deployment remained build 1282 / `527be7e6` were stale. Two old copies are broken and must not be reused: `llama.cpp/build/bin` (Sep 8) and the preserved prior deployment `/home/eaman/llama/bin_vulkan_previous` both segfault on `--version` (inconsistent `.so` copies; the previous dir has `libggml-base.so.0` symlinks to `0.23.0` while only `0.22.0` is present). ## 2026-09-16 mainline refresh to `c6824a9e` The recurring update workflow advanced `latest` from `73a43d1f` to `c6824a9e42ceeda5d58089fa274ddd816e59e68e` (upstream `origin/master`, 34+ commits) and merged it into `eaman` as `ddddf03d69f743701a562e146a565e5112bbe73a`. Recovery branch `eaman-pre-latest-c6824a9e` preserves the preceding tested tip `9ee65ec3`. The old recovery branch `eaman-pre-latest-73a43d1` (tip `7f07026c`) was verified as an ancestor and pruned. Merge conflicts and adaptations (3 files): - `src/llama-graph.cpp` and `src/models/minimax-m3.cpp`: upstream renamed `ggml_flash_attn_ext_set_prec` to `ggml_prec_set_acc`. Kept the renamed upstream call plus the Eaman `ggml_flash_attn_ext_set_force_vec` call (op-parameter slot 5, unchanged by this mainline). - `tests/test-recurrent-state-rollback.cpp`: upstream had since extended its own version of this test (cache-fill parameter, `init_ctx`/buffer collector, Kimi-K3 and multi-seq split-replay coverage). Resolved by keeping upstream's file untouched and layering the Eaman tests on top as pure additions: `test_fragmented_on_device_fallback`, `test_preallocated_device_checkpoints`, and `test_recompute_fallback`, with Eaman's `make_ctx` renamed to `make_ctx_rs` to avoid clashing with upstream's fill-based `make_ctx`. All three Eaman tests run in `main` before upstream's fill loop. The test-file delta versus `latest` is 239 insertions, 0 deletions. Build: reconfigured and built the persistent `llama.cpp/build` tree on clean `eaman` (`Release`, `gfx1030;gfx1031`, `GGML_HIP=ON`, `GGML_VULKAN=ON`, `GGML_CUDA_FA_ALL_QUANTS=ON`, `LLAMA_BUILD_SERVER=ON`, ROCm core-10.0 / HIP 7.15.26333). HIP compiled (188 HIP objects, incl. all `fattn-vec` instances), Vulkan compiled, and `llama-server` linked: **build 1676 / `ddddf03d`**. No errors; the only warnings are pre-existing upstream ones (`ggml-vulkan.cpp` nonnull, `mtmd/clip-graph.h` deprecated enum ops). Logs: `logs/build_c6824a9e.log`, `logs/configure_c6824a9e.log`. The custom delta versus `latest` is the same 40 Eaman patch files as before (1582 insertions, 72 deletions); `git diff --check` against `latest` is clean. No binary was deployed; `/home/eaman/llama/bin_vulkan` remains build 1493 / `9ee65ec3`. Regenerated cumulative patch, validated in a temporary detached worktree at `c6824a9e` (clean `git apply --check` without `--3way`; applied diff exactly reproduces the `eaman` tree `6478afb9`; worktree removed): ```text latest_rocm_improvement_c6824a9e.patch SHA-256: ab3ee93974930b239fd62aa61c2bfc1926e8dc3e5ba6e31872e453d4e2a7a4cf ``` The previous cumulative export (base `73a43d1`, eaman `9ee65ec3`, tested version 1492, SHA-256 `1c67b3d62ab618a8cec4771618b5e9e8f8ad05d034dd66f907944bee42d1cab9`) is now historical. The standalone branch `eaman-mtp-compact-rollback` was advanced to the same base in the same refresh: merged as `2586df93`, recovery branch `eaman-mtp-compact-rollback-pre-c6824a9e` preserves `2fe7df5f`. Its only conflict was the same upstream `test-recurrent-state-rollback.cpp` extension and was resolved identically (Eaman tests as pure additions, Eaman `make_ctx` renamed to `make_ctx_rs`). Build 1658 / `2586df93` compiled HIP and Vulkan and linked `llama-server` in the standalone worktree `build-standalone`; `test-arg-parser` passed. The full recurrent-model rollback tests were not re-run because no recurrent/hybrid model is present locally. Log: `logs/build_standalone_c6824a9e.log`. Regenerated and validated standalone patch (clean apply to detached `c6824a9e`, exact standalone tree `a9fb7032`): ```text mtp_compact_rollback_c6824a9e.patch SHA-256: f9894a567a2d6cd930366bb6bc634cf64bbce080dceb42e447f03aa6c676654c ``` The previous standalone export (base `73a43d1`, MTP CR `2fe7df5f`, SHA-256 `82a9b537e04074e17f7c1caa5c908ade9d4d2f10800bae5110bbcdbf4e9acf7a`) is now historical. ## Conclusion The current cumulative Eaman patch aligns MTP fitting with runtime allocation, fits fixed multi-GPU layouts against per-device limits, controls target pipeline scheduling, reduces target recurrent rollback memory through MTP Compact Rollback, provides opt-in adaptive MTP draft sizing, exposes HIP VEC forcing as a default-off diagnostic/capacity control, and now also carries the ported rdna-boosts decode-kernel improvements (blocks 09/10), measured at +5.2% TG on the daily Qwen3.8-27B-UD-Q6_K_M setup with flat MTP acceptance. The standalone MTP CR export contains Compact Rollback plus adaptive sizing for independent mainline testing. Further rdna3-fork candidates are tracked on the `eaman-rdna3-port` worktree branch.