# Patch History: ROCm MTP and HIP VEC Changes

This document answers: **what does the Eaman patch change compared with llama.cpp mainline?**

Author and maintainer: eaman. Historical "Qwen" and "Sol" labels identify the
LLM-assisted development series and do not indicate authorship.

## One-line summary of all implemented changes

- MTP now uses the target context size after fitting instead of the original requested context size.
- The server performs a preliminary target fit before estimating MTP memory.
- MTP fitting reserves the memory that the runtime really allocates rather than a misleading standalone estimate.
- MTP memory is measured again after fitting and the target fit is repeated with the refined reservation.
- Fixed multi-GPU layouts are fitted against each GPU's individual memory limit instead of only a generic aggregate allowance.
- MTP fitting and runtime initialization use a disabled pipeline scheduler.
- The target context gains `--pipeline-parallel auto|on|off`, with `auto` remaining the default.
- Supported HIP quantized-KV flash-attention shapes use VEC dispatch to avoid the larger F16 conversion workspace used by the fallback path.

## At-a-glance list

Each item below starts with a one-line summary, followed by the implementation
and historical context.

### 1. Use the fitted target context for the runtime MTP context

MTP now uses the target context size after fitting instead of the original requested context size.

**Files:** `common/speculative.cpp`

The target context may be reduced substantially by GPU memory fitting. The MTP
draft context is created with `llama_n_ctx(ctx_tgt)`, so its context-dependent
KV and flash-attention allocations match the context that will actually run.

This was the first functional fix and appears in `old/fix.patch`,
`old/fix_all.patch`, the `fix2` branch, and the later Sol-assisted patch series.

### 2. Measure the preliminary MTP estimate at a fitted context

The server performs a preliminary target fit before estimating MTP memory.

**Files:** `tools/server/server-context.cpp`

Previously, the MTP estimate could be measured at the full requested context,
often 262,144 tokens, even though the target could only fit a smaller context.
The preliminary fit obtains a realistic target context and uses it for the MTP
measurement, avoiding an inflated reservation.

This was introduced in the early three-fix sequence documented in
[`old/summary_qwen.md`](old/summary_qwen.md), and was subsequently refined in
`sol2` and later patches.

### 3. Count MTP memory according to the actual runtime allocation

MTP fitting reserves the memory that the runtime really allocates rather than a misleading standalone estimate.

**Files:** `tools/server/server-context.cpp`

The early `fix_all`/`fix2` implementation treated the MTP compute buffer as
shared with the target and excluded it from the estimate. Later testing showed
that the complete MTP context and compute allocation must be represented in the
target reservation. The final Eaman patch measures and adds the complete
MTP allocation per device.

This distinction is important: the early patch restored context sizing for the
tested case, but the final patch is the authoritative memory-accounting model.

### 4. Refine the MTP reservation after the target fit changes

MTP memory is measured again after fitting and the target fit is repeated with the refined reservation.

**Files:** `tools/server/server-context.cpp`

MTP memory depends on the target context, while the target context depends on
the MTP reservation. `sol2` introduced a bounded iterative procedure:

1. Fit the target with the current MTP reservation.
2. Measure MTP at the newly fitted context.
3. Replace the reservation with the measured values.
4. Repeat once and perform the final target fit.

The bounded approach resolves most of the dependency without introducing an
open-ended fitting loop.

### 5. Fit explicitly fixed multi-GPU layouts per device

Fixed layer/tensor placements are fitted against each GPU's individual memory limit instead of only a generic aggregate allowance.

**Files:** `common/fit.cpp`

When the user explicitly fixes `n_gpu_layers` and uses layer splitting with a
tensor split, the fitter cannot freely redistribute layers. The Eaman patch fitter
therefore calculates the context supported by each selected device, applies the
configured margin, rounds to the required 256-token boundary, and selects the
limiting device.

Automatic GPU-layer placement retains the existing conservative behavior.

This is the principal fitting change in [`old/patches/sol2.patch`](old/patches/sol2.patch).

### 6. Keep MTP scheduling non-pipeline

MTP fitting and runtime initialization use a disabled pipeline scheduler.

**Files:** `common/speculative.cpp`, `src/llama-context.cpp`, `tools/server/server-context.cpp`

On the tested asymmetric two-GPU system, MTP pipeline scheduling created
unnecessary scheduler and compute allocations. The MTP context now uses the
same explicit non-pipeline policy during both estimation and runtime creation,
so fitting does not select a cheaper layout than the one used at execution time.

The target context was not globally forced off by this change; target pipeline
behavior became an explicit user-selectable option in the next stage.

### 7. Add explicit target pipeline-parallel control

The target context gains `--pipeline-parallel auto|on|off`, with `auto` remaining the default.

**Files:** `include/llama.h`, `src/llama.cpp`, `src/llama-context.cpp`,
`common/common.h`, `common/common.cpp`, `common/arg.cpp`

The option exposes pipeline policy through the public context parameters and
the common command-line parser:

```text
--pipeline-parallel auto|on|off
```

`auto` preserves normal llama.cpp behavior; `on` requests pipeline scheduling;
`off` disables it. The tested ROCm launcher selects `off` because the single-slot
asymmetric workload gained context from avoiding duplicated scheduler workspace.

This change is the main addition in [`old/patches/sol3.patch`](old/patches/sol3.patch).

### 8. Dispatch supported HIP quantized-KV attention shapes to VEC

Supported HIP quantized-KV flash-attention shapes use VEC dispatch to avoid the larger F16 conversion workspace used by the fallback path.

**Files:** `ggml/src/ggml-cuda/fattn.cu`

The final `sol4` patch adds HIP VEC dispatch for the supported quantized K/V
formats and shapes. This allows in-register dequantization for those paths and
can remove a context-sized temporary F16 conversion allocation.

The dispatch is deliberately limited by supported formats, head dimensions,
and attention shapes. It is not a general guarantee that every quantized-KV
flash-attention operation uses VEC. The initial Q4/Q4 MTP run reached 147,712
tokens, but long-context stability, delayed-OOM, throughput, and acceptance
testing remain separate validation work.

The canonical current patch is [`sol4.patch`](sol4.patch), based historically on
llama.cpp commit `7bd8282`. The rebased equivalent is
[`latest_rocm_improvement_5f754ea.patch`](latest_rocm_improvement_5f754ea.patch),
based on mainline commit `5f754ea`.

## Historical patch lineage

| Stage | Reference | Main purpose | Modified source files |
|---|---|---|---|
| Initial fix | [`old/fix.patch`](old/fix.patch) | Runtime MTP context and preliminary MTP sizing fixes | `common/speculative.cpp`, `tools/server/server-context.cpp` |
| Three-fix version | [`old/fix_all.patch`](old/fix_all.patch) | Adds the early MTP compute-accounting fix | Same two files |
| `fix2` | llama.cpp commit [`e4e11e6`](https://github.com/ggml-org/llama.cpp/commit/e4e11e637518151ed0fe6f5e1ef64426b1e3d2dd) | Early patched version plus `[Q6-fit]` diagnostic logging | `common/fit.cpp`, `common/speculative.cpp`, `tools/server/server-context.cpp` |
| Sol-assisted 2 | [`old/patches/sol2.patch`](old/patches/sol2.patch) | Per-device fitting, MTP refinement, and non-pipeline MTP | 4 files |
| Sol-assisted 3 | [`old/patches/sol3.patch`](old/patches/sol3.patch) | Public target pipeline control | 9 files |
| Sol-assisted 4 | [`sol4.patch`](sol4.patch) | Sol-assisted 3 plus HIP VEC dispatch | 10 files |

The `fix2` branch is based on an older llama.cpp mainline (`b10150` ancestry),
not current `master`. Relative to the local `latest` mainline reference, its
custom delta is limited to `common/fit.cpp`, `common/speculative.cpp`, and
`tools/server/server-context.cpp`.

## Final source-file inventory

These are the files changed by the complete `sol4.patch` relative to its
historical mainline base:

| File | What our change does |
|---|---|
| `common/arg.cpp` | Parses `--pipeline-parallel auto|on|off`. |
| `common/common.cpp` | Transfers the common pipeline setting to context parameters. |
| `common/common.h` | Stores the common pipeline setting. |
| `common/fit.cpp` | Fits fixed multi-GPU layouts using per-device limits. |
| `common/speculative.cpp` | Uses fitted MTP context sizing and disables MTP pipeline scheduling. |
| `ggml/src/ggml-cuda/fattn.cu` | Dispatches supported HIP quantized-KV attention to VEC. |
| `include/llama.h` | Exposes the public pipeline-parallel context parameter. |
| `src/llama-context.cpp` | Applies pipeline policy when creating schedulers. |
| `src/llama.cpp` | Provides pipeline-mode naming/support helpers. |
| `tools/server/server-context.cpp` | Measures, refines, and reserves complete MTP memory. |

## Upstream references and relationship

These links are related upstream work or problem reports. They are references
for comparison, not claims that the Eaman patch is identical to or copied from
those changes.

- [Issue #23903](https://github.com/ggml-org/llama.cpp/issues/23903) — MTP draft-path buffer allocation after backend-sampling changes; this was the early context for the investigation.
- [Issue #26038](https://github.com/ggml-org/llama.cpp/issues/26038) — excessive ROCm MTP compute reservation; directly related to the memory-accounting problem.
- [Issue #25408](https://github.com/ggml-org/llama.cpp/issues/25408) and [PR #25465](https://github.com/ggml-org/llama.cpp/pull/25465) — upstream speculative-context fitting work; useful for comparing future fitting refactors.
- [PR #21830](https://github.com/ggml-org/llama.cpp/pull/21830) — upstream HIP flash-attention work for quantized KV; related to the VEC path, but not recorded here as the source of the Eaman implementation.
- [PR #22094](https://github.com/ggml-org/llama.cpp/pull/22094) — HIP flash-attention temporary-allocation behavior; relevant to the workspace measurements.
- [Issue #19036](https://github.com/ggml-org/llama.cpp/issues/19036), [Issue #23873](https://github.com/ggml-org/llama.cpp/issues/23873), and [Discussion #21526](https://github.com/ggml-org/llama.cpp/discussions/21526) — quantized-KV flash-attention memory behavior and ROCm/Vulkan differences.
- [Discussion #20252](https://github.com/ggml-org/llama.cpp/discussions/20252) — pipeline-parallel behavior and tradeoffs.

If a future patch is copied or adapted directly from an upstream PR, add the
PR number to the relevant change above and record the source commit or hunk;
the current VEC entry is intentionally marked as related work rather than a
direct copy.

## Not modifications

The following material is investigative or proposed, not part of the final
source delta:

- The backend files listed in [`old/summary_qwen.md`](old/summary_qwen.md) as
  root-cause locations were inspected but not changed by the Eaman patch.
- [`old/possible_improvements.md`](old/possible_improvements.md) contains
  future ideas such as broader VEC coverage, recurrent-state reductions, and
  further allocator work.
- `rocm_improvement.patch` is a duplicate of `sol4.patch`, not an additional
  modification.
- `latest_rocm_improvement_5f754ea.patch` is a rebased equivalent of the same
  complete patch, not a separate feature set.

## Patch identity

- Historical base: `7bd8282`
- Historical Eaman patch solution commit: `e37aae4`
- Current `eaman` branch tip: `773364c` (mainline merge `d00e998` plus MoE fitter fix)
- Rebased mainline base: `5f754ea`
- Complete historical patch checksum: `afc4216b15e833712c9c9228b41a1134d6f703614f6aec334c73e53f2ef46839`
- Rebased patch checksum: `5b653a6aaa50fd30cfc05cb9cf50b7a75adc491395c285b587f7e5b237b847c9`