# ROCm MTP Context Sizing — Consolidated Bug Report

## Overview

Native MTP (`--spec-type draft-mtp`) on ROCm reserved disproportionately large compute workspace, reducing available context by ~50% initially, then ~21% after a first round of fixes. Three targeted patches restored context to near-baseline levels.

---

## Problem Statement

| Model | Quantization | Layers | Hardware |
|-------|-------------|--------|----------|
| Qwen3.6-27B | i1-Q4_XS / i1-Q6_K | 65 | ROCm0 (RX 6800, 16 GB) + ROCm1 (RX 6700 XT, 12 GB) |

**Symptom:** MTP draft context caused context to shrink far more than the actual memory it consumed at runtime.

---

## Evidence Timeline

### Phase 1: Initial Discovery (Q4_XS Model)

**With MTP (before any fix):**
```
MTP draft (n_ctx=262,144, 1 layer):
  KV cache:   288 MiB
  Compute:    1,108 MiB  (62 graph nodes, flash attention F16 K/V copies)
  → Maximum context: ~19.4K tokens
```

**Without MTP (baseline):**
```
Main context (n_ctx=19,456, 65 layers):
  KV cache:   456 MiB
  RS buffer:  449 MiB
  Compute:    101 MiB
  → Maximum context: ~37K tokens
```

### Phase 2: Q6 Regression (After Fix 1 + Fix 2)

Old build context: **139,776** tokens  
New build context: **110,080** tokens (lost ~29,696 / ~21%)

Same actual VRAM allocation, but fit algorithm received inflated estimates:

| Scenario | ROCm0 context | ROCm1 context | ROCm0 compute | ROCm1 compute |
|----------|---------------|---------------|---------------|---------------|
| Target only | 166 MiB | 99 MiB | 134 MiB | 164 MiB |
| Target + MTP | 446 MiB | 267 MiB | 134 MiB | 164 MiB |
| **MTP addition** | **+280 MiB** | **+168 MiB** | **+0 MiB** | **+0 MiB** |

### Phase 3: Final Verification (All 3 Fixes)

```
preliminary fit: n_ctx = 139,776
MTP estimate: 153.56 MiB (ctx only, compute skipped)
Final fit: n_ctx = 135,936
  → Within 0.2% of old build baseline (135,680) ✓
```

---

## Root Cause Analysis

### Why ROCm (not Vulkan)

**CUDA/HIP `get_alloc_size`** includes F16 K/V flash-attention workspace copies proportional to context length:

```
K (F16): n_ctx × head_dim × n_kv_heads × 2 bytes
V (F16): n_ctx × head_dim × n_kv_heads × 2 bytes
```

**Vulkan** returns `ggml_nbytes(tensor)` — flash attention temp buffers are pre-allocated on-demand during execution, not counted in measurement.

### Three Distinct Issues

#### Issue 1: MTP draft created with full `n_ctx`

`common/speculative.cpp` used the original requested `n_ctx` instead of the fit-reduced value. The draft context was oversized relative to what the target context actually got.

#### Issue 2: Pre-fit MTP estimate used full `n_ctx`

`tools/server/server-context.cpp` estimated MTP memory with `n_ctx=262,144`, adding ~1,396 MiB to `fit_params_target`. This forced the fit algorithm to shrink context aggressively, even though the actual MTP context would use the smaller fit-reduced `n_ctx`.

#### Issue 3: MTP estimate counted compute buffer

The memory measurement sums `context + compute` for a **standalone** MTP context. But at runtime, MTP draft **shares** the target's compute buffer — it allocates zero additional compute memory. The estimate counted ~1,076 MiB of compute that was never actually allocated.

---

## Fixes

### Fix 1: `common/speculative.cpp` — Runtime MTP draft uses target's fit-reduced n_ctx

```cpp
if (spec_mtp) {
    cparams.ctx_type = LLAMA_CONTEXT_TYPE_MTP;
    cparams.n_ctx = llama_n_ctx(ctx_tgt);  // fit-reduced n_ctx
}
```

Ensures MTP draft context matches the target's (possibly fit-reduced) context size at runtime.

### Fix 2: `tools/server/server-context.cpp` — Pre-fit estimate uses fit-reduced n_ctx

```cpp
if (spec_mtp) {
    cparams_dft.ctx_type = LLAMA_CONTEXT_TYPE_MTP;
    {
        auto mparams_tgt = common_model_params_to_llama(params_base);
        auto cparams_tgt = common_context_params_to_llama(params_base);
        common_fit_params(params_dft.model.path.c_str(), &mparams_tgt, &cparams_tgt,
            params_base.tensor_split,
            params_base.tensor_buft_overrides.data(),
            params_base.fit_params_target.data(),
            params_base.fit_params_min_ctx,
            GGML_LOG_LEVEL_ERROR);
        cparams_dft.n_ctx = cparams_tgt.n_ctx;
    }
}
```

Does a preliminary target-only fit to determine reduced `n_ctx`, then uses that for the MTP estimate. (Note: calls `common_fit_params()` twice; a two-pass design would be cleaner upstream.)

### Fix 3: `tools/server/server-context.cpp` — Skip compute buffer for MTP estimate

```cpp
const size_t bytes = (measure_model_bytes ? dmd[j].model : 0)
    + dmd[j].context
    + (spec_mtp ? 0 : dmd[j].compute);  // MTP shares compute, draft models don't
```

Only counts context memory for MTP since the compute buffer is shared with the target.

---

## Execution Flow (After All Fixes)

```
1. common_base_params_to_speculative(params_base) → params_dft
2. Fix 2: preliminary fit (target-only, n_rs_seq=3, margins=50 MiB/device)
   → cparams_tgt.n_ctx = 139,776
   → cparams_dft.n_ctx = 139,776
3. common_get_device_memory_data with cparams_dft (ctx_type=MTP, n_ctx=139,776)
   → MTP context = 153 MiB, MTP compute = 1,076 MiB (standalone measurement)
4. Fix 3: params_base.fit_params_target[i] += context only (skip compute for MTP)
   → ROCm1: 50 + 153 = 203 MiB (was 50 + 1,229 = 1,280 MiB)
5. common_init_from_params(params_base) → final fit with correct margins
   → cparams.n_ctx = 135,936
6. Fix 1: MTP draft created with n_ctx = llama_n_ctx(ctx_tgt) = 135,936
```

---

## Files Involved

| File | Role |
|------|------|
| `common/speculative.cpp:2275-2279` | Fix 1 — MTP draft uses fit-reduced n_ctx at runtime |
| `tools/server/server-context.cpp:1092-1106` | Fix 2 — pre-fit MTP estimate at fit-reduced n_ctx |
| `tools/server/server-context.cpp:1137-1141` | Fix 3 — skip compute buffer for MTP estimate |
| `common/fit.cpp:29-150` | `common_get_device_memory_data_impl()` — memory measurement |
| `common/fit.cpp:175-358` | `common_params_fit_impl()` — linear interpolation fit |
| `common/common.cpp:1588` | Context params conversion (n_ctx) |
| `ggml/src/ggml-cuda/ggml-cuda.cu:906` | CUDA/HIP `get_alloc_size` (FA workspace) |
| `ggml/src/ggml-cuda/fattn-common.cuh:53` | F16 workspace calculation |
| `ggml/src/ggml-cuda/fattn.cu:536` | Alloc size function |
| `ggml/src/ggml-vulkan/ggml-vulkan.cpp:15642` | Vulkan `get_alloc_size` (no FA workspace) |

---

## Patch Files

| File | Contents |
|------|----------|
| `fix.patch` | Original two-fix patch (superseded) |
| `fix_all.patch` | Combined three-fix patch (current, applies cleanly to main) |

---

## Deployment Notes

`llama-server` is a 17KB thin wrapper. The actual code lives in shared libraries:

- `libllama-server-impl.so` (~7.5 MB) — server logic including the patch
- `libllama-common.so.0.0.832` (~6 MB) — common/fit/speculative code

**Both `.so` libraries must be updated** when deploying a new build. Updating only `llama-server` has no effect.

---

## Conclusion

The `fix_all.patch` (3 fixes, 19 lines total) applies cleanly to `llama.cpp` main branch and restores MTP context sizing to near-baseline levels across both Q4_XS and Q6_K quantizations on multi-GPU ROCm setups.

**Related:** Issue #23903 (MTP draft path buffer allocation after backend-sampling changes).