Benchmark Leaderboard · Cross-Substrate Evaluation

K-Bench Leaderboard

Evaluating LLM unlearning across memory substrates and observable channels in agentic deployments — without misleading cross-substrate averages.

Paper (arXiv:2609.12808) ↗ Project Site ↗ Code (GitHub) ↗ Assets (HuggingFace) ↗

4 Memory Substrates Parametric (P), Context (C), Retrieval (R-text, R-struct)

6 Observable Channels CoT, tool query, wide query, answer, summary, all-channel OR

0 Cross-Substrate Averages Scores are reported per substrate; averaging erases failure modes

The Substrate Matrix

A single method earns fundamentally different verdicts on different memory substrates. Comparing rows reveals that unlearning is substrate-dependent.

Base Model:

Verdicts:

K-REF∞ Full suppression across channels

K-REF 10x ≥10× suppression

K-REF 2x ≥2× suppression

K-SUP Target suppressed, leaks elsewhere

MF Measured failure (<2× floor)

below gate The cell was evaluated, but the un-intervened baseline fails the substrate-validity gate on this base model, so no K-Score is defined.

no-op Indistinguishable from baseline

base Un-intervened baseline

— not evaluated The method was never run on this substrate, because its mechanism cannot act there.

Method	P Parametric (Weights)	C In-Context Memory	R-text Unstructured Retrieval	R-struct Structured Retrieval

💡

Why is the matrix ragged? A method is routed only to the substrates its mechanism can act on — weight editing reaches the parametric substrate, input-side interventions reach all four.

Linked method names open their source paper from the paper's own bibliography; NOISE and None are controls, not published methods, and carry no reference.

Per-Substrate Ranking

Rankings are computed strictly within each individual substrate and base model. Because each base model's K-Score floor depends on its own no-intervention baseline, absolute values are compared within a base model rather than across models, and are never combined across substrates.

Substrate:

Base Model:

Loading verified rows... Sorted by K-Score (Descending)

#	Method ↕	OR(all) ↕	Retain ↕	Δsel ↕	Degeneration % ↕	K-class ↕	K-Score ↓

Cross-Model Evaluation · Substrate P

Twenty published methods, three base models

Y-Axis Metric:

Gate Filter:

ranked on every base model passes the gate somewhere scored, gate not passed segment touching a collapse

Collapse Rule

How to read this chart

Gate Rule

⚖️

Reading Rule:

⚠️

Measurement Setup Disparity:

Quick Inspect Method (Click to pin / open):

Observation-Budget Recovery

Baseline forget-set recovery rate across nested attacker observation budgets (A₁ ⊆ … ⊆ A₅). Secrets suppressed in final answers frequently leak when reasoning traces or tool execution channels are observed.

How to Submit

Every score on K-Bench is backed by a hash-verified bundle to ensure reproducible evaluation.

Submissions go through a pull request against github.com/OniReimu/kbench under leaderboard/submissions/.

Each submission ships a hash-verified bundle built by kbench bundle and validated by kbench report against the frozen evaluation harness.

The Space displays merged results only. This strict verification gate guarantees that every row and verdict here can be independently checked and audited.

# 1. Evaluate your unlearning method on the frozen K-Bench benchmark

# 2. Package into a cryptographically sealed submission bundle

kbench bundle --method <METHOD_NAME> --model Llama-3.1-8B --out bundle.json

# 3. Verify retain utility, stability gates, and internal checksums

kbench report bundle.json

# 4. Open a Pull Request adding bundle.json to leaderboard/submissions/

View Submissions Directory ↗ Benchmark Repository ↗