One pipeline, eight runs. Confounds left in.

We took the graph GLM-5.3 built and answered from it with local qwen. That run scored 69.89 macro and 77.24 pooled, which is +13.63 over the baseline (p = 1.7e−30). The answer model, the sampling preset and the judge were the same in both runs. The graph was not: GLM-5.3 built it, and with curation batch 32 where the baseline used batch 1, so this step measures the ingest model and the batch size together. The code build also differs, v0.1.0 + 16 commits for the baseline and v0.1.0 + 23 commits for the GLM-graph run, though the product source is the same in both.

The last +2.46 up to GLM end to end is shown as unattributed (p = 1.7e−3). Three things change at once in that step: the answer model, the sampling preset, and whether the judge is grading its own model's answers. The build differs too (v0.1.0 + 23 commits against v0.1.0 + 17 commits), with the same product source. We cannot split the step between these.

The local qwen runs cost nothing in provider fees. GLM-5.3 cost $71.34, computed from the recorded token counts at Z.ai's published rates: $11.45 for ingest and $59.90 for answers. The judge ran locally in every arm.

The GLM-graph run is placed at $11.45 because it reused the vaults GLM built. It was not run end to end, so read that point as an estimate of what a GLM ingest plus local answers would cost. gpt-5.6-luna is not on the chart: it ran on a flat-rate subscription, so there is no per-call cost to plot.

After each run we re-traced every question offline through the graph and recorded the first stage where its evidence broke: no claim held it, recall did not return it, it was not cited, or it was cited and the answer was still judged wrong. These are trace-stage counts, not the score. A question can be judged correct and still count as a miss here, because the answer may come from other context, so "traced ok" is smaller than the judge-correct count (870 against 938 for qwen run 2).

On the three qwen runs, retrieval misses are the largest group: 452 to 475 of the 1,540 questions. The GLM-5.3 run has 189, which is 39.8% to 41.8% of each qwen run's count. The gpt-5.6-luna run has 228, 48.0% to 50.4%, so about half.

• Scope. LoCoMo categories 1 to 4, 1,540 questions per run. Category 5 is left out because its scorer accepts only two literal refusal phrases.
• Judge. qwen3.8-27b, served locally, temperature 0, with the Mem0/MemGPT-lineage judge prompt reproduced verbatim below (version label mem0-memgpt-v1). The judge is the same model as the answers in the qwen runs and the GLM-graph run, so those arms are self-judged. GLM-5.3, gpt-5.6-luna and the gemma runs are not. A question whose judge call returned no label is dropped from that run's denominator.
• Macro and pooled. Macro is the mean of the four per-category percentages. Pooled is correct answers over judged answers across all four categories. Pooled weights category 4 most, because it holds 841 of the 1,540 questions.
• Paired test. Exact two-sided McNemar on the discordant pairs. A question without a clean label in either run is excluded from that pair.
• Confounds. Builds differ between arms (see the table above). Sampling presets differ: qwen ran on an instruct preset, GLM-5.3, gpt-5.6-luna and gemma on the vendor default. gpt-5.6-luna ran with max_tokens 16,000 where the others used 32,768, and ingest curation settings differ: the GLM-5.3 graph (also used by the GLM-graph run) was built with batch 32, gpt-5.6-luna ingested with concurrency 8, and the other arms used batch 1, concurrency 1. Each ingest model mints a different number of claims (from 1,053 for qwen run 2 to 4,202 for gpt-5.6-luna), so the graphs differ in size. The bundle lists the confounds for every pair.
• Known defects. gemma4-26b thinking off lost 3 session documents of one conversation at ingest (a case-collision bug, fixed later). gemma4-26b thinking on had 2 of 272 ingest units time out. The GLM-graph run did no ingest of its own. One gpt-5.6-luna category 5 question failed at ask and was re-asked; category 5 is not scored here.
• Cost and tokens. GLM-5.3 cost comes from recorded token counts at Z.ai's published rates. The local runs have no provider cost; electricity and hardware are not counted. gpt-5.6-luna has no per-call price. Wall-clock time is not in the bundle.
• Code versions. Every arm was measured on a development build between v0.1.0 and v0.2.0, when the project was named Marginalia. Each is labelled as v0.1.0 plus the number of commits after that tag, from v0.1.0 + 8 commits to v0.1.0 + 44 commits; v0.2.0 is v0.1.0 + 59 commits. The three qwen runs sit on three different builds. No run has been made on 0.3.0, and these numbers do not describe it.
• Data. LoCoMo is licensed CC BY-NC 4.0, so the bundle carries only aggregates and one verdict label per question, never question or answer text. The harness that produced it is kept private for the same reason, so this page describes how to reproduce the method but does not link to the harness.

Judge prompt, verbatim

Your task is to label an answer to a question as "CORRECT" or "WRONG". You will be given
the following data: (1) a question (posed by one user to another user), (2) a 'gold'
(ground truth) answer, (3) a generated answer which you will score as CORRECT/WRONG.

The point of the question is to ask about something one user should know about the other
user based on their prior conversations. The gold answer will usually be a concise and
short answer that includes the referenced topic, for example:
Question: Do you remember what I got the last time I went to Hawaii?
Gold answer: A shell necklace
The generated answer might be much longer, but you should be generous with your grading -
as long as it touches on the same topic as the gold answer, it should be counted as
CORRECT.

For time related questions, the gold answer will be a specific date, month, year, etc. The
generated answer might be much longer or use relative time references (like 'last Tuesday'
or 'next month'), but you should be generous with your grading - as long as it refers to
the same date or time period as the gold answer, it should be counted as CORRECT. Even if
the format differs (e.g., 'May 7th' vs '7 May'), consider it CORRECT if it's the same date.

Now it's time for the real question:
Question: {question}
Gold answer: {gold_answer}
Generated answer: {generated_answer}

First, provide a short (one sentence) explanation of your reasoning, then finish with
CORRECT or WRONG. Do NOT include both CORRECT and WRONG in your response, or it will break
the evaluation script.

Just return the label CORRECT or WRONG in a json format with the key as "label".

This is the LLM-judge prompt from the Mem0 and MemGPT line of LoCoMo evaluations, used unchanged so the scoring rule is a known one. It is theirs, not ours.