--- title: "Evaluation methodology" description: "How NoemaAI evaluates its models: paired comparisons with the base model, identical prompts and graders, stated presets and output limits, and confidence intervals." version: "Noema 1.5 ยท Noema 2B" applies_to: "Model pages and model cards" reviewed: "2026-10-23" canonical: "https://noemaai.com/docs/evaluation-methodology" --- # Evaluation methodology Paired comparisons with each model's base, identical prompts and graders, stated presets and limits, and 95% intervals. Every number on a Noema model page comes with the setting it was measured under. This page collects the rules those evaluations follow; the [model pages](https://noemaai.com/models) carry the full tables, intervals and limitations. ## Compare each model with its own base Each model is compared with the model it was trained from, on identical items, with the same prompts, answer extraction, graders, sampling settings and output limits. Noema 1.5 4B is compared with Qwen3.5-4B. Noema 1.5 2B is compared with stock Qwen3.5-2B in its final evaluation and with Noema 2B in development gates. Noema 2B is compared with Qwen3.5-2B. Scores that other labs publish for their own models appear on model pages only as labeled context. They were not produced in NoemaAI's harness, and prompts, modes, token budgets and graders may differ. ## Report paired differences with intervals A difference is paired: the Noema model's score minus the base model's on the same items, with a 95% confidence interval from 10,000 bootstrap resamples. A difference counts as a gain or a loss only when its interval excludes zero. Small benchmarks such as AIME 2026 (30 problems) and HMMT February 2026 (33) have intervals several points wide. The Noema 1.5 2B final evaluation also reports paired significance tests and used a preregistered non-inferiority test for MMLU-Pro knowledge retention. ## State the setting with every score A score travels with its mode, decoding preset and output limit. Noema 1.5 4B is reported under three presets: | Preset | Thinking | Temperature | Top-p | Top-k | Presence penalty | | --- | --- | --- | --- | --- | --- | | General reasoning (recommended default) | On | 0.6 | 0.95 | 20 | 0 | | Qwen's general thinking preset | On | 1.0 | 0.95 | 20 | 1.5 | | Direct answers, chat and structured output | Off | 0.7 | 0.8 | 20 | 1.5 | Output limits follow Qwen's guidance for the base model: 81,920 tokens for competition mathematics and programming, and 32,768 for other thinking tasks. The loop stress test uses greedy decoding with a 32,768-token limit. ## Count unfinished answers as wrong An answer still being written at the output limit is scored as wrong, for every model alike. Token means count those answers at the limit, which understates what the base model would have spent if allowed to finish. Where NoemaAI's measurement of a base model differs from the score its publisher reports, the model page shows both and explains the difference. ## Keep evaluation data out of training - The Noema 1.5 4B benchmark panel and loop stress test were held out from training and checkpoint selection. Its training data was 13-gram decontaminated against every evaluation set. - The Noema 1.5 2B final comparison was run once, after the release candidate was frozen. Development gates informed checkpoint selection but are not part of that final evaluation. - Noema 2B's training data was 8-gram decontaminated against the evaluation subsets. The Noema 1.5 2B and Noema 2B results used no retrieval, external tools or inference-time answer repair; the measured changes are in the weights. ## How each release was evaluated ### Noema 1.5 4B Every comparison against Qwen3.5-4B is paired on identical items, with the same prompts, answer extraction, graders, sampling settings and output limits. The benchmark panel and the loop stress test were held out from training and checkpoint selection. Full results: [Noema 1.5 4B](https://noemaai.com/models/noema-1-5-4b). - One harness for every model: same prompts, answer extraction and graders - BF16 weights served by vLLM 0.30.0; 4-bit rows use Q4_K_M under llama.cpp - Qwen's recommended output limits; an answer still running at the limit counts as wrong - AIME, HMMT and arxivmath avg@16; LiveCodeBench avg@4; GPQA avg@8 - Paired 95% intervals from 10,000 bootstrap resamples of the items - Training data 13-gram decontaminated against every evaluation set - Checkpoints chosen on development sets alone, by rules written down in advance ### Noema 1.5 2B The final comparison was run once, after the release candidate was frozen, with no score inspection before both arms were complete. No retrieval, external tools, answer repair or outside model calls were used to produce benchmark answers. Full results: [Noema 1.5 2B](https://noemaai.com/models/noema-1-5-2b). - Frozen model and tokenizer revisions - Identical prompts and settings across both arms - BF16 inference with vLLM 0.19.1 on NVIDIA L40S hardware - Pinned official graders - Paired bootstrap intervals and McNemar tests - Deterministic keys and exact-count verification ### Noema 2B Every number is non-thinking, measured on a frozen, seed-pinned harness. All gains are in the weights: no retrieval, tools or inference-time scaffolds. Full results: [Noema 2B](https://noemaai.com/models/noema-2b). - Frozen harness: transformers 5.12.1, BF16, A100-class hardware, seed 1234 - EvalPlus and MMLU-Pro on the standard EvalPlus harness with Q8_0 GGUF builds - Identical prompts for Noema 2B and its base - Training data 8-gram decontaminated against the evaluation subsets ## Losses are reported Each model page lists where the model trails its base and what was not evaluated, such as languages other than English, image input and long-document understanding. Noema 1.5 results come from one training run per model. ## Related documentation - [Noema models](https://noemaai.com/docs/models) - [Prompt format and sampling](https://noemaai.com/docs/prompt-format) - [Run Noema models](https://noemaai.com/docs/run-noema-models)