Withholding Answers

Evaluator last updated July 29, 2026.

Overview

The Withholding Answers evaluator assesses whether a piece of teacher feedback guides the student toward a revision without doing the cognitive work for them:

Scaffolds through hints, guiding questions, or naming the kind of content or evidence that is missing
Doesn’t just supply the evidence, rewrite the response, or otherwise provide a complete or near-complete answer

The evaluator considers whether the feedback:

Points toward where to find evidence vs. supplying the actual evidence
Prompts the student to revise rather than rewriting or modeling the full response
Leaves core thinking (selecting, explaining, or wording the evidence) to the student

At a glance


Inputs	Teacher feedback on a student writing response
Supported grades	8–9
Rubric	Productive Coaching rubric developed by Quill.org ↗ and Leanlab Education ↗, in partnership with Anastasiya A. Lipnevich ↗

The evaluator was built and validated using the model and temperature below (other configurations will produce different results and may have lower accuracy):


Model used	GPT-5.4 (`gpt-5.4-2026-03-05`)
Temperature	1
Optimization	DSPy + GEPA (Genetic-Pareto) prompt optimization against expert-annotated labels

Getting started

Follow the Quickstart to start using this evaluator:

Access method
Evaluators playground	View in Platform ↗
GitHub	Python notebook ↗ Prompts (`system.txt` ↗ and `user.txt` ↗)

Inputs

Input	Description	Required
Student text	Student’s written response	Yes
Feedback text	Teacher- or AI-generated feedback to be evaluated	Yes

Output

Field	Description
Quality score	Binary judgment (1 / 0). A 1 applies when the feedback guides revision through hints, guiding questions, or by naming what kind of content or evidence is missing — without supplying it — or when the student’s response didn’t require revision and there was no answer to give away. A 0 signals feedback that does the work for the student: supplying the specific evidence, rewriting the response, or otherwise providing a complete or near-complete answer.
Reasoning	High-level summary of why the feedback received this judgment
Key features	One entry per factor driving the judgment (points toward evidence without supplying, prompts revision without rewriting, leaves core thinking to student), each marked met (1) / not met (0) with a justification
Proposed adjustment	Suggested moves to convert answer-giving into scaffolding, for developers iterating on prompts

Example output

{
  "reasoning": "The feedback asks the student to explain and provide examples of why an AI pet costs less, but does not supply those examples or specific costs itself. The cognitive work — locating and articulating the evidence — is left to the student.",
  "key_features": {
    "points_toward_evidence_without_supplying": {
      "met": 1,
      "justification": "Asks the student to explain and give examples of why cost is lower, without naming the specific costs or quoting the text."
    },
    "prompts_revision_without_rewriting": {
      "met": 1,
      "justification": "Prompts the student to revise; it does not rewrite or model the response for them."
    },
    "leaves_core_thinking_to_student": {
      "met": 1,
      "justification": "Selecting, explaining, and wording the evidence is left to the student."
    }
  },
  "proposed_adjustment": "None needed for withholding-answers. The feedback scaffolds without completing the task.",
  "withholding_answers_score": 1
}

Interpreting results

Output	How to use it
Withholding answers score + Reasoning	Validate that your AI-generated feedback is coaching students through the work rather than doing it for them. Aggregate reasoning across runs to detect answer-giving drift — models tuned to be maximally helpful tend to drift toward supplying evidence or modeling full revisions as they’re tuned for helpfulness.
Key features + Proposed adjustment	Pinpoint exactly where a comment crosses from guiding into answer-giving. Adjust prompts to require the model to point toward what’s missing (the kind of evidence, or where to look) rather than supply it, and to withhold any full or near-full revision.

Evaluator release history

Date	Changed
July 29, 2026	First release

Feedback evaluators

Explore other evaluators that assess the quality of coaching feedback.

Quickstart

Run an evaluator in the Evaluators playground, a Python notebook, or with the SDK.

Understanding evaluators

Getting started

SDK API reference

Student-Facing Text evaluators

Feedback evaluators

Academic standards evaluators

Durable skills evaluators

Datasets

Resources

Overview

At a glance

Getting started

Inputs

Output

Interpreting results

Evaluator release history

Feedback evaluators

Quickstart

​Overview

​At a glance

​Getting started

​Inputs

​Output

​Interpreting results

​Evaluator release history

​Related topics

Feedback evaluators

Quickstart

Overview

At a glance

Getting started

Inputs

Output

Interpreting results

Evaluator release history

Related topics