Withholding Answers

Evaluator last updated Jul 29, 2026.

Overview

The Withholding Answers evaluator assesses whether a piece of teacher feedback guides the student toward a revision without doing the cognitive work for them:

Scaffolds through hints, guiding questions, or naming the kind of content or evidence that is missing
Doesn’t just supply the evidence, rewrite the response, or otherwise provide a complete or near-complete answer

The evaluator considers whether the feedback:

Points toward where to find evidence vs. supplying the actual evidence
Prompts the student to revise rather than rewriting or modeling the full response
Leaves core thinking (selecting, explaining, or wording the evidence) to the student

At a glance


Supported grades	6–12
Rubric	Productive Coaching rubric developed by Quill.org ↗ and Leanlab Education ↗, in partnership with Anastasiya A. Lipnevich ↗
Model used	GPT-5.4 (`gpt-5.4-2026-03-05`) with temperature of 1 The evaluator was built and validated using this specific model and temperature. Other configurations will produce different results and may have lower accuracy.

Getting started

Follow the Quickstart to start using this evaluator:

Access method
Evaluators playground	View in Platform ↗
SDK	Python ↗ and TypeScript ↗
Other	Python notebook ↗ Prompts: `system.txt` ↗ and `user.txt` ↗

Inputs

Inputs must be de-identified. Do not submit students' Personally Identifiable Information (PII) or any regulated or sensitive personal information.

Input	Description	Required
Student text	Student’s written response	Yes
Feedback text	Teacher- or AI-generated feedback to be evaluated	Yes

Output

Field	Description
Quality score	Binary judgment of 0/1 0 (Feedback gives away the answer) - Does the work for the student: supplying the specific evidence, rewriting the response, or otherwise providing a complete or near-complete answer 1 (Feedback withholds the answer) - Guides revision through hints, guiding questions, or by naming what kind of content or evidence is missing — without supplying it — or when the student’s response didn’t require revision and there was no answer to give away
Reasoning	High-level summary of why the feedback received this judgment
Key features	List of factors that contributed to the quality score (e.g., points toward evidence without supplying, prompts revision without rewriting, leaves core thinking to student), each with a 0/1 score and justification
Proposed adjustment	How the feedback could be modified to meet the withholding answers criterion (if applicable)

Example output

{
  "reasoning": "The feedback asks the student to explain and provide examples of why an AI pet costs less, but does not supply those examples or specific costs itself. The cognitive work — locating and articulating the evidence — is left to the student.",
  "key_features": {
    "points_toward_evidence_without_supplying": {
      "met": 1,
      "justification": "Asks the student to explain and give examples of why cost is lower, without naming the specific costs or quoting the text."
    },
    "prompts_revision_without_rewriting": {
      "met": 1,
      "justification": "Prompts the student to revise; it does not rewrite or model the response for them."
    },
    "leaves_core_thinking_to_student": {
      "met": 1,
      "justification": "Selecting, explaining, and wording the evidence is left to the student."
    }
  },
  "proposed_adjustment": "None needed for withholding-answers. The feedback scaffolds without completing the task.",
  "quality_score": 1
}

Interpreting results

Output	How to use it
Withholding answers score + Reasoning	Validate that your AI-generated feedback is coaching students through the work rather than doing it for them. Aggregate reasoning across runs to detect answer-giving drift — models tuned to be maximally helpful tend to drift toward supplying evidence or modeling full revisions as they’re tuned for helpfulness.
Key features + Proposed adjustment	Pinpoint exactly where a comment crosses from guiding into answer-giving. Adjust prompts to require the model to point toward what’s missing (the kind of evidence, or where to look) rather than supply it, and to withhold any full or near-full revision.

Evaluator release history

Date	Changed
Jul 29, 2026	First release

Feedback evaluators

Explore other evaluators that assess the quality of coaching feedback.

Quickstart

Run an evaluator in the Evaluators playground, a Python notebook, or with the SDK.

Understanding evaluators

Getting started

SDK API reference

Text Complexity evaluators

Graphics evaluators

Feedback evaluators

Academic standards evaluators

Durable skills evaluators

Datasets

Resources

Overview

At a glance

Getting started

Inputs

Output

Interpreting results

Evaluator release history

Feedback evaluators

Quickstart

​Overview

​At a glance

​Getting started

​Inputs

​Output

​Interpreting results

​Evaluator release history

​Related topics

Feedback evaluators

Quickstart

Overview

At a glance

Getting started

Inputs

Output

Interpreting results

Evaluator release history

Related topics