Actionability

Evaluator last updated June 24, 2026.

Overview

The Actionability evaluator assesses whether a piece of feedback gives the student a clear, usable next step they can reasonably act on without additional clarification. The evaluator considers:

Presence of a directive verb (e.g., add, replace, clarify, explain, revise) or focused question that points somewhere specific
Whether feedback goes beyond evaluative comments (e.g., “good job,” “this is incomplete”)
Clarity of the target (is it clear what should be revised?)
Specificity of the next move (a concrete directive rather than a vague instruction)
Whether the student could reasonably act without further clarification or more information
Whether a next step is warranted at all (a response that didn’t require revision should not draw a forced directive)

At a glance


Input type	Teacher feedback on a student writing response
Supported grades	8–9
Rubric	Productive Coaching rubric developed by Quill.org ↗ and Leanlab Education ↗, in partnership with Anastasiya A. Lipnevich ↗

The evaluator was built and validated using the model and temperature below (other configurations will produce different results and may have lower accuracy):


Model used	GPT-5.4
Temperature	1.0
Optimization	DSPy + GEPA (Genetic-Pareto) prompt optimization against expert-annotated labels

Getting started

Follow the Quickstart to start using this evaluator:

Access method
Evaluators Playground	View in the Learning Commons Platform ↗
Python notebook	View in GitHub ↗
Prompts	View in GitHub: `system.txt` ↗ and `user.txt` ↗

Inputs

Inputs must be de-identified. Do not submit student PII or any regulated or sensitive personal information.

Input	Description	Required
Student text	Student’s constructed-response writing	Yes
Feedback text	Teacher or AI-generated feedback to be evaluated	Yes

Example input

{
  "student_text": "Some people think AI-powered pets are a good alternative to real pets because they could help around the house etc.",
  "feedback_text": "You're right, the AI pets could help around the house. Can you find some other details from the article that you could add to make your claim stronger?"
}

Output

Field	Description
Actionable revision score	Binary judgment (1 / 0). A 1 applies when feedback names a concrete action the student can take — add, replace, clarify, explain, ask, revise — or appropriately omits a next step when the response required no revision. A 0 signals feedback that is purely evaluative, too vague to act on, or too thin to direct revision.
Reasoning	High-level summary of why the feedback received this judgment
Key features	One entry per factor driving the judgment (directive verb or focused question, clarity of the target, specificity of the next move, whether the student could reasonably act without further clarification), each marked met (1) / not met (0) with a justification
Proposed adjustment	Suggested moves to strengthen the next step, for developers iterating on prompts or for teachers using the directive instructionally

Example output

{
  "reasoning": "The student gives a vague reason ('less of a responsibility') without specific text-based support, so revision is needed. The feedback affirms the idea and adds related context, but names no action the student could take — it does not provide a usable next step.",
  "key_features": {
    "directive_verb_or_focused_question": {
      "met": "no",
      "justification": "Affirms the student ('That's right') and adds a comment, but contains no directive verb or focused question prompting revision."
    },
    "clarity_of_target": {
      "met": "no",
      "justification": "Does not identify any part of the response to revise."
    },
    "specificity_of_next_move": {
      "met": "no",
      "justification": "Names no concrete action the student could take."
    },
    "reasonable_student_action": {
      "met": "no",
      "justification": "The student is left without a clear sense of what to change or add."
    }
  },
  "proposed_adjustment": "Add a directive verb or focused question. Example: 'That's a reason from the article. Can you add a specific example of what the article says makes AI pets less of a responsibility?'",
  "actionable_revision_score": "no"
}

Interpreting results

Output	How to use it
Actionable revision score + Reasoning	Evaluate whether your generated feedback is actually driving revision. Validate that your prompts produce feedback students can act on; aggregate reasoning across runs to detect drift toward describe-only, evaluative feedback
Key features + Proposed adjustment	Pinpoint and correct the specific failure. Require the model to include a directive verb or focused question before closing; surface 0-rated outputs to flag where students may need teacher follow-up

Accuracy and validation

This evaluator is provided as Early access. Reported metrics come from a small held-out test split (19 examples) with wide confidence intervals and should be read as directional. Validation testing is ongoing.

We assessed performance against Quill.org ↗ classroom writing data (45 labeled pairs; 13 train / 13 validation / 19 test) — expert-annotated student-response and teacher-feedback pairs labeled by Leanlab Education ↗ using the Productive Coaching rubric.

Metric	Description	Result
Accuracy	How accurately the evaluator’s actionable-revision score matches expert annotations on the held-out test split.	95% (naive baseline: 95%)
Macro-F1	Macro-F1 averaged over both classes on the held-out test split.	94% (naive baseline: 94%)
Model selection signal	Mean optimized performance of GPT-5.4 across the 5 feedback dimensions, which drove model selection.	GPT-5.4 had the best mean optimized macro-F1 (≈0.81) and accuracy (≈0.82) across the five feedback dimensions

On this dimension, GEPA optimization did not improve GPT-5.4 over its naive baseline on the held-out test set — both reached 95% accuracy. GPT-5.4 was selected for its strong mean performance across the full suite rather than its margin on this single dimension.

Evaluator release history

Date	Changed
June 24, 2026	First release

Understanding evaluators

Getting started

SDK API Reference

Literacy evaluators

Feedback evaluators

Standards evaluators

Datasets

Resources

Overview

At a glance

Getting started

Inputs

Output

Interpreting results

Accuracy and validation

Evaluator release history

​Overview

​At a glance

​Getting started

​Inputs

​Output

​Interpreting results

​Accuracy and validation

​Evaluator release history

Overview

At a glance

Getting started

Inputs

Output

Interpreting results

Accuracy and validation

Evaluator release history