Core concepts

How evaluators are designed

Concept	Definition
Evaluator	Evaluates AI-generated content by focusing on one dimension of a rubric Example: The Vocabulary evaluator focuses on one dimension (vocabulary) of the Qualitative Text Complexity Rubric for Informational Text ↗
Dimension	Describes one specific attribute or criterion of AI-generated content Example: Evaluators can look at Vocabulary, Sentence Structure, and other dimensions of the text when evaluating its complexity.
Rubric	Provides a framework for evaluating a concept (e.g., Qualitative Text Complexity) through multiple dimensions Example: The Qualitative Text Complexity Rubric for Informational Text ↗ looks at dimensions like Vocabulary and Sentence Structure to make a judgment on a text’s complexity.
Evaluator family	Collection of evaluators that evaluate dimensions of one rubric Example: The Literacy evaluator family contains multiple evaluators that each assess one dimension of the Qualitative Text Complexity Rubric for Informational Text ↗.

How outputs are validated

Concept	Definition
Evaluator dataset	Dataset rigorously annotated by human domain experts according to a rubric These annotated datasets train and inform our “LLM as a judge” assessments, making our evaluators more accurate with every iteration. Example: We partnered with literacy experts to create the Literacy dataset, which we also used to develop our Literacy evaluators.
LLM as a judge	Evaluation method that automates human-like reasoning at scale Learning Commons uses this evaluation method to develop our evaluators. We prompt LLMs to use our expert-annotated evaluator datasets to validate our evaluators’ outputs.
Baseline prompt	Simulates what an edtech developer might construct quickly without expert-annotated datasets, structured rubric alignment, or prompt optimization Learning Commons uses baseline prompts to measure how much our expert-annotated evaluator datasets improve our evaluators’ performance.
Accuracy	Degree to which an evaluator aligns with our evaluator datasets, or how often an evaluator agrees with a human expert Learn more about How accuracy is measured and our recommended use cases.

For literacy evaluators

Concept	Definition
Tier 2	General academic words that appear across subject areas and are more common in writing than speech (e.g., vary, factors, determine) Students may have encountered these words, but are unlikely to have fully mastered them at lower grade levels.
Tier 3	Domain- or subject-specific words that are rare outside a particular field (e.g., precipitation, latitude, equator) Students are unlikely to know these words without prior instruction in that subject.
Archaic	Words that are outdated or no longer common in contemporary writing Students may be unfamiliar with these words, regardless of grade level.

How accuracy is measured

An evaluator’s accuracy is calculated on every evaluation run per input (single-run accuracy). Because LLM outputs are probabilistic and inherently non-deterministic, accuracy results may vary across evaluation runs (even with the same inputs).

To reduce your evaluators’ variability and increase their reliability in production:

Run the evaluator multiple times on the same input
Treat each output as a vote
Select the majority result (for example, 3 runs with majority selection)

Metrics

Not all evaluators will return all accuracy metrics.

Accuracy metric	Definition
Overall accuracy	Percentage of predictions that exactly match the expert-annotated evaluator dataset Some disagreements between our evaluators and human experts are expected, especially in qualitative domains and borderline cases.
Expert agreement rate	Percentage of reviewed cases where experts agree with evaluator output
Reasoning quality score	Expert rating of explanation quality (often on a numeric scale)
Baseline comparison accuracy	Performance relative to a minimal baseline prompt

Use cases

Use accuracy metrics to inform implementation decisions, not replace human judgment.

Recommended	Not recommended
Estimate expected agreement with expert judgment	Fully automate high-stakes decisions
Set appropriate thresholds for downstream logic	Assume consistent performance outside the validated scope
	Replace domain expertise where interpretation is critical

Datasets

See how expert-annotated datasets ground evaluator accuracy.

Literacy evaluators

Explore evaluators that assess qualitative text complexity.

Feedback evaluators

Explore evaluators that assess the quality of coaching feedback.

Standards evaluators

Explore evaluators that assess content alignment to academic standards.

Understanding evaluators

Getting started

SDK API Reference

Literacy evaluators

Feedback evaluators

Standards evaluators

Datasets

Resources

How evaluators are designed

Evaluator

Dimension

Rubric

Evaluator family

How outputs are validated

Evaluator dataset

LLM as a judge

Baseline prompt

Accuracy

For literacy evaluators

Tier 2

Tier 3

Archaic

How accuracy is measured

Metrics

Overall accuracy

Expert agreement rate

Reasoning quality score

Baseline comparison accuracy

Use cases

Datasets

Literacy evaluators

Feedback evaluators

Standards evaluators

​How evaluators are designed

​How outputs are validated

​For literacy evaluators

​How accuracy is measured

​Metrics

​Use cases

​Related topics

Datasets

Literacy evaluators

Feedback evaluators

Standards evaluators

How evaluators are designed

How outputs are validated

For literacy evaluators

How accuracy is measured

Metrics

Use cases

Related topics