Improve quality by refining prompts

Make data-driven and measurable improvements to your prompts.

What you’ll do

Get a passage generated by the current prompt and a passage generated by the revised prompt
Score both passages using Grade Level Appropriateness, Vocabulary Complexity, and Sentence Structure
Determine whether the revised prompt has improved your output’s quality

What you’ll need

Node.js or Python
Google API key ↗ (needed for Grade Level Appropriateness and Vocabulary Complexity)
OpenAI API key ↗ (needed for Vocabulary Complexity and Sentence Structure)
Current prompt and revised prompt, both for the same grade and task

Steps

Install the SDK

Install the package for the language your pipeline uses:

TypeScript
Python

TypeScript

npm install @learning-commons/evaluators@latest
npm install @ai-sdk/google @ai-sdk/openai # Provider adaptors for the required API keys

Python

pip install --upgrade learning-commons-evaluators

Score both prompt versions

Inputs must be de-identified. Do not submit students' Personally Identifiable Information (PII) or any regulated or sensitive personal information.

Score each prompt’s output with all 3 evaluators: Grade Level Appropriateness, Vocabulary Complexity, and Sentence Structure.

import {
  GradeLevelAppropriatenessEvaluator,
  SentenceStructureEvaluator,
  VocabularyComplexityEvaluator,
} from "@learning-commons/evaluators";

const promptA =
  "Bees visit flowers to collect nectar. They carry it back to the hive and turn it into honey.";
const promptB =
  "Honeybees embark on foraging expeditions, extracting nectar from blossoms and transporting it to the colony, where enzymatic activity transforms it into honey.";

const gradeLevel = new GradeLevelAppropriatenessEvaluator({
  googleApiKey: process.env.GOOGLE_API_KEY,
});
const vocabulary = new VocabularyComplexityEvaluator({
  googleApiKey: process.env.GOOGLE_API_KEY,
  openaiApiKey: process.env.OPENAI_API_KEY,
});
const sentence = new SentenceStructureEvaluator({
  openaiApiKey: process.env.OPENAI_API_KEY,
});

async function scorePassage(text: string) {
  const [grade, words, sentences] = await Promise.all([
    gradeLevel.evaluate({ text }),
    vocabulary.evaluate({ text, grade_level: "4" }),
    sentence.evaluate({ text, grade_level: "4" }),
  ]);

  return {
    gradeBand: grade.result.grade_band, // "K-1" | "2-3" | "4-5" | "6-8" | "9-10" | "11-12"
    vocabulary: words.result.complexity_score, // "slightly_complex" | "moderately_complex" | "very_complex" | "exceedingly_complex"
    sentence: sentences.result.complexity_score,
  };
}

const before = await scorePassage(promptA);
const after = await scorePassage(promptB);

console.log("Current prompt", before);
console.log("Revised prompt", after);

import os

from learning_commons_evaluators import (
  GradeLevelAppropriatenessEvaluator,
  SentenceStructureEvaluator,
  VocabularyComplexityEvaluator,
)

prompt_a = (
  "Bees visit flowers to collect nectar. They carry it back to the hive "
  "and turn it into honey."
)
prompt_b = (
  "Honeybees embark on foraging expeditions, extracting nectar from "
  "blossoms and transporting it to the colony, where enzymatic activity "
  "transforms it into honey."
)

grade_level = GradeLevelAppropriatenessEvaluator(
  google_api_key=os.environ["GOOGLE_API_KEY"],
)
vocabulary = VocabularyComplexityEvaluator(
  google_api_key=os.environ["GOOGLE_API_KEY"],
  openai_api_key=os.environ["OPENAI_API_KEY"],
)
sentence = SentenceStructureEvaluator(
  openai_api_key=os.environ["OPENAI_API_KEY"],
)

def score_passage(text):
  grade = grade_level.evaluate_sync(text=text).result
  words = vocabulary.evaluate_sync(text=text, grade_level=4).result
  sentences = sentence.evaluate_sync(text=text, grade_level=4).result

  return {
    "grade_band": grade.grade_band,  # "K-1" | "2-3" | "4-5" | "6-8" | "9-10" | "11-12"
    "vocabulary": words.complexity_score,  # "slightly_complex" | "moderately_complex" | "very_complex" | "exceedingly_complex"
    "sentence": sentences.complexity_score,
  }

before = score_passage(prompt_a)
after = score_passage(prompt_b)

print("Current prompt", before)
print("Revised prompt", after)

Determine whether the revised prompt should be shipped

Ship the revised prompt only if its output’s:

Evaluated grade level is at or below your target
Vocabulary Complexity and Sentence Structure scores have not gotten more complex

If the revised prompt is meant to produce a more difficult text, be sure to reverse the check and ensure that your Vocabulary Complexity and Sentence Structure scores HAVE gotten more complex.

const GRADE_BANDS = ["K-1", "2-3", "4-5", "6-8", "9-10", "11-12"] as const;
const COMPLEXITY = [
  "slightly_complex",
  "moderately_complex",
  "very_complex",
  "exceedingly_complex",
] as const;
const TARGET_BAND = "4-5";

const bandOk =
  GRADE_BANDS.indexOf(after.gradeBand) <= GRADE_BANDS.indexOf(TARGET_BAND);
const vocabularyOk =
  COMPLEXITY.indexOf(after.vocabulary) <= COMPLEXITY.indexOf(before.vocabulary);
const sentenceOk =
  COMPLEXITY.indexOf(after.sentence) <= COMPLEXITY.indexOf(before.sentence);

if (bandOk && vocabularyOk && sentenceOk) {
  console.log("Scores held. The revised prompt can ship.");
} else {
  console.log("A dimension moved the wrong way. Keep the current prompt.");
  console.log({ bandOk, vocabularyOk, sentenceOk });
}

GRADE_BANDS = ["K-1", "2-3", "4-5", "6-8", "9-10", "11-12"]
COMPLEXITY = [
  "slightly_complex",
  "moderately_complex",
  "very_complex",
  "exceedingly_complex",
]
TARGET_BAND = "4-5"

band_ok = GRADE_BANDS.index(after["grade_band"]) <= GRADE_BANDS.index(TARGET_BAND)
vocabulary_ok = COMPLEXITY.index(after["vocabulary"]) <= COMPLEXITY.index(before["vocabulary"])
sentence_ok = COMPLEXITY.index(after["sentence"]) <= COMPLEXITY.index(before["sentence"])

if band_ok and vocabulary_ok and sentence_ok:
  print("Scores held. The revised prompt can ship.")
else:
  print("A dimension moved the wrong way. Keep the current prompt.")
  print({
    "band_ok": band_ok,
    "vocabulary_ok": vocabulary_ok,
    "sentence_ok": sentence_ok,
  })

For any dimension that fails its check, change the prompt to address that evaluator’s reasoning.To monitor one dimension of a sample over time, see Monitor quality in production.

A single passage’s scores can move for reasons that have nothing to do with the prompt.Run several evaluations for each passage-dimension pair before moving forward.

Resource Library - Improve quality by refining prompts

Score outputs from before and after a prompt change on the same rubric so you can see what shifted.

Quickstart

Run an evaluator in the Evaluators playground, a Python notebook, or with the SDK.

Tutorial: Vet content quality in production

Score each output and rewrite anything above your grade-band target.

Tutorial: Monitor quality in production

Track grade band across a sample of outputs and catch a shift over time.

Understanding evaluators

Getting started

SDK API reference

Text Complexity evaluators

Graphics evaluators

Feedback evaluators

Academic standards evaluators

Durable skills evaluators

Datasets

Resources

What you’ll do

What you’ll need

Steps

Resource Library - Improve quality by refining prompts

Quickstart

Tutorial: Vet content quality in production

Tutorial: Monitor quality in production

​What you’ll do

​What you’ll need

​Steps

​Related topics

Resource Library - Improve quality by refining prompts

Quickstart

Tutorial: Vet content quality in production

Tutorial: Monitor quality in production

What you’ll do

What you’ll need

Steps

Related topics