> ## Documentation Index
> Fetch the complete documentation index at: https://docs.learningcommons.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitor quality in production

> Learn how to track evaluator scores on production outputs so you can catch gradual quality drift before it reaches students.

Monitor shifts in the quality of your models before they affect students at scale.

## What you'll do

* Compile a sample of text passages
* Use [Grade Level Appropriateness](https://docs.learningcommons.org/evaluators/text-complexity-evaluators/grade-level-appropriateness.md) to evaluate the passages' grade levels
* Make a model, prompt, or infrastructure change
* Run the evaluation again to see any grade level changes
  * If there are changes, use [Vocabulary Complexity](https://docs.learningcommons.org/evaluators/text-complexity-evaluators/vocabulary-complexity.md) to evaluate the passages' vocabulary

## What you'll need

* Node.js or Python
* [Google API key](https://aistudio.google.com/) ↗ (needed for Grade Level Appropriateness and Vocabulary Complexity)
* [OpenAI API key](https://platform.openai.com/) ↗ (needed for Vocabulary Complexity)
* Small sample of text passages

## Steps

**1. Install the SDK**

Install the package for the language your pipeline uses:

**TypeScript**

```shell TypeScript theme={null}
npm install @learning-commons/evaluators@latest
npm install @ai-sdk/google @ai-sdk/openai # Provider adaptors for the required API keys
```

**Python**

```shell Python theme={null}
pip install --upgrade learning-commons-evaluators
```

**2. Evaluate grade levels for your sample**

> [!WARNING]
>
> Inputs must be de-identified. Do not submit students' Personally
> Identifiable Information (PII) or any regulated or sensitive personal
> information.

Use [Grade Level Appropriateness](https://docs.learningcommons.org/evaluators/text-complexity-evaluators/grade-level-appropriateness.md) to evaluate the grade level of each passage in your sample.

```typescript TypeScript theme={null}
import { GradeLevelAppropriatenessEvaluator } from "@learning-commons/evaluators";

const sample = [
  {
    id: "passage-1",
    text: "Bees visit flowers to collect nectar. They carry it back to the hive and turn it into honey.",
  },
  {
    id: "passage-2",
    text: "Honeybees forage for nectar and convert it into honey through enzymatic processing inside the colony.",
  },
  {
    id: "passage-3",
    text: "A hive is a home for bees. The bees make honey and store it there.",
  },
];

const gradeLevel = new GradeLevelAppropriatenessEvaluator({
  googleApiKey: process.env.GOOGLE_API_KEY,
});

const scored = [];

for (const item of sample) {
  const { result } = await gradeLevel.evaluate({ text: item.text });
  scored.push({ id: item.id, gradeBand: result.grade_band });
  console.log(item.id, result.grade_band); // "K-1" | "2-3" | "4-5" | "6-8" | "9-10" | "11-12"
}
```

```python Python theme={null}
import os

from learning_commons_evaluators import GradeLevelAppropriatenessEvaluator

sample = [
  {
    "id": "passage-1",
    "text": "Bees visit flowers to collect nectar. They carry it back to the hive and turn it into honey.",
  },
  {
    "id": "passage-2",
    "text": "Honeybees forage for nectar and convert it into honey through enzymatic processing inside the colony.",
  },
  {
    "id": "passage-3",
    "text": "A hive is a home for bees. The bees make honey and store it there.",
  },
]

grade_level = GradeLevelAppropriatenessEvaluator(
  google_api_key=os.environ["GOOGLE_API_KEY"],
)

scored = []

for item in sample:
  result = grade_level.evaluate_sync(text=item["text"]).result
  scored.append({"id": item["id"], "grade_band": result.grade_band})
  print(item["id"], result.grade_band)  # "K-1" | "2-3" | "4-5" | "6-8" | "9-10" | "11-12"
```

Count how many passages sit above your target grade level – this will be your baseline count.

**3. Make product changes**

Implement a set of feature and updates in your product.

This could include model updates, prompt edits, shifted inputs, or even infrastructure changes.

**4. Rerun grade level evaluations on your sample**

With your product changes implemented, repeat the [Grade Level Appropriateness](https://docs.learningcommons.org/evaluators/text-complexity-evaluators/grade-level-appropriateness.md) evaluations on the same sample.

Count how many passages now sit above your target grade level, and compare this to your baseline count from the earlier step.

```typescript TypeScript theme={null}
const GRADE_BANDS = ["K-1", "2-3", "4-5", "6-8", "9-10", "11-12"] as const;
const TARGET_BAND = "4-5";

// Count saved from the previous run. Replace this with your baseline.
const baselineAboveTarget = 0;

const currentAboveTarget = scored.filter(
  (item) => GRADE_BANDS.indexOf(item.gradeBand) > GRADE_BANDS.indexOf(TARGET_BAND),
).length;

console.log(`${currentAboveTarget} of ${scored.length} passages are above ${TARGET_BAND}.`);

if (currentAboveTarget > baselineAboveTarget) {
  console.log("Drift flagged. Investigate before this sample's pattern reaches students.");
} else {
  console.log("No increase above the target band.");
}
```

```python Python theme={null}
GRADE_BANDS = ["K-1", "2-3", "4-5", "6-8", "9-10", "11-12"]
TARGET_BAND = "4-5"

# Count saved from the previous run. Replace this with your baseline.
baseline_above_target = 0

current_above_target = sum(
  1
  for item in scored
  if GRADE_BANDS.index(item["grade_band"]) > GRADE_BANDS.index(TARGET_BAND)
)

print(f"{current_above_target} of {len(scored)} passages are above {TARGET_BAND}.")

if current_above_target > baseline_above_target:
  print("Drift flagged. Investigate before this sample's pattern reaches students.")
else:
  print("No increase above the target band.")
```

If more passages sit above the target than last time, investigate the changes in your last release to see what may have caused this regression.

If the count rose after a prompt edit, see [Improve quality by refining prompts](https://docs.learningcommons.org/evaluators/getting-started/tutorials/improve-quality-by-refining-prompts.md).

> [!TIP]
>
> Store each new count with the date and product version (i.e. model version, prompt version, etc.), so a later shift has something to compare against.

**5. Monitor vocabulary complexity to anticipate grade level changes**

Vocabulary level often rises before the grade band does. As a result, you can treat passages' rising vocabulary complexity as a signal to investigate whether their grade level may also be rising soon.

Use [Vocabulary Complexity](https://docs.learningcommons.org/evaluators/text-complexity-evaluators/vocabulary-complexity.md) to evaluate your sample before and after product changes and compare how many passages are scored `very_complex` or `exceedingly_complex`.

```typescript TypeScript theme={null}
import { VocabularyComplexityEvaluator } from "@learning-commons/evaluators";

const HARD_SCORES = ["very_complex", "exceedingly_complex"];

const vocabulary = new VocabularyComplexityEvaluator({
  googleApiKey: process.env.GOOGLE_API_KEY,
  openaiApiKey: process.env.OPENAI_API_KEY,
});

let hardVocabulary = 0;

for (const item of sample) {
  const { result } = await vocabulary.evaluate({
    text: item.text,
    grade_level: "4",
  });

  console.log(item.id, result.complexity_score);
  console.log(result.tier_3_words);

  if (HARD_SCORES.includes(result.complexity_score)) {
    hardVocabulary += 1;
  }
}

console.log(`${hardVocabulary} of ${sample.length} passages have very or exceedingly complex vocabulary.`);
```

```python Python theme={null}
from learning_commons_evaluators import VocabularyComplexityEvaluator

HARD_SCORES = {"very_complex", "exceedingly_complex"}

vocabulary = VocabularyComplexityEvaluator(
  google_api_key=os.environ["GOOGLE_API_KEY"],
  openai_api_key=os.environ["OPENAI_API_KEY"],
)

hard_vocabulary = 0

for item in sample:
  result = vocabulary.evaluate_sync(text=item["text"], grade_level=4).result

  print(item["id"], result.complexity_score)
  print(result.tier_3_words)

  if result.complexity_score in HARD_SCORES:
    hard_vocabulary += 1

print(f"{hard_vocabulary} of {len(sample)} passages have very or exceedingly complex vocabulary.")
```

If a passage's `very_complex` or `exceedingly_complex` counts go up, even when its grade level has not, investigate your product changes now.

Waiting for the grade band to rise gives you less time to correct the issue and more chance of the regression reaching students.

## Related topics

> **[Resource Library - Monitor quality in production](https://learningcommons.org/resources/monitor-quality-in-production/)**
>
> Track scores on a sample of production outputs so you catch a shift before
> it reaches students.

> **[Quickstart](https://docs.learningcommons.org/evaluators/getting-started/quickstart.md)**
>
> Run an evaluator in the Evaluators playground, a Python notebook, or with
> the SDK.

> **[Tutorial: Vet content quality in production](https://docs.learningcommons.org/evaluators/getting-started/tutorials/vet-content-quality-in-production.md)**
>
> Score each output and rewrite anything above your grade-band target.

> **[Tutorial: Improve quality by refining prompts](https://docs.learningcommons.org/evaluators/getting-started/tutorials/improve-quality-by-refining-prompts.md)**
>
> Compare two prompt versions on grade band, vocabulary, and sentence
> structure.