Skip to main content

​
What you’ll need

To run our evaluators using any of our available methods, you must have:
  • Text you want to evaluate
  • Grade level of the intended audience
  • API keys required by the evaluators you want to use

​
Required API keys

By default, each evaluator uses a provider and model that we have tested for reliability for that task. As a result, different evaluators require different API keys.
EvaluatorRequired API key(s)
Grade Level Appropriateness ↗Google
Subject Matter Knowledge ↗Google
Vocabulary ↗Google and OpenAI
Sentence Structure ↗OpenAI
Conventionality ↗Google
Purpose ↗Google
You can override our evaluators’ defaults with any Google, OpenAI, or Anthropic model. When overriding with an Anthropic model, an Anthropic API key is required.

​
Running evaluators

How to run evaluatorsWhen to use
Evaluators PlaygroundFor a quick demo of how evaluators work
SDKTo integrate into your TypeScript or Python project
Python notebooksFor quick prototyping

​
Evaluators Playground

The Evaluators Playground on the Learning Commons Platform ↗ is the easiest way to see our evaluators in action. It evaluates text using all our literacy evaluators, and provides an output to help you:
  • Understand the pedagogical attributes of your text
  • Assess the performance of different prompts
  • Compare models for any AI-generated content in your products
The Evaluators Playground is primarily an introductory demo – we recommend the SDK or Python notebooks when implementing evaluators into your automated processes.

​
SDK

Run evaluators from your project by installing the SDK of your choice.
LanguageInstallationLatest version
TypeScriptGitHub ↗npm version
PythonGitHub ↗PyPI version
Contact us ↗ to request additional SDK language support. You can also sign up on the Learning Commons Platform ↗ for updates on availability.
Next, import evaluators to start evaluating text in your project: 
import { GradeLevelAppropriatenessEvaluator } from "@learning-commons/evaluators";

const evaluator = new GradeLevelAppropriatenessEvaluator({
  googleApiKey: process.env.GOOGLE_API_KEY,
});

const text = "The cat's out of the bag now.";

const result = await evaluator.evaluate(text);
console.log(result.score); // 4-5
See the SDK API Reference for more implementation details.

​
Python notebooks

The Python interpreter powers our evaluators. All downloadable examples ↗ and tutorials are provided as Python snippets.
Install Python 3.10 or newer. To verify your version of Python:
python3 --version
1

STEP 1: Create a virtual environment

Creating an isolated environment prevents conflicts between Python packages used in this project and others on your system:
python3 -m venv .venv

source .venv/bin/activate # Activates environment
Remember to activate the virtual environment for each new shell session.
2

STEP 2: Install dependencies

Install all required packages listed in requirements.txt ↗:
pip install -r evals/requirements.txt
3

STEP 3: Set your API keys

Set your API keys as environment variables in your shell session:
export OPENAI_API_KEY="sk-your-key-here"
export GOOGLE_API_KEY="your-key-here"
4

STEP 4: Run an evaluator

Start a Jupyter Notebook in your web browser (usually at http://localhost:8888):
jupyter lab
Create a new notebook by clicking the Notebook: Python 3 (ipykernel) tile in the Jupyter Lab launcher. Alternatively, select File > New > Notebook > Python 3 (ipykernel) in the menu.Run an evaluator by copying the code from our examples ↗ into the cells of your notebook.
You can also use an IDE with Python and Jupyter Notebook support (e.g. VS Code with Microsoft’s Python and Jupyter extensions).