Batch evaluator

​

What you’ll do

​

What you’ll need

• Install the SDK globally
    npm install -g @learning-commons/evaluators
  
• Create a CSV file with the text you want to evaluate
  • Must be 50 or fewer input rows (unless using the --bypass-row-limit option)
  • Must have text and grade columns
  • May include additional columns (will be preserved as-is in the output)

text,grade
"The cat sat on the mat.",3
"Photosynthesis is the process by which plants convert sunlight into energy.",5
"The mitochondria are the powerhouse of the cell.",8

​

Running the batch evaluator

npx evaluators-batch

• CSV file path
• Google and OpenAI API keys
  • Copy and paste directly in terminal window
  • Alternatively, provide as environment variables (GOOGLE_API_KEY and OPENAI_API_KEY, by default)
• Output directory
  • Defaults to a folder in the current directory with a human-readable timestamp (e.g. batch-results-2024-02-07_14-30-22/)

​

Options

evaluators-batch --concurrency 5 --max-retries 3 --no-telemetry

​

Results

Processing evaluations...
████████████░░░░░░░░ 60% (30/50)
  ✓ grade-level-appropriateness: 6/10 successful
  ✓ subject-matter-knowledge: 6/10 successful
  ✓ vocabulary: 6/10 successful
  ✓ sentence-structure: 6/10 successful
  ⏳ conventionality: 6/10 successful

⏱  Elapsed: 2m 15s | Estimated remaining: 1m 30s

batch-results-2024-02-07_14-30-22/
├── results.csv
└── results.html

• Spreadsheet-compatible format
• Original CSV columns preserved
• New CSV columns for each evaluator
  • {evaluator}_score
  • {evaluator}_reasoning
  • {evaluator}_status

• Summary dashboard with grade-level distribution and text complexity charts
• Scores and reasoning for each evaluator

​

Graceful shutdown

• In-flight evaluations finish processing
• Pending tasks are cancelled
• Completed results are saved to results-partial.* files to preserve progress

⚠️  Shutdown requested. Saving partial results...
   (Press Ctrl+C again to force quit)

✓ Saved 15 results to:
  ./batch-results-2024-02-07_14-30-22/
    ├── results-partial.csv
    └── results-partial.html