Skip to main content
Import and configure the evaluator of your choice.
example.py
from learning_commons_evaluators import (
  GoogleLLMProviderConfig,
  OpenAILLMProviderConfig,
  AnthropicLLMProviderConfig,
  ConventionalityEvaluator,
  create_config,
)

# Create provider config
google_config = GoogleLLMProviderConfig(api_key="...")
openai_config = OpenAILLMProviderConfig(api_key="...")
anthropic_config = AnthropicLLMProviderConfig(api_key="...")

# Create evaluator config
# NOTE: Telemetry is not yet implemented in v0.1.0
config = create_config(
  google_llm_provider_config=google_config,
  telemetry_partner_id="your-learning-commons-api-key",
)

# Instantiate evaluator
evaluator = ConventionalityEvaluator(config)

​
create_config methods

The SDK provides several methods to create a configuration object: 
from learning_commons_evaluators import (
  create_config, # Collects telemetry (Not yet supported)
  create_config_no_telemetry, # Does not collect telemetry
  create_config_telemetry_with_full_input, # Collects telemetry with input text (Not yet supported)
)
Each method takes in a list of configuration options.

​
Options

Our evaluators are validated against a particular provider and model during development. Evaluators default to that same provider and model at runtime – for example, Grade Level Appropriateness will always use Google Gemini out of the box, because it was validated against Google Gemini during development.As a result, each evaluator requires specific API keys when being configured (e.g., google_llm_provider_config, openai_llm_provider_config, etc.).We recommend using the validated provider and model, but the SDK does allow you to override that default in your per-instance settings or per-call settings. Please note that your milage may vary if you use a different model.
FieldTypeDescription
google_llm_provider_configGoogleLLMProviderConfigGoogle API key, required for evaluators that default to Google models
openai_llm_provider_configOpenAILLMProviderConfigOpenAI API key, required for evaluators that default to OpenAI models
anthropic_llm_provider_configAnthropicLLMProviderConfigAnthropic API key, required for evaluators that default to Claude or when overriding a default model with Claude
loggerLoggerCustom logger for evaluator output. Can set logging verbosity level.
telemetry_partner_id  Not yet supportedstringYour Learning Commons API key, used for authenticated and anonymous telemetry data collection. Required for create_config and create_config_telemetry_with_full_input only

​
Logging

Customize how your evaluator logs information.

​
Log level

Control logging verbosity: 
# SDK uses Python's standard logging module
import logging

# By default, EvaluatorConfig uses the package logger,
# which propagates to the root logger once your app configures handlers

logging.basicConfig(level=logging.DEBUG)
logging.getLogger("learning_commons_evaluators").setLevel(logging.WARNING)

from learning_commons_evaluators import (
    create_config_no_telemetry,
    create_logger,
    create_silent_logger,
)

logger = create_logger(level=logging.DEBUG)

# Discards logs + does not send as telemetry data
config = create_config_no_telemetry(logger=create_silent_logger())

​
Custom logger

You can configure your evaluator with a custom logger: 
import logging

from learning_commons_evaluators import (
  create_config,
)

# Route SDK logs through your own logger
my_logger = logging.getLogger("my_app.evaluators")
custom_logger_config = create_config(..., logger=my_logger)

​
Per-instance settings

Override the default for every call on a particular evaluator instance using default_evaluation_settings. Per-call settings will override per-instance settings for a single run. 
from dataclasses import replace

from learning_commons_evaluators import ConventionalityEvaluator

settings = ConventionalityEvaluator.default_evaluation_settings.model_copy(deep=True)
settings.prompt_settings_step_conventionality_evaluation = replace(
  settings.prompt_settings_step_conventionality_evaluation,
  temperature=0.2,
  model="gemini-2.5-pro",
)

evaluator = ConventionalityEvaluator(config, default_evaluation_settings=settings)

result = evaluator.evaluate_sync(input) # uses instance default

​
Per-call settings

Override models, temperatures, or other configurable evaluator settings for a single run using evaluation_settings: 
from dataclasses import replace

from learning_commons_evaluators import ConventionalityEvaluator

evaluator = ConventionalityEvaluator(config)
settings = evaluator.default_evaluation_settings.model_copy(deep=True)
settings.prompt_settings_step_conventionality_evaluation = replace(
  settings.prompt_settings_step_conventionality_evaluation,
  temperature=0.2,
)

result = evaluator.evaluate_sync(input, evaluation_settings=settings)
When omitted, the evaluator uses a deep copy of its default settings.