The SDK provides specific error types to help you handle different scenarios.
import time
from learning_commons_evaluators import EvaluatorError, RateLimitError
for attempt in range(3):
try:
result = evaluator.evaluate_sync(input)
break
except EvaluatorError as e:
# Every EvaluatorError exposes a boolean retryable attribute
if not e.retryable or attempt == 2:
raise
delay = e.retry_after if isinstance(e, RateLimitError) and e.retry_after else 2 ** attempt # retry_after is in seconds
time.sleep(delay)
EvaluatorError
Evaluators can throw 3 types of EvaluatorErrors:
| Type | Description |
|---|
ConfigurationError | Missing or invalid API keys |
InputValidationError | Invalid input (text too short, invalid grade, etc.) |
APIError | Other API errors (Subtypes) |
APIError
APIErrors can be broken down further into more specific error types:
| Type | Description |
|---|
AuthenticationError | Invalid API keys |
RateLimitError | Rate limit exceeded; wait and retry |
NetworkError | Network connectivity issues |
RequestTimeoutError | Request timed out |
OutputValidationError | Output failed to parse or didn’t match the expected schema |
Retries
Every EvaluatorError exposes a boolean retryable attribute:
| Retryable by default | Not retryable by default |
|---|
RateLimitError | ConfigurationError |
NetworkError | InputValidationError |
RequestTimeoutError | AuthenticationError |
OutputValidationError | APIError with a 4xx status code |
APIError with a 5xx status code | |
To flag a specific instance (e.g., a permanently-bad hostname), APIErrors and NetworkErrors also accept retryable as an init kwarg. APIErrors with status code >= 500 and an unspecified retryable attribute will default to retryable: True.
