Evaluators project banner logo

Try it in the Playground • Quickstart • Core concepts

Evaluators help you to measure the attributes of LLM-generated text through the lens of learning science.

We build learning-science-backed systems that follow LLM-as-a-judge methodology and can be directly integrated to your product or evaluation stack.

Use cases include:

• Feature optimization: Use fine-grained literacy evaluation to sharpen and consistently deliver a feature’s AI-generated content so it aligns with pedagogy and your goals.
• Maintaining performance: Ensure content is generated as expected by using the evaluators as product analytics for your LLM output.
• Model selection: Make a confident decision about which model is right for your product by testing the output of models you’re considering.

Evaluators and the supporting datasets are built in collaboration with leading literacy experts from Student Achievement Partners and the Achievement Network.

Check out the Evaluators docs for complete setup instructions and usage examples.

You can test the evaluators with your own text in the Evaluators Playground on the Learning Commons Platform.

To use the evaluators, clone the repository and follow the instructions below.

If you’d like to download or access our evaluators and datasets directly, follow the links below.

• Evaluators literacy package
  • Prompts
  • Notebooks
• Datasets
  • Learning Commons annotations of CLEAR for qualitative text complexity v1.0 2025年09月02日.csv

We rely on the Python interpreter to power the evaluators. All examples and tutorials are provided as Python code snippets.

You’ll need Python 3.10 or newer. To verify your version of Python, run the following code in the terminal:

python3 --version

Creating an isolated environment is a best practice that prevents conflicts between Python packages used in this project and others on your system.

python3 -m venv .venv
source .venv/bin/activate

Remember to activate the virtual environment for each new shell session when working with Evaluators.

The required packages are listed in the requirements.txt file.

pip install -r evals/requirements.txt

We are using both OpenAI and Google Gemini for different evaluators. You need API keys from both platforms:

• OpenAI: https://platform.openai.com/
• Gemini: https://aistudio.google.com/

Set the key(s) as environment variables in your shell session:

export OPENAI_API_KEY="sk-your-key-here"
export GOOGLE_API_KEY="your-key-here"

python --version

python -m venv .venv
.venv\Scripts\activate

python -m venv .venv
.venv\Scripts\Activate.ps1

pip install -r evals/requirements.txt

set OPENAI_API_KEY=sk-your-key-here
set GOOGLE_API_KEY=your-key-here

$env:OPENAI_API_KEY="sk-your-key-here"
$env:GOOGLE_API_KEY="your-key-here"

You are now ready to run the evaluator examples. We recommend using a Jupyter Notebook for interactive exploration.

1. Start Jupyter Notebooks Lab:

jupyter lab

Jupyter will open in your web browser (usually at http://localhost:8888).

2. Browse into the evals folder, then double click on the evaluator you want to try.
3. You can now copy the text you want to evaluate into the last code cell of the notebook to run an evaluator on your text sample.

If you prefer using an IDE with Python and Jupyter notebook support, such as VSCode with Microsoft's Python and Jupyter extensions, please refer to Microsoft's instructions for their installation and configuration.

We want to hear from you. For questions or feedback, please open an issue or reach out to us at support@learningcommons.org

Sign up for a Learning Commons account to receive news about the latest Evaluators updates, and releases.

If you believe you have found a security issue, please responsibly disclose by contacting us at security@learningcommons.org.

Use of the resources provided in this repository is subject to our Terms of Use.