LLM Eval Harness

A modern evaluation harness for Large Language Models. It runs prompts against multiple providers, scores responses with configurable rubrics, and exposes the results in a React dashboard.

Features

• Multi-provider support for OpenAI, Anthropic, Google Gemini, and Groq.
• Automated scoring with configurable judge models.
• FastAPI backend with async scoring and database persistence.
• React + Vite frontend for building suites and reviewing results.
• PostgreSQL storage for suites, runs, and scores.

Getting Started

Prerequisites

• Python 3.10+
• Node.js and npm
• Docker
• API keys for the providers you want to use

Database Setup

docker run --name evaldb -e POSTGRES_PASSWORD=change-me -e POSTGRES_USER=postgres -e POSTGRES_DB=evaldb -p 5432:5432 -d postgres

Backend Setup

1. Create and activate a virtual environment.
2. Install dependencies:

   pip install -r requirements.txt
   

3. Create a .env file in the project root:

   DATABASE_URL=postgresql://postgres:change-me@localhost:5432/evaldb
   OPENAI_API_KEY=your_openai_api_key_here
   ANTHROPIC_API_KEY=your_anthropic_api_key_here
   GROQ_API_KEY=your_groq_api_key_here
   GEMINI_API_KEY=your_gemini_api_key_here
   

4. Initialize the database:

   python main.py --init-db
   

5. Run the API server:

   uvicorn api:app --reload --port 8000
   

Frontend Setup

1. Open a new terminal and navigate to frontend.
2. Install dependencies:

   npm install
   

3. Start the Vite dev server:

   npm run dev
   

4. Open http://localhost:5173.

Architecture

• api.py: FastAPI endpoints.
• evaluator.py: Core evaluation engine.
• scorers.py: Scoring logic.
• providers.py: Provider adapters.
• db.py: Async database helpers.
• frontend/: React single-page app.

Created with love by Hounderd.