my-homelab-configs/infra/ai-evals/README.md

Homelab AI Evals

This directory contains deterministic eval cases for Jeannie and the local homelab RAG workflow. The goal is to test whether explanations and retrieved context point to useful commands, avoid dangerous actions, and stay grounded in repo-owned runbooks.

Run the evals locally:

./jeannie ai-evals run

Useful commands:

./jeannie ai-evals list
./jeannie ai-evals show explain_capacity_missing_resources
./jeannie ai-evals run --case explain_capacity_missing_resources --details
./jeannie ai-evals run --json

The harness is intentionally read-only. It does not call an external model by default and does not mutate the cluster. Cases use fixtures plus allowlisted repo commands so they are suitable for Gitea Actions.