27 lines
763 B
Markdown
27 lines
763 B
Markdown
# Homelab AI Evals
|
|
|
|
This directory contains deterministic eval cases for Jeannie and the local
|
|
homelab RAG workflow. The goal is to test whether explanations and retrieved
|
|
context point to useful commands, avoid dangerous actions, and stay grounded in
|
|
repo-owned runbooks.
|
|
|
|
Run the evals locally:
|
|
|
|
```bash
|
|
./jeannie ai-evals run
|
|
```
|
|
|
|
Useful commands:
|
|
|
|
```bash
|
|
./jeannie ai-evals list
|
|
./jeannie ai-evals show explain_capacity_missing_resources
|
|
./jeannie ai-evals run --case explain_capacity_missing_resources --details
|
|
./jeannie ai-evals run --json
|
|
```
|
|
|
|
The harness is intentionally read-only. It does not call an external model by
|
|
default and does not mutate the cluster. Cases use fixtures plus allowlisted
|
|
repo commands so they are suitable for Gitea Actions.
|
|
|