evalship

evalship checks the quality of LLM evals rather than running them. It reviews an existing eval suite for flaws, such as a judge that scores a case 1.0 when the judge call fails, and flags AI calls in the code that have no eval.

Once connected to a GitHub repository it comments on pull requests, separating real regressions from flaky cases by comparing recent runs on the main branch. It works with existing eval frameworks and keeps eval history for 30 days to 2 years depending on the plan.

The Free plan covers public repositories and one private repository. Paid plans start at $19/month. It is run by Garm Tech B.V. in Amsterdam.