Detect LLM prompt regressions with Data Tables, OpenAI APIs, and Telegram

Quick overview Continuous quality checks for your production LLM prompts. Keeps a golden test set in n8n Data Tables, runs it on a schedule against any OpenAI-compatible API, compares each run to a blessed baseline, and pings Telegram only when quality actually changes. How it works A Schedule Trigger starts each run. The Execute workflow button runs it on demand. Prompt profiles (system prompt, model, temperature) and test cases (input, expected output, check type) load from two Data Tables. When both tables are empty the workflow seeds demo rows, so the first run works immediately. Every case calls its own model through a generic OpenAI-compatible HTTP request with retries. Failed calls are recorded and reported, never silently dropped. Deterministic checks (exact, contains, regex, numeric) grade most answers for free. An LLM judge scores only the free-form cases marked llm-judge on a 1-5 scale. The run is aggregated, snapshotted to an eval history table, and compared to the blessed baseline: a flipped case must fail two runs in a row to count, while a broad pass-rate or judge-score drop alerts at once.

Tags