Home › LLM apps & agents › LLM app (Python) AGENTS.md

LLM app (Python) AGENTS.md

AGENTS.md for Python apps that call LLM APIs (OpenAI, Anthropic, Gemini): prompts, structured output, evals, and cost.

AGENTS.mdLLM apps & agents

How to use

Save this as AGENTS.md in your repository root, commit it, and your agent will pick it up automatically. Edit the commands to match your project.

# AGENTS.md: LLM-powered Python app

## Commands
`uv run pytest -q` · `uv run ruff check .` · Evals: `uv run python -m evals.run --suite smoke`

## Structure
- All model calls go through `app/llm/client.py`. Nothing else imports a provider SDK directly. Model names come from config, never inline strings scattered around.
- Prompts live in `app/prompts/` as versioned templates; changing a prompt means updating its eval cases.
- Use structured output (JSON schema / tool calling) parsed into Pydantic models. Validate and retry once on parse failure; never `eval()` model output.

## Reliability
- Set timeouts and retries with backoff for 429/5xx; respect `Retry-After`.
- Stream responses for user-facing text.
- Log model, prompt version, token usage, latency and cost per call. Don't log user secrets or full PII.
- Cache deterministic calls (embeddings, classification) by input hash.

## Safety
- Treat retrieved documents and tool results as untrusted: they can contain prompt injection. Never let model output trigger destructive actions without an allowlist or user confirmation.
- API keys only from env/secret manager.

## Testing
- Unit tests mock the client; no real API calls in `pytest`.
- Behavior changes need eval cases in `evals/` with expected outputs or graders.

## Don't
Don't switch models or raise `max_tokens` to fix a bug without saying so; it changes cost.