QA Engineer (AI + Python)

Role summary: Own quality on the two hardest surfaces of a production AI coaching platform:

the LLM agents that answer differently every run, and the API layer everything else depends on.

As AI & Backend QA Engineer you own the quality of what the model says and what the APIs return. On the AI side that means building the evaluation practice: golden sets, quality metrics, and regression detection that catch a degraded answer before a user does. On the backend side it means API test suites and consumer-driven contract testing with Pact — agreed with the client's US engineering team as a standing requirement for this role, not an optional technique.

The defining skill is localization: when an answer is wrong, you can tell whether the problem sits in the model, in the service code, or in the input data — and you bring the evidence, not a guess.

What you'll be testing:

• AI core — a Python service with LangGraph agents producing streaming, non-deterministic coaching responses; this
• role's primary surface
• Backend APIs — a NestJS REST gateway behind Auth0, plus the services behind it
• Mobile app — React Native, shipping to iOS and Android app stores; AI features surface directly in the app that
• members use

Must-have skills:

• 3+ years in technical QA/test automation, incl. API testing on production
• Python — writing/maintaining tests, pytest, fixtures
• HTTP/API fundamentals — REST, JSON, status codes, headers, auth tokens
• Contract testing with Pact — consumer-driven contracts (hard requirement)
• Security-aware API testing — auth boundaries, token handling, malformed/hostile input
• Comfort with non-determinism — assert on properties/contracts, not exact output
• LLM evaluation — datasets/golden sets, response metrics, detect regressions across prompts/models
• Prompt engineering — modify prompts and measure effects; aware of hallucinations and prompt injection
• Backend savvy — HTTP/REST, JSON, tokens, logs, DB queries; isolate model vs service vs input issues
• Postman and Swagger/OpenAPI daily
• Git in team workflow — branches, PRs, reviews
• TestRail for test-case management and runs
• Mobile product context (Android/iOS) — testing app features, release timing, store builds

Nice-to-haves:

• Langfuse or equivalent (LangSmith, Phoenix) — traces and attached evaluations
• LangGraph — test agents as state graphs (single node and full flow)
• CI/CD — run evaluations in pipelines and quality gates on PRs
• Python with Appium — mobile automation exposure alongside API work