Newpage - Quality Assurance Engineer

Job Description

We are hiring QA Engineers to validate that AI products do what they're meant to doand don't do what they shouldn't.

AI outputs aren't deterministic; testing them isn't either.

You will design golden-question suites, hunt hallucinations, verify evidence citations, and confirm guardrails hold under combinatorial pressure.

You will build coverage that survives an LLM swap, a prompt change, or a new retrieval index.

You sit close to the engineers building the systems you test, and you raise the quality bar by asking questions the model itself can't answer.

You treat AI as the substrate of the products you validate, and you stay current on how these systems break.

What You'll Do

Plan And Design :

• Build test plans for AI products across large combinatorial matrices : input variants, user roles, content states, deployment surfaces.
• Maintain golden-question test suites that catch regressions when models, prompts, retrieval, or content change.
• Design coverage for the things AI products specifically get wrong : hallucinations, citation drift, prompt injection, context bleed, guardrail bypass, role escalation, stale or unapproved content surfacing where it shouldn't.
• Translate ambiguous quality requirements into concrete, repeatable test casesespecially where audit trails, access controls, and content provenance matter.
  
  

• Run structured test passes across web UIs and APIs; investigate failures deeply enough to give engineers a useful starting point, not just a screenshot.
• Validate evidence citations point to the right sources and that "pending review" or restricted content stays excluded from AI responses where the rules require it.
• Verify role-based access at the UI and API layersusers see what they should and nothing more.
• Verify data integrity using read-only SQL queries; cross-check what the UI shows against what the database holds.
• Exercise APIs directly via Postman, Insomnia, or curl; read OpenAPI specs and form requests without help.
• File defects that are clear, reproducible, and rankedstructured enough that an engineer can act without a meeting.
  
  

• Maintain regression coverage as products evolve : prompts change, retrieval indexes update, models get swapped, content gets added.
• Partner with engineers on evaluation rubrics, golden datasets, and acceptance criteria for AI behaviourthen verify the criteria actually hold in production.
• Stay current on AI/LLM evaluation tooling and apply what's useful : golden datasets, regression rubrics, hallucination scoring, structured eval frameworks.
• Maintain test documentation that survives team turnover : clear, structured, and findable.
  
  

• 5+ years manual and automation QA experience in product environments, production-grade.
• Hands-on experience testing AI / LLM-based products : output validation, hallucination detection, edge-case coverage, prompt injection awareness.
• Strong test-planning skills for large combinatorial test matrices.
• Hands-on with test management tooling : TestRail, Zephyr Scale, Xray, or equivalent.
• API testing fluency : Postman or Insomnia; comfortable reading OpenAPI/Swagger specs and exercising endpoints directly.
• Browser-based debugging skills : Chrome DevTools (Network, Console, Application), cross-browser verification.
• Bug tracking and workflow tooling : Jira, Linear, or equivalent.
• SQL basics : read-only queries to verify data state in PostgreSQL or equivalent.
• Documentation discipline : clear, structured test plans and defect reports in Confluence, Notion, or equivalent.
• Comfort with enterprise content workflows, audit-trail verification, and role-based access checks.
• A clear-eyed view of where current AI tooling helps and where it falls short.
• Curiosity, persistence, and a willingness to dig past the surface symptom into the underlying cause.
  
  

• Understanding of MLOps, model serving, scaling and monitoring workflows (e.g., BentoML, MLflow, Vertex AI, AWS Sagemaker)
• Exposure to evaluation tools (Ragas, Promptfoo, DeepEval, LangSmith)read-only or interpretive use is fine.
• Automation experience with Playwright, Cypress, or equivalenteven if the role is primarily manual.
• Browser Stack or similar cross-browser/device platforms.
• GDPR-style data handling verification experience.
• Pharma, healthcare, or other regulated-industry exposure.
• Familiarity with EU AI Act or similar AI governance and compliance frameworks.
  
  

• A people-first culture - Supportive peers, open communication and a strong sense of belonging
• Smart, purposeful collaboration - Work with talented colleagues to create technologies that solve meaningful business challenges
• Balance that lasts - We respect your time and support a healthy integration of work and life
• Room to grow - Opportunities for learning, leadership and career development, shaped around you
• Meaningful rewards - Competitive compensation that recognises both contribution and potential