LLM Ops Lead

Build your future with Sovos.

At Sovos, we're transforming tax compliance through innovative technology and AI-powered solutions used by customers worldwide. We're looking for an LLM Ops Lead with 2-3 years of experience to help scale, govern, and operationalize Generative AI across the business while ensuring quality, safety, observability, and continuous improvement.

What You'll Do

• Own AI quality standards, evaluation frameworks, and release gates across Ask Sovi products
• Lead prompt and model versioning, regression detection, and AI governance initiatives
• Coordinate Safety, Guardrails, and Red Team testing programs
• Build observability practices across latency, cost, quality, and safety metrics using tools such as Langfuse
• Manage Jira epics, ticket triage, and cross-functional coordination across Product and Engineering teams
• Develop LLMOps playbooks, best practices, and operational standards
• Lead AI incident response, troubleshooting, and root-cause analysis
  
  

• 2-3 years of experience in Applied AI, LLMOps, AI Quality Engineering, AI Operations, Test Automation, or related fields
• Hands-on experience with Langfuse, LangSmith, or similar AI observability and evaluation tools
• Strong understanding of AI quality, evaluation pipelines, prompt versioning, and AI governance
• Experience with AI safety, guardrails, content moderation, audit logging, and release management
• Strong stakeholder management and communication skills
• Ability to support teams operating across UK business hours
  
  

• Applied AI Engineers from AI-native or AI-first organizations with hands-on Langfuse/LangSmith experience
• Product-minded Engineers who combine strong technical skills with customer and business impact awareness
• QA/Test Automation Engineers who have transitioned into AI evaluation, quality, or LLM testing
• AI Quality Analysts with expertise in hallucination detection, model drift monitoring, LLM-as-a-Judge methodologies, and AI quality governance
  
  

• Work on cutting-edge Generative AI products and platforms
• Collaborate with global AI, Product, and Engineering leaders
• Bi-Weekly Meeting Free Days
• Mentoring and career development programs
• Globally recognized training opportunities
• Comprehensive health and wellness benefits
• Flexible and collaborative work environment