AI Testing

Job Requirements

Role Overview

We are seeking a Lead Software Engineer – Automation Testing with 6–8 years of experience in designing, developing, and executing automation frameworks for AI/ML-driven applications. The ideal candidate brings hands-on experience with GenAI Testing, LangSmith, PyTest, Agentic frameworks, and strong AI/ML evaluation expertise.

Key Responsibilities

AI / GenAI Testing

• Design and execute V&V strategies for classical ML and GenAI/ agentic systems.
• Define test plans, tests cases, and execute them
• Evaluate AI models for accuracy, performance, safety, robustness, hallucinations, bias, and correctness.
• For GenAI, rely on LangSmith for LLM observability, tracing, monitoring, and evaluation, for offline and oline evaluation
• For classical ML UC, ensure V&V tests are done during the whole life cycle of the model (ML monitoring)
  
  

• Build scalable automation frameworks using PyTest integrated with CI/CD pipelines.
• Develop automated test suites for APIs, microservices, and AI pipelines.
• Perform sanity, integration, functional, regression, and UAT testing across QA, UAT, and Prod environments.
• Implement automation strategies: Keyword-driven, Data-driven, BDD, Hybrid frameworks.
  
  

• Analyze test results and provide insights to engineering teams.
• Log and track defects using standard bug-reporting tools.
• Maintain high-quality documentation for all test processes and reports.
  
  

• 6–8 years of experience in software testing & automation.
• Strong programming expertise in Python.
• Good understanding of Data Science related metrics (for both classical ML and GenAI UC)Experience with AI/Chatbot products and testing AI applications.
• Experience with Git, GitHub Copilot, and CI/CD pipelines.
• Knowledge of Azure/AWS cloud environments.
• Familiarity with MySQL, PostgreSQL, Oracle, SQL Server, NoSQL databases.
• Strong verbal & written communication skills.
• Experience with multi-stakeholder delivery models including onshore–offshore.
• Qualification: B.E./B.Tech
  
  

• Experience in AI/ML model evaluation.
• Exposure to AI observability tools and automated prompt validation.
• Hands-on expertise with LangSmith, Agentic frameworks, PyTest, BDD/V&V frameworks.
  
  

• Analytical & Critical Thinking
• Curiosity & Continuous Learning
• Strategic Communication(Active Listening, Technical Translations to business impacts)