Senior Lead Data Engineer - Azure Databricks

Blend is a premier AI services provider, committed to co-creating meaningful impact for its clients through the power of data science, AI, technology, and people. We help organisations solve complex business challenges by combining deep domain understanding with modern data and AI capabilities. Our teams work across strategy, analytics, engineering, and product delivery to create scalable, high-value solutions that improve decision-making, efficiency, and growth.

We are looking for an experienced Senior Data Engineer to support delivery of an Agentic Executive Scorecard - an AI-powered executive performance management capability delivering governed scorecards, recurring executive reporting and controlled conversational analytics. This role will be central to building the semantic layer and KPI model, including Databricks Metric Views, and the underlying data pipelines that give the solution a deep, trusted understanding of the data across different business domains in the client’s primary markets. The ideal candidate has strong hands-on experience in cloud data engineering and semantic/KPI modelling with Databricks Metric Views, and is comfortable working directly with business owners to define and build the KPIs that support AI-generated analytical narratives and conversational analytics.

Responsibilities

• Build and maintain the semantic layer and KPI model, including Databricks Metric Views, underpinning governed executive scorecards and recurring executive reporting
• Work directly with business owners to define, validate and build KPIs and the underlying business logic
• Profile source data quality, ownership, history, grain and reconciliation requirements across priority source systems
• Design and build data integration and transformation pipelines to prepare enterprise data for AI-generated narratives and conversational analytics
• Define conformed dimensions and market-specific variations across primary markets
• Work closely with the Business Analyst and Lead AI Engineer to translate KPI definitions and business logic into reusable data models
• Support the minimum viable business ontology and its integration with underlying data structures
• Ensure data quality, validation and monitoring across all data assets feeding the application
• Follow best practices for security, access control and governance aligned with agreed platform and AI governance requirements
• Contribute to technical documentation and knowledge transfer at the end of each delivery phase
• Support production readiness recommendations and deployment of solutions to production environments

• 5+ years of experience in Data Engineering, ideally supporting FMCG/CPG retail data (POS, SKU, category and market performance datasets)
• Hands-on experience with Databricks Metric Views (or equivalent semantic/metric layer tooling)
• Strong hands-on experience building semantic layers and KPI/metric models
• Proficiency in SQL and Python for data processing and transformation
• Experience with cloud data platforms (Azure Databricks) and modern ELT/ETL tooling
• Understanding of data modelling techniques, conformed dimensions, and Medallion-style architectures
• Experience profiling data quality, lineage, and reconciliation across multiple source systems
• Comfortable working directly with business owners and stakeholders to define, validate and build KPIs
• Understanding of business ontology/semantic modelling concepts
• Experience with Git version control
• Understanding of how data engineering supports AI/LLM-based analytics, including feature preparation for narrative generation and conversational analytics

• Prior experience working with FMCG/CPG clients on category, market share, or finance performance data
• Experience supporting agentic AI or LLM-powered analytics solutions
• Exposure to CI/CD pipelines (Azure DevOps)
• Familiarity with cloud security and RBAC in Azure and Databricks
• Experience working across multi-market data models