Lead Data Engineer

Job description:

Lead data engineer

Lead Data Engineer

Role Summary

We are seeking a highly skilled and experienced Lead Data Engineer to drive the modernization and design the future-state architecture of our Sales Validation platform. As a technical leader, you will be responsible for guiding the transition from legacy systems to advanced data engineering solutions, primarily focused on Databricks and Azure cloud technologies. This role requires a passion for innovation, performance optimization, and building scalable, reusable engineering frameworks.

Key Responsibilities

• Lead the architecture, implementation, and delivery of Databricks solutions to support data processing and analytics initiatives.
• Oversee the migration efforts from Synapse Analytics to Databricks, ensuring minimal downtime and aligned best practices.
• Develop and enforce engineering standards, reusable frameworks, and documentation to promote consistency and efficiency across teams.
• Mentor data engineers through design reviews, knowledge sharing sessions, and technical guidance to foster professional growth and code quality.
• Own end-to-end delivery of multiple complex workstreams, coordinate across teams, and manage technical risks.
• Drive continuous performance tuning and scalability improvements for data pipelines and platforms.
• Collaborate with cross-functional teams including data scientists, architects, and business stakeholders to deliver aligned solutions.

Core Competencies

• Client Centricity: Demonstrate a strong commitment to understanding and fulfilling client needs, ensuring solutions deliver measurable value.
• Passion for Results: Exhibit a persistent focus on achieving outcomes through dedication and proactive problem-solving.
• Execution Excellence: Deliver consistent quality and meet deadlines through disciplined planning and thorough validation.
• Collaborative Working: Foster teamwork and open communication across diverse stakeholders and disciplines.
• Learning Agility: Embrace change and continuously seek to expand knowledge and skills in evolving technologies.
• Problem Solving & Decision Making: Analyze complex issues, identify root causes, and implement effective, timely solutions.
• Effective Communication: Articulate ideas clearly and persuasively in both technical and non-technical contexts to diverse audiences.

Technical Skills Required

Databricks Ecosystem (Highest Priority):

• Advanced experience with Databricks and Spark frameworks using PySpark and Spark SQL.
• Deep understanding of Delta Lake architecture and Unity Catalog for secure data governance.
• Practical knowledge of Databricks workflows and asset bundles to orchestrate data pipelines effectively.
• Expertise with modern Lakehouse architecture principles to integrate data lakes and warehouses.

Microsoft Azure Platform:

• Hands-on experience architecting and managing Azure Data Factory (ADF) pipelines and ecosystem components.
• Proficiency with Azure Data Lake Storage Gen2 (ADLS Gen2) for secure and scalable storage solutions.
• Strong familiarity with Azure SQL databases and implementation of Azure security best practices.

Migration and Modernization:

• Experience migrating Synapse Pipelines, SQL Pools, Notebooks, and Data Flows into scalable Databricks platforms.
• Successful track record in modernizing legacy ETL workloads for improved reliability and performance.

Software Engineering & Development:

• Advanced SQL skills, data modeling expertise, and performance tuning abilities.
• Experience with GitLab CI/CD pipelines for automation of testing, deployment, and code quality checks.
• Strong adherence to code review best practices and collaborative development methodologies.

Preferred Skills and Experience

• Familiarity with Apache Airflow orchestration for task scheduling and workflow management.
• Exposure to Snowflake data platform technologies and data governance strategies.
• Experience deploying enterprise-scale Azure platforms with considerations for security, scalability, and operational excellence.

Areas of Responsibility

In this role, you will participate actively in gathering detailed and structured requirements for various data engineering modules. You will engage with multiple business units to understand dependencies, process linkages, and end-user needs ensuring comprehensive and clear documentation.

You will also be involved in facilitating solution design workshops under project leadership guidance, ensuring the technical architecture aligns with both business objectives and compliance standards. This includes translating functional requirements into robust, scalable data solutions.