Lead Data Engineer

Description

• Lead the design, development, and implementation of scalable data pipelines and ELT processes using Databricks, DLT, dbt, Airflow, and other tools.
• Collaborate with stakeholders to understand data requirements and deliver high-quality data solutions.
• Optimize and maintain existing data pipelines to ensure data quality, reliability, and performance.
• Develop and enforce data engineering best practices, including coding standards, testing, and documentation.
• Mentor junior data engineers, providing technical leadership and fostering a culture of continuous learning and improvement.
• Monitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal disruption to business operations.
• Stay up to date with the latest industry trends and technologies, and proactively recommend improvements to our data engineering practices.
  
  

• Systems (MIS), Data Science or related field.
• 15 years of experience in data engineering and/or architecture, with a focus on big data technologies.
• Extensive production experience with Databricks, Apache Spark, and other related technologies.
• Familiarity with orchestration and ELT tools like Airflow, dbt, etc.
• Expert SQL knowledge.
• Proficiency in programming languages such as Python, Scala, or Java.
• Strong understanding of data warehousing concepts.
• Experience with cloud platforms such as Azure, AWS, Google Cloud.
• Excellent problem-solving skills and the ability to work in a fast-paced, collaborative environment.
• Strong communication and leadership skills, with the ability to effectively mentor and guide
• Experience with machine learning and data science workflows
• Knowledge of data governance and security best practices
• Certification in Databricks, Azure, Google Cloud or related technologies.