Senior Specialist - Data Engineering

Role description

The Azure Databricks Engineer will be responsible for designing, developing, and implementing large-scale data processing solutions on the Azure Databricks platform. The role involves collaborating with data scientists, analysts, and business stakeholders to deliver robust, scalable, and high-performance analytics solutions that support business objectives.

Key Responsibilities

• Develop and optimise data pipelines using Azure Databricks, Spark, and related technologies.


• Design and implement ETL (Extract, Transform, Load) workflows to ingest data from diverse sources into Azure Data Lake and other storage solutions.


• Collaborate with cross-functional teams to understand data requirements and deliver end-to-end solutions.


• Monitor, troubleshoot, and enhance Databricks jobs for performance, reliability, and scalability.


• Ensure data quality, security, and compliance with organisational and regulatory standards.


• Document processes, technical designs, and best practices for ongoing maintenance and knowledge sharing.

Required Skills and Qualifications

• Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.


• Proven experience working with Azure Databricks and Apache Spark.


• Strong proficiency in Python, Scala, or SQL for data engineering tasks.


• Familiarity with Azure cloud services such as Azure Data Lake, Azure Data Factory, and Azure Synapse Analytics.


• Experience in developing and optimising ETL/ELT pipelines.


• Excellent problem-solving, analytical, and communication skills.