Lead Data Engineer

Job Description

Job Title: Lead Data Engineer

Job Location: Dubai

Job Description:

Key Responsibilities:

Data Acquisition

• Manage the existing Data pipelines built for data ingestion.
• Create and manage new data pipelines following the best practices for the new ingestion of data.
• Continuously monitor the data ingestion through Change Data Capture for the incremental load
• Any failed batch job schedule to be analysed and fixed to capture the data
• Maintaining and continuously updating on the technical documentation of the ingested data and maintaining the centralized datadictionary, with necessary data classifications.
  
  

• Exposing of Data views or Data models to Reporting and source systems using Hive or Impala, or similar tools provided by DM
• Exposing of cleansed data to Artificial Intelligence team for building data science models
  
  

• Design and implementation Using Big Data Technologies ( Hadoop, Spark, Hive, Kafka and NiFi) for data ingestion and processing. Implementing data transformation and modeling techniques to optimize data for analysis and reporting. Identifying and resolving performance bottlenecks in data processing and storage systems to ensure optimal performance and efficiency.
  
  

• Implementation of Data Pipelines (Structured and semi[1]structured)
  
  

• Certified Big Data Engineer from Cloudera/AWS/Azure
• Expertise with Big data products – Cloudera stack
• Expertise in Big Data querying tools, such as Hive, HBase and Impala.
• Expertise in SQL, writing complex queries/views, partitions, bucketing Strong Experience in Spark using Python/Scala
• Expertise in messaging systems, such as Kafka or RabbitMQ
• Hands on experience in Management of Hadoop cluster with all included services.
• Implementing ETL process using Sqoop/Spark Implementation including loading from disparate data sets, Pre-processing using Hive.
• Ability to design solutions independently based on high-level architecture. Collaborate with other development teams
• Expertise in building stream-processing systems, using solutions such as Spark- Streaming, Apache NIFI, KAFKA
• Expertise with NoSQL databasessuch as HBase
• Experience with Informatica Enterprise Data Catalogue (EDC) implementation and administration.
• Strong knowledge of data management, data governance, and metadata management concepts.
• Proficiency in SQL and experience with various databases (e.g., Oracle, SQL Server, PostgreSQL) and data formats (e.g., XML, JSON, CSV).
• Experience with data integration, ETL/ELT processes, and Informatica Data Integration