Data Engineer

Overview

About Us

Core42, a leader in AI-powered cloud and digital infrastructure, is driving transformative technology solutions globally. Leveraging advanced resources and partnerships, Core42 empowers clients to harness sovereign AI infrastructure, especially in sectors with stringent regulatory needs. With a mission to redefine digital transformation, we combine sovereign capabilities with scalable, high-performance compute infrastructure, positioning itself at the forefront of AI innovation in the Middle East and beyond.

The opportunity

We are seeking a highly skilled and versatile Data Engineer with expertise in building robust data solutions using the Microsoft Data Platform and hands-on experience in AI/ML model development. This role focuses on designing, developing, and maintaining scalable and efficient data pipelines and architectures while also contributing to the development and deployment of AI scoring engines and predictive models to support advanced analytics.The ideal candidate will have deep expertise in Azure cloud services, data governance, data modeling, and advanced tools like Microsoft Fabric and Databricks, with a complementary focus on AI/ML workflows and MLOps. This position offers the opportunity to work at the intersection of data engineering and artificial intelligence to drive meaningful business outcomes

Responsibilities

Your key responsibilities

Primary Responsibilities – Microsoft Data Platform Expertise

• Design & Develop Data Pipelines:
• Build, optimize, and maintain ETL/ELT pipelines to ingest, transform, and process large volumes of structured and unstructured data from diverse sources.
• Leverage Azure Data Factory, Azure Data Lake, Azure Synapse
• Analytics, Microsoft Fabric, or Databricks for scalable data integration and transformation.
• Data Modeling & Architecture:
• Design and implement data models and schemas optimized for analytics, machine learning, and AI-driven decision-making.
• Create and manage data warehouses, data lakes, and lakehouses to support data analytics and AI workloads.
• Data Governance & Security:
• Ensure data quality, compliance, and security by implementing governance frameworks and leveraging tools like Microsoft Purview.
• Enforce data security protocols, including role-based access control, data masking, and encryption.
• AI/ML Data Integration:
• Collaborate with data scientists to integrate data pipelines with machine learning workflows, enabling seamless training and deployment of models.
• Performance Optimization:
• Monitor and optimize the performance of data pipelines and cloud resources to ensure high availability, scalability, and cost efficiency.
  
  

Secondary Responsibilities – AI/ML & Scoring Engine Development

• Build AI Scoring Engines:
• Develop and implement AI scoring engines to automate decision-making processes, such as fraud detection, customer segmentation, and recommendation systems.
• Data Preparation for AI/ML Models:
• Partner with data scientists to prepare and preprocess data for machine learning models, including handling missing values, scaling, and feature engineering.
• AI/ML Model Development:
• Contribute to the development of predictive and prescriptive models using frameworks like scikit-learn, TensorFlow, PyTorch, and Azure ML Studio.
• Model Deployment and Integration:
• Deploy machine learning models and scoring engines to production environments using Azure Machine Learning, integrating real-time and batch workflows.
• MLOps Implementation:
• Build and maintain MLOps pipelines for versioning, monitoring, and retraining AI models in production, ensuring continuous improvement.
• AI-Driven Insights:
• Support the integration of AI models with business applications to deliver actionable insights and improve operational efficiency.
  
  

Qualifications

• Education: Bachelor’s or Master’s in Computer Science, Data Engineering, Data Science, or related field.
• 6–10 years of experience in data engineering, with a strong focus on Microsoft Data Platform technologies.
• 4+ years of hands-on experience in AI/ML model development, including building and deploying scoring engines.
  
  

Technical Skills

Microsoft Data Platform

• Advanced knowledge of Azure Data Factory, Azure Synapse Analytics, Azure
• Data Lake, Microsoft Fabric, Databricks, and SQL Server.
• Experience with Azure DevOps for CI/CD pipelines in data engineering workflows.
  
  

AI & Machine Learning

• Proficient in Python and SQL for data engineering and ML tasks.
• Hands-on experience with AI/ML libraries such as scikit learn, TensorFlow, PyTorch, or Azure ML Studio.
  
  

Big Data Processing

• Experience with distributed data processing frameworks like Apache Spark forhandling large-scale datasets.
  
  

Data Governance Tools

• Experience with tools like Microsoft Purview for data cataloging, governance, andcompliance.
  
  

AI/ML Deployment & MLOps Practices

• Strong understanding of MLOps best practices, including model versioning, monitoring, and retraining.
  
  

Certifications

• Azure Data Engineer Associate
• Azure AI Engineer Associate
• Databricks Certified Data Engineer Associate
• Microsoft Fabric Certification
• Experience with data lakehouse architectures and advanced tools like Microsoft Fabric.
• Exposure to advanced AI domains such as NLP, computer vision, or time series forecasting.
• Familiarity with containerization tools like Docker and orchestration tools like Kubernetes.
  
  

Why Join Us

What Working At Core42 Offers

With a diverse team of 1,100+ employees from 68 nationalities, we foster an inclusive, innovative and collaborative environment. At Core42, we foster a culture grounded in trust, accountability and high performance. We are united by our values: Grit, where we overcome challenges with resilience and determination, Passion, which drives us to pursue excellence in everything we do, and Impact, as we aim to inspire progress and create meaningful change. Our team members thrive in an environment where each person’s contributions propel us forward, and together, we commit to achieving extraordinary results.

• Competitive Salary: We offer an attractive salary package based on your skills and experience
• Yearly Bonus: In recognition of your contributions, you will receive a performance-based annual bonus
• Exclusive Discount Cards: Access special benefits with Esaad and Fazaa cards, offering discounts across a wide range of services
• Premium Family Insurance: We provide comprehensive health coverage, including dental, vision and life insurance, ensuring the well-being of you and your family
• Learning & Development: We offer access to top-tier learning platforms to help you grow in your career. Learn at your own pace with unlimited access to premium courses.

---