Site Reliability Engineer — HPC Observability And Incident Response

The Institute

Technology Innovation Institute (TII) a publicly funded research institute based in Abu Dhabi, the United Arab Emirates. It is home to a diverse community of leading scientists, engineers, mathematicians, and researchers from across the globe, transforming problems and roadblocks into pioneering research and technology prototypes that help move society ahead.

AI and Digital Science Research Centre

The Artificial Intelligence and Digital Science Research Centre (AIDRC) is the machine learning powerhouse of TII. We work closely with other research centers to harness the full potential of AI across multiple domains — from healthcare and cybersecurity to government and advanced technology systems.

We're looking for a talented Site Reliability Engineer with expertise in HPC and observability to join our growing team and help maintain the reliability of our computational infrastructure. As an engineer in our team, you'll develop monitoring and incident response capabilities for large-scale CPU and GPU clusters, detect operational issues early, and drive their resolution across compute, storage, networking, and workload orchestration systems.

Responsibilities

• Design, implement, and maintain observability systems covering infrastructure and workloads through metrics, logs, and traces.
• Develop actionable alerts, dashboards, and health checks for HPC clusters, Kubernetes platforms, GPU workloads, storage, and networking.
• Define service-level indicators and objectives in collaboration with platform owners and research teams.
• Detect, triage, and respond to incidents, coordinating investigation, escalation, mitigation, and service recovery.
• Participate in an on-call rotation and improve operational readiness through runbooks, escalation procedures, and incident exercises.
• Diagnose complex failures and performance degradation, including GPU errors, scheduling issues, storage bottlenecks, and network faults.
• Conduct root-cause analysis and blameless incident reviews, and implement corrective actions to prevent recurrence.
• Develop automation for incident enrichment, diagnostics, remediation, and recovery.
• Improve alert quality by reducing noise, identifying monitoring gaps, and detecting persistent resource underutilization or abnormal workload behavior.
• Collaborate with infrastructure engineers, developers, and researchers to improve reliability, capacity planning, and workload performance.

Requirements

• Proven experience in site reliability engineering, HPC operations, or infrastructure operations, with responsibility for incident detection and response.
• Strong Linux administration and troubleshooting skills, including processes, memory, filesystems, networking, and cgroups.
• Experience operating clustered CPU/GPU infrastructure and workload orchestration platforms such as Kubernetes or Slurm.
• Hands-on experience with observability tools such as Prometheus, VictoriaMetrics, Grafana, and centralized logging systems.
• Ability to write monitoring queries, investigate telemetry, and design alerts that support timely operational decisions.
• Experience investigating failures across compute, GPU, storage, and network systems.
• Proficiency in Python, Bash, Go, Rust, or another language used for operational automation.
• Familiarity with infrastructure-as-code, configuration management, and GitOps practices.
• Understanding of incident management, escalation, service recovery, and root-cause analysis.
• Strong analytical and communication skills, including the ability to communicate clearly during incidents.
• Experience with GPU telemetry, high-performance storage, RDMA networking, distributed training, or parallel computing frameworks is a plus.
• Experience operating infrastructure on AWS, GCP, or Azure is a plus.

Qualifications

• Bachelor's or Master's degree in Computer Science, Electrical Engineering, or a related field.