AI Architect

Job Summary

Key Responsibilities

• Design, build and maintain batch and real-time data pipelines.
• Develop data products using Databricks Lakeflow Declarative Pipelines, PySpark and SQL.
• Build streaming solutions using Spark Structured Streaming and Lakeflow declarative pipelines
• Optimise pipelines for scalability, reliability and performance.
• Develop automated tests and participate in code reviews.
• Contribute to CI/CD and engineering best practices.
• Monitor, troubleshoot and support production workloads.
• Use AI-assisted development tools to improve productivity while maintaining high engineering standards.
  
  

• Databricks
• Lakeflow Declarative Pipelines and DXQ, DB Genie, MCP Server
• Apache Spark (PySpark)
• Spark Structured Streaming
• Auto Loader
• Unity Catalog
• Python
• SQL
• Delta Lake
• Azure Event Hubs or equivalent event streaming platforms
• Git and collaborative development workflows
• Unit testing
• CI/CD
• Strong analytical and problem-solving skills
  
  

• Performance tuning and Spark optimisation
• Data modelling
• Azure DevOps
• AI coding assistants (GitHub Copilot, Cursor or similar)
• Experience working in Agile or SAFe environments
  
  

• Takes ownership from design through production support.
• Writes clean, maintainable and well-tested code.
• Builds reusable, production-ready solutions.
• Collaborates effectively with teammates and stakeholders.
• Continuously learns and embraces modern engineering practices.