Senior Data Engineer

Job Description

• Analysis, design, development, support and maintenance of DW, Data Lake and Lakehouse infrastructures
• Technical expertise on data pipelines and ETL processes.
• Solid understanding of SQL and Data Modelling
• Hands on experience with Databricks on AWS cloud solutions
• In-depth knowledge of Databricks ETL and Orchestration, notebook development and job scheduling
• Collaborate with cross-functional teams of business analysts, business users and IT stakeholders gathering, documenting, analysing and implementing requirement
• Collaborate with multiple teams to develop and maintain data pipelines, harvesting data from systems of records to create data products
• Create and optimize relational databases schema, tables, indexes, views, and stored procedures
  
  

• Perform data migration, transformation, and integration tasks between different database systems
• Develop and maintain ETL processes to load and transform data from various sources
• Work with other development groups to advise, guide and assist in their integrations
• Familiarity with version control systems (e.g. GIT) and Agile development methodologies
• Interact with vendor for support and consultancy
• Work directly with reporting manager to ensure clear and accurate communication of current status, dependencies and estimated delivery timelines
• Clearly communicate issues, risks and proposed solutions to relevant stakeholders
• Coordinate the release process with business groups, operations in compliance with ITIL Change Management discipline
• Communicate detailed descriptions of functionality changes and provide follow up support as required
• Support QA testing
• Provide direction & support to the Business Divisions for User Acceptance Testing
• Utilizing Reporting tools to rapidly create, modify, refresh and update dashboards and reports
  
  

• Experience with business intelligence, data analysis, data modelling and visualization solutions on Cloud, on-premises and hybrid
• experience in Databricks on Amazon Web Services cloud computing platforms
• Experience with Databricks platform
• Experience with Python, SQL, relational database objects (Oracle and/or SQL Server) and procedural languages (PL/SQL and/or T-SQL)
• Experience with Version Control systems
• Web Services integration using REST/SOAP/JSON