AWS ETL

We are seeking a Senior Data Engineer with more than 5 years of experience to lead the development of high performance data infrastructure. In this role the candidate will bridge the gap between serverless AWS compute and enterprise scale data warehousing in Snowflake. The candidate will be responsible for building lean, automated and highly observable data pipelines that transform raw data into actionable insights.

The role involves designing and deploying event driven data processing tasks using AWS Lambda and Python. The candidate will build and manage sophisticated workflows using Apache Airflow and AWS Step Functions and develop scalable ETL and ELT pipelines using AWS Glue with Spark and Python along with Snowflake native capabilities such as Streams, Tasks and Snowpipe.

The candidate will perform ad hoc analysis using Amazon Athena for schema on read queries and rapid data discovery within S3 data lakes. They will implement comprehensive monitoring, logging and alerting for data pipelines using Amazon CloudWatch and optimize queries by writing and tuning complex SQL for efficient processing in Snowflake and traditional relational database environments.

The ideal candidate will have over 5 years of experience in a data engineering role with a focus on cloud native architectures. Strong proficiency in Python for data manipulation and SQL is required along with deep expertise in AWS services including Glue, Lambda, Step Functions, Athena and CloudWatch. Proven experience in designing and managing data in Snowflake including virtual warehouses, clustering and data sharing is expected along with hands on experience in Apache Airflow.

The candidate should have strong operational expertise including monitoring distributed systems and debugging cloud based ETL failures. Experience with CI CD practices, version control using Git and unit testing is required.

Technical expertise includes Python, AWS Lambda, AWS Glue using PySpark, Apache Airflow, AWS Step Functions, Snowflake, Amazon S3, Amazon Athena, Amazon CloudWatch and SQL. Additional experience with DataStage, Oracle and other relational database systems is an advantage.

Preferred qualifications include prior experience working with banking clients and exposure to cloud platforms such as AWS and