AWS Pyspark

Skill: Pyspark, AWS

Exp: 6 to 12 years

Location: Pune

Job Summary

We are seeking an experienced AWS PySpark Developer to design, develop, and optimize large-scale data processing pipelines in the cloud. The ideal candidate will have strong expertise in PySpark, AWS services, and ETL workflows, with a focus on building scalable, high-performance data solutions for analytics and reporting.

Key Responsibilities

• Design, develop, and maintain ETL pipelines using PySpark on AWS.
• Work with large datasets to perform data cleansing, transformation, and aggregation.
• Optimize Spark jobs for performance and cost efficiency.
• Integrate data from multiple sources including AWS S3, RDS, Redshift, DynamoDB, and APIs.
• Implement data quality checks and ensure compliance with governance policies.
• Collaborate with data analysts, data scientists, and business stakeholders to deliver data solutions.
• Deploy and monitor jobs using AWS Glue, EMR, Step Functions, or Lambda.
• Troubleshoot and resolve issues in production data pipelines.
• Maintain documentation for data flows, transformations, and architecture.
  
  

• 3–7 years of experience in data engineering or big data development.
• Strong proficiency in PySpark and Spark SQL.
• Hands-on experience with AWS services such as:
• S3, Glue, EMR, Lambda, Athena, Redshift, IAM
• Solid understanding of ETL concepts and data warehousing.
• Proficiency in Python for scripting and automation.
• Experience with version control (Git) and CI/CD pipelines.
• Strong problem-solving skills and ability to work in an agile environment.
  

• Experience with AWS Glue Studio and Glue Catalog.
• Knowledge of Airflow or other orchestration tools.
• Familiarity with streaming data using Kafka or Kinesis.
• Exposure to Terraform or CloudFormation for infrastructure as code.
• Understanding of data security and encryption in AWS.