Databricks With Pyspark

Description - External

Job Title: - Databricks with Pyspark

Location: - AIA Kolkata

Responsibilities

Minimum 6 to 13 years of experience

• Develop and maintain scalable data pipelines using Spark in Scala to process large datasets efficiently.
• Implement Delta Sharing and Databricks Unity Catalog Admin to ensure secure and seamless data sharing across platforms.
• Utilize Databricks CLI and Delta Live Pipelines to automate data workflows and enhance operational efficiency.
• Design and execute Structured Streaming applications to process real-time data streams with precision.
• Collaborate with cross-functional teams to integrate risk management solutions into data processing frameworks.
• Leverage Apache Airflow to orchestrate complex data workflows and ensure timely data delivery.
• Manage data storage and retrieval using Amazon S3 and Amazon Redshift to optimize performance and cost.
• Apply Python programming skills to develop robust data processing scripts and applications.
• Utilize Databricks SQL to perform advanced data analysis and generate actionable insights.
• Implement Databricks Delta Lake to ensure data reliability and consistency across the data lifecycle.
• Configure and manage Databricks Workflows to streamline data operations and improve productivity.
• Develop PySpark applications to process and analyze large-scale datasets with high efficiency.
• Contribute to the companys data strategy by delivering innovative solutions that drive business growth and societal impact.
  
  

• Possess a strong understanding of Spark in Scala and its application in data processing.
• Demonstrate expertise in Delta Sharing and Databricks Unity Catalog Admin for secure data management.
• Have experience with Databricks CLI and Delta Live Pipelines for workflow automation.
• Be proficient in Structured Streaming for real-time data processing.
• Show capability in risk management integration within data frameworks.
• Have hands-on experience with Apache Airflow for workflow orchestration.
• Be skilled in using Amazon S3 and Amazon Redshift for data storage solutions.
• Exhibit proficiency in Python for data processing and scripting.
• Be adept at using Databricks SQL for data analysis.
• Understand the principles of Databricks Delta Lake for data consistency.
• Have experience managing Databricks Workflows for operational efficiency.
• Be skilled in developing PySpark applications for large-scale data processing.