AWS Databricks

Job Summary

We are seeking a AWS Databricks with 6 to 10 years of experience to join our team. The ideal candidate will have expertise in Spark in Scala Delta Sharing and Databricks Unity Catalog . This role involves working with cutting-edge technologies like Databricks CLI Delta Live Pipelines and Structured Streaming. The candidate will play a crucial role in managing risk and ensuring data integrity using tools such as Apache Airflow Amazon S3 and Python. The position is hybrid with no travel

Responsibilities

• Develop and maintain scalable data pipelines using Spark in Scala to ensure efficient data processing and analysis.
• Implement Delta Sharing and Databricks Unity Catalog to manage and secure data access across the organization.
• Utilize Databricks CLI and Delta Live Pipelines to automate data workflows and improve operational efficiency.
• Design and execute Structured Streaming processes to handle real-time data ingestion and processing.
• Apply risk management strategies to identify and mitigate potential data-related risks.
• Integrate Apache Airflow for orchestrating complex data workflows and ensuring seamless data operations.
• Leverage Amazon S3 for data storage solutions ensuring high availability and durability of data assets.
• Utilize Python for scripting and automation tasks to enhance productivity and streamline processes.
• Develop and optimize Databricks SQL queries to extract meaningful insights from large datasets.
• Implement Databricks Delta Lake to ensure data reliability and consistency across various data sources.
• Manage Databricks Workflows to automate and schedule data tasks improving overall data management efficiency.
• Collaborate with cross-functional teams to ensure alignment on data strategies and objectives.
• Contribute to the continuous improvement of data practices and methodologies to support the companys mission.
  
  

• Possess strong expertise in Spark in Scala and Databricks technologies.
• Demonstrate proficiency in Delta Sharing and Databricks Unity Catalog .
• Have experience with Databricks CLI and Delta Live Pipelines.
• Show capability in Structured Streaming and risk management.
• Be skilled in Apache Airflow and Amazon S3.
• Have a strong command of Python for data-related tasks.
• Be familiar with Databricks SQL and Delta Lake.
• Understand Databricks Workflows and their applications.
• Exhibit problem-solving skills and attention to detail.
• Be able to work in a hybrid model with a focus on day shifts.
• Have excellent communication and collaboration skills.
• Be committed to continuous learning and professional development.
• Be adaptable to changing technologies and business needs.