Databricks Spark

Skill: Azure databricks

Experience: 6 to 12 years

Location: AIA Chennai

Job Summary

Architect role in a global organization focusing on end to end design and optimization of data platforms using Databricks SQL Databricks Workflows and PySpark. The architect will shape hybrid model solutions that support business analytics ensure scalable performance and drive efficient data engineering practices while mentoring teams and aligning platforms with enterprise standards.

Responsibilities

• Design robust Databricks based data platform architectures that enable scalable analytics and reporting for complex enterprise use cases and align with long term business growth objectives.
• Drive end to end solution design using Databricks SQL and PySpark to optimize data ingestion transformation and consumption flows that deliver consistent performance for diverse workloads.
• Define standardized patterns and reusable components for Databricks Workflows that improve orchestration reliability simplify maintenance and reduce operational overhead across multiple projects.
• Collaborate with data engineers analysts and product teams in a hybrid work environment to translate business needs into technical blueprints that ensure clear traceability from requirement to implementation.
• Establish governance practices for data quality security and compliance on Databricks platforms that support responsible data usage and protect sensitive information across regions.
• Optimize SQL queries PySpark jobs and cluster configurations to reduce processing time and cost while maintaining high availability and resilience for critical data applications.
• Guide teams on best practices for version control environment management and deployment pipelines so that Databricks solutions are consistently delivered with predictable quality.
• Review solution designs and technical deliverables for complex data initiatives to identify risks propose improvements and ensure alignment with architectural standards and organizational guidelines.
• Coordinate with infrastructure and cloud operations stakeholders to ensure Databricks environments are correctly sized monitored and tuned for day shift usage patterns without requiring travel.
• Promote data engineering excellence through documentation technical knowledge sharing and coaching that help colleagues build stronger skills in Databricks SQL Workflows and PySpark.
• Engage with business sponsors to explain architectural decisions in clear terms demonstrating how the data platform improves decision making and contributes to broader societal value through better insights.
• Assess new features and capabilities in the Databricks ecosystem and recommend pragmatic adoption strategies that balance innovation stability and regulatory considerations.
• Monitor production platforms and incident trends to drive continuous improvement initiatives that enhance reliability reduce defects and protect the organization reputation.
  
  

• Possess extensive experience of twelve to sixteen years in data architecture or advanced data engineering roles with a strong focus on modern cloud based analytic platforms.
• Demonstrate expert level proficiency in Databricks SQL including complex query design performance tuning and implementation of data models for large scale analytical workloads.
• Show deep hands on capability in PySpark for building scalable batch and streaming data pipelines that integrate disparate data sources into unified and well structured datasets.
• Hold strong practical experience designing and managing Databricks Workflows including job orchestration scheduling dependency management and error handling for business critical processes.
• Exhibit solid understanding of data warehousing lakehouse concepts and distributed computing principles that underpin resilient architectures for enterprise analytics.
• Display experience working in hybrid work models with global teams using collaborative tools and structured communication practices that keep projects aligned and efficient.
• Bring familiarity with cloud security data governance and regulatory considerations so that architecture decisions safeguard data assets and meet organizational policies.
• Prefer exposure to adjacent tools such as data catalog solutions and mainstream visualization platforms that consume outputs produced from Databricks environments.
• Value clear documentation architectural diagrams and concise technical narratives that improve knowledge transfer and reduce onboarding time for new project members.