Data Engineer

Data Engineer

Location- Kochi/Coimbatore/Chennai

Job Overview

We are seeking a strong Data Engineer - Databricks Migration with 10+ years of experience and hands-on expertise in Databricks, PySpark, and SQL/T-SQL interpretation to support a legacy-to-Databricks migration program. The ideal candidate should have worked predominantly as a Data Engineer throughout their career, ideally beginning with Oracle, SQL Server/T-SQL, ETL, or Data Warehouse technologies and later transitioning into modern cloud-based Data Engineering. The role requires the ability to understand existing T-SQL logic and migrate transformation rules into scalable Databricks and PySpark solutions. Important screening criterion: The candidate must have worked predominantly and consistently as a Data Engineer throughout their career. Profiles that are primarily BI, reporting, dashboarding, analytics, or only recently shifted into data engineering should not be prioritized. 

Key Responsibilities:

• Analyze legacy data warehouse, SQL Server, Oracle, ETL, and T-SQL based implementations for migration to Databricks. 
• Read and interpret complex T-SQL scripts, stored procedures, functions, views, joins, transformation rules, and data flows. 
• Convert business logic from legacy SQL/T-SQL processes into optimized Databricks and PySpark workflows. 
• Design, develop, and maintain scalable ETL/ELT pipelines in Databricks. 
• Support ingestion, transformation, validation, and reconciliation of enterprise data across legacy and modern platforms. 
• Collaborate with architects, senior engineers, business analysts, and migration teams to understand source-to-target mapping requirements. 
• Perform data quality checks, unit testing, integration testing, and defect resolution during migration execution. 
• Optimize PySpark jobs, SQL queries, and Databricks notebooks for better performance and reliability. 
• Document migration logic, transformation rules, dependencies, and technical design details. 
• Participate in code reviews and follow Data Engineering best practices. 

Key Skills: 

• 5+ years of relevant Data Engineering experience, with the majority of career spent in Data Engineering roles. 
• Strong hands-on experience in Databricks development. 
• Strong PySpark programming experience for data transformation and pipeline development. 
• Strong SQL/T-SQL understanding, especially the ability to read and interpret existing T-SQL code. 
• Experience in legacy data technologies such as SQL Server, Oracle, traditional ETL tools, or Data Warehouse platforms. 
• Experience in ETL/ELT development, data ingestion, transformation, and data pipeline implementation. 
• Good understanding of Data Warehousing, Data Modeling, and dimensional concepts. 
• Ability to migrate or re-engineer legacy SQL/ETL logic into Databricks/PySpark. 
• Strong analytical, debugging, and problem-solving skills. 
• Good communication skills to work with technical and business teams. 

Preferred Qualifications: 

• Bachelors or Masters degree in Computer Science, Information Technology, Engineering, or a related discipline. 
• Azure Data Factory or Azure Data Platform exposure. 
• Delta Lake experience. 
• Unity Catalog exposure. 
• Azure Synapse Analytics experience. 
• CI/CD exposure for data pipelines. 
• Experience working in Agile/Scrum delivery models. 
• Data governance, security, and lineage awareness. 
• Relevant certifications in Databricks, Azure Data Engineering, or Cloud Data Platforms will be an added advantage. 
• Strong, hands-on Data Engineering background throughout career, not a partial or recent transition. 
• Avoid profiles that are mainly BI Developers, Power BI Developers, Data Analysts, or reporting-only candidates.