Big Data Engineer - Python SQL ETL

Key Responsibilities

• Design, develop, and support robust ETL pipelines to extract, transform, and load data into analytical products that drive strategic organizational goals.
• Develop and maintain data workflows on platforms like Databricks and Apache Spark using Python and Scala.
• Create and support data visualizations using tools such as MicroStrategy, Power BI, or Tableau, with a preference for MicroStrategy.
• Implement streaming data solutions utilizing frameworks like Kafka for real-time data processing.
• Collaborate with cross-functional teams to gather requirements, design solutions, and ensure smooth data operations.
• Manage data storage and processing in cloud environments, with strong experience in AWS cloud services.
• Use knowledge of data warehousing, data modeling, and SQL to optimize data flow and accessibility.
• Develop scripts and automation tools using Linux shell scripting and other languages as needed.
• Ensure continuous integration and continuous delivery (CI/CD) practices are followed for data pipeline deployments using containerization and orchestration technologies.
• Troubleshoot production issues, optimize system performance, and ensure data accuracy and integrity.
• Work effectively within Agile development teams and contribute to sprint planning, reviews, and Skills & Experience :
• 7+ years of experience in technology with a focus on application development and production support.
• At least 5 years of experience in developing ETL pipelines and data engineering workflows.
• Minimum 3 years hands-on experience in ETL development and support using Python/Scala on Databricks/Spark platforms.
• Strong experience with data visualization tools, preferably MicroStrategy, Power BI, or Tableau.
• Proficient in Python, Apache Spark, Hive, and SQL.
• Solid understanding of data warehousing concepts, data modeling techniques, and analytics tools.
• Experience working with streaming data frameworks such as Kafka.
• Working knowledge of Core Java, Linux, SQL, and at least one scripting language.
• Experience with relational databases, preferably Oracle.
• Hands-on experience with AWS cloud platform services related to data engineering.
• Familiarity with CI/CD pipelines, containerization, and orchestration tools (e.g., Docker, Kubernetes).
• Exposure to Agile development methodologies.
• Strong interpersonal, communication, and collaboration skills.
• Ability and eagerness to quickly learn and adapt to new Qualifications :
• Bachelors or Masters degree in Computer Science, Information Technology, or related fields.
• Experience working in large-scale, enterprise data environments.
• Prior experience with cloud-native big data solutions and data governance best practices.