Staff Data Engineer 8 Years Hadoop Hive Scala Python

About Us

Visa is a world leader in payments technology, facilitating transactions between consumers, merchants, financial institutions and government entities across more than 200 countries and territories, dedicated to uplifting everyone, everywhere by being the best way to pay and be paid.

At Visa, youll have the opportunity to create impact at scale — tackling meaningful challenges, growing your skills and seeing your contributions impact lives around the world.

Join Visa and do work that matters – to you, to your community, and to the world. Progress starts with you.

Responsibilities

Job Description

• Strong technology and leadership background building and managing enterprise scale Data Engineering pipelines (Hadoop, Hive, Spark)
• Should posses a solid understanding of data engineering principles and best practices
• Responsible for the design, implementation and test of scalable distributed systems that take advantage of technology to allow standardization, security, timeliness and quality of data.
• Lead code reviews, ensuring adherence to coding standards, and promoting clean, efficient code within the team.
• Experience creating/supporting production software/systems and a proven track record of identifying and resolving performance bottlenecks for production systems.
• Work closely with senior engineers to develop the best technical design and approach for new product development.
• Work with product managers in developing a strategy and road map to provide compelling capabilities that helps them succeed in their business goals.
• Identify opportunities for further enhancements and refinements to standards and processes.
  
  

Essential Functions

• Collaborate with customers to understand their requirements and build solutions that deliver real value.
• Architect, design, and implement secure, robust, and scalable solutions.
• Drive proof of concept initiatives and lead implementation.
• Mentor team members and foster a culture of mutual accountability.
• Engage in meetings and discussions with cross-functional teams.
  
  

This is a hybrid position. Expectation of days in office will be confirmed by your hiring manager.

Qualifications

Qualifications:

8 or more years of work experience with a Bachelors Degree or with an Advanced Degree

Working knowledge of Hadoop ecosystem and associated technologies, (For e.g. Apache Spark, etc.)

Strong experience in writing and optimizing spark code and Hive code to process large data sets in Hadoop environments

Experience with Advanced SQL for extracting, aggregating, and processing big data using Hadoop

Strong Development experience in one or more of the following: Scala, Python or Java.

Experience with Linux systems with Unix/Shell or Python scripting and exposure to Scheduling tools like Airflow and Control – M.

Experience with Continuous Integration and Automated Test tools such as Jenkins, Artifactory, Git, Selenium, Chef desirable

Familiarity with RDBMs viz, MS SQL, DB2, Oracle, etc for data retrieval

Knowledge on Streaming technologies like Kafka, Spark streaming, etc with Apache Hudi will be a plus

Experience in managing and operating AWS (other cloud) services including EC2, S3, SageMaker etc.

Exposure to public cloud equivalents, and ecosystem shall be a plus.

Knowledge on Visualization Tools like Tableau, Power BI, is a plus.

Visa is an EEO Employer

Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status. Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.

---