Sr SW Engineer PySpark Spark SQL Scala Big Data ETL

About Us

Visa is a world leader in payments technology, facilitating transactions between consumers, merchants, financial institutions and government entities across more than 200 countries and territories, dedicated to uplifting everyone, everywhere by being the best way to pay and be paid.

At Visa, youll have the opportunity to create impact at scale — tackling meaningful challenges, growing your skills and seeing your contributions impact lives around the world.

Join Visa and do work that matters – to you, to your community, and to the world. Progress starts with you.

Job Description

At Visa, the Corporate Information Technology, Billing & Incentives Platforms team, enables Visas revenue growth through flexible rules-based pricing engines and global revenue applications built on next-generation technologies. This includes managing system requirements, evaluating cutting-edge technologies, design, development, integration, quality assurance, implementation, and maintenance of corporate revenue applications. The team works closely with business owners of these services to deliver custom developed solutions, as well as implement industry leading packaged software. This team has embarked on a major transformational journey to build and implement best of breed revenue and billing applications to transform our business as well as technology.

We are looking for a Senior Software Engineer who will work in Core Billing Platform with strong hands-on experience in PySpark, Spark SQL, and Scala to design, develop, and optimize scalable data pipelines for large-scale data processing.

The ideal candidate should have deep expertise in distributed data processing, ETL development, performance tuning, and big data ecosystem technologies. This role requires end-to-end ownership of data pipelines, from ingestion and transformation to storage and consumption, while ensuring high performance, reliability, and data quality.

Exposure to AI/GenAI enabled data engineering solutions is a plus.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark, Spark SQL, and Scala
• Build and optimize complex ETL workflows for extracting, transforming, and loading data across multiple systems such as Oracle, PostgreSQL, Hive, Hadoop, and cloud-based data platforms
• Develop and optimize Spark jobs for large-scale batch and, where applicable, streaming data processing
• Write efficient Spark SQL queries and optimize data transformations for performance and scalability
• Implement data processing strategies including partitioning, caching, parallel processing, file format optimization, and job tuning
• Architect and implement big data solutions using Spark, Hive, Hadoop ecosystem, Delta Lake, and related technologies
• Build end-to-end data flows from data ingestion → transformation → storage → consumption layers
• Develop config-driven, reusable ETL frameworks, automation utilities, and common data engineering components
• Ensure data quality, integrity, accuracy, and consistency across data pipelines
• Perform performance tuning, debugging, troubleshooting, and root-cause analysis of data pipeline issues
• Collaborate with business stakeholders, data analysts, architects, and engineering teams to translate business requirements into scalable technical solutions
• Create and maintain technical documentation, data flow diagrams, data models, and architecture artifacts
• Follow engineering best practices around code quality, version control, CI/CD, testing, deployment, monitoring, and alerting
• Mentor junior engineers and promote best practices in data engineering and big data development
• Explore and integrate AI/GenAI capabilities where applicable for automation, data validation, anomaly detection, and ETL optimization
  
  

This is a hybrid position. Expectation of days in office will be confirmed by your hiring manager.

Qualifications

Qualifications:

• Bachelor’s or master’s degree in Computer Science, Software Engineering or other relevant Engineering discipline with 3-5 years of experience in Data Engineering, Big Data Engineering, or ETL development
• Strong hands-on experience with PySpark, Spark SQL, and Scala
• Solid experience building and maintaining large-scale ETL/ELT data pipelines
• Strong understanding of Apache Spark architecture, including executors, drivers, DAGs, stages, tasks, shuffle, caching, and memory management
• Proven experience in Spark performance tuning and optimization
• Strong SQL skills with experience writing complex queries, joins, aggregations, window functions, and query optimization
• Experience working with big data technologies such as Hive, Hadoop, HDFS, Delta Lake, Parquet, ORC, and related ecosystem tools
• Experience working with relational databases such as Oracle, PostgreSQL, SQL Server, or MySQL
• Strong programming skills in Python and Scala
• Good understanding of distributed computing, data partitioning, data modeling, and large-scale data processing patterns
• Experience with data quality checks, reconciliation, validation, exception handling, and audit controls
• Familiarity with CI/CD pipelines, Git/version control, code reviews, and deployment processes
• Experience working in Agile/Scrum delivery environments
• Strong problem-solving, analytical, and communication skill
  
  

AI / GenAI Exposure — Good To Have

• Exposure to AI/ML or Generative AI concepts
• Experience integrating AI/LLM APIs into data pipelines, applications, or automation workflows
• Understanding of data pipelines supporting ML/AI model development and deployment
• Experience using AI tools for:
• ETL code generation
• Data validation
• Anomaly detection
• Data profiling
• Workflow automation
• Ability to identify opportunities to improve data engineering productivity using AI-assisted development tools
  

Visa is an EEO Employer

Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status. Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.

---