Data Engineer

As a Data Engineer, you will help build the data foundation for our agentic AI platform. You will work with senior data architects, AI/ML engineers, and platform engineers to implement data ingestion, transformation, profiling, enrichment, validation, and preparation pipelines across structured and unstructured enterprise data sources.

This is a hands-on engineering role for someone who enjoys working with real-world enterprise data, building reliable pipelines, writing robust Python and SQL, and helping convert raw enterprise information into AI-ready data assets.

Key Responsibilities

• Build and maintain data ingestion pipelines for structured enterprise systems such as ERP, CRM, billing, finance, HR, OSS/BSS, ServiceNow, Salesforce, SAP, Oracle, databases, and APIs.
• Build pipelines for unstructured and semi-structured data sources such as documents, emails, logs, transcripts, PDFs, spreadsheets, and media metadata.
• Develop ETL/ELT workflows using Python, SQL, PySpark, Apache Spark, Airflow, dbt, Dagster, cloud-native services, or equivalent technologies.
• Support data profiling routines to identify missing values, duplicates, inconsistent formats, incomplete master data, schema changes, and conflicting records.
• Implement data quality checks using frameworks such as Great Expectations, dbt tests, AWS Glue DataBrew, custom validation scripts, or equivalent tools.
• Support data labelling, contextualization, harmonization, enrichment, and classification workflows required for AI agent configuration.
• Prepare data outputs for downstream AI consumption, including embeddings, metadata, semantic tags, graph-ready datasets, and retrieval-ready document chunks.
• Work with vector databases and search indexes to support semantic retrieval and hybrid search use cases.
• Support knowledge graph and ontology implementation by preparing entities, relationships, attributes, mappings, and validation-ready datasets.
• Implement data masking, anonymization, access control, PII handling, and privacy-aware data processing patterns as guided by senior architects.
• Support pipeline testing, monitoring, troubleshooting, performance tuning, and documentation.
• Collaborate with AI/ML engineers to ensure data outputs meet agent input contracts, retrieval requirements, and evaluation needs.
• Participate in code reviews, sprint planning, data design discussions, and engineering quality improvement initiatives.

Must-Have Qualifications

• Min 7 years of experience in data engineering, data pipeline development, software engineering, analytics engineering, or enterprise data delivery.
• Strong hands-on experience with Python and SQL.
• Working knowledge of data pipeline development using PySpark, Apache Spark, Airflow, dbt, Dagster, or equivalent technologies.
• Experience working with structured data from databases, APIs, enterprise applications, data lakes, warehouses, or lakehouse platforms.
• Exposure to cloud data platforms such as Databricks, Snowflake, BigQuery, Azure Data Lake, AWS S3, Google Cloud Storage, or equivalent platforms.
• Understanding of data modelling, schema design, joins, keys, relationships, data validation, and data quality concepts.
• Practical experience with data profiling, cleansing, transformation, and reconciliation.
• Familiarity with Git, CI/CD basics, unit testing, and production-grade engineering practices.
• Ability to troubleshoot data issues, performance bottlenecks, schema mismatches, and pipeline failures.
• Strong learning orientation and ability to work in a fast-moving product engineering environment.

Good to Have

• Exposure to vector databases such as Pinecone, Milvus, Weaviate, Qdrant, pgvector, Chroma, or equivalent technologies.
• Exposure to knowledge graphs, Neo4j, Cypher, RDF, OWL, SHACL, or ontology-driven data models.
• Experience with metadata management, data catalogues, lineage tools, governance platforms, or enterprise data quality frameworks.
• Experience with unstructured data processing, document parsing, OCR, semantic tagging, or text extraction pipelines.
• Exposure to telecom, BFSI, manufacturing, retail, or other complex enterprise domains.
• Experience with SAP, Salesforce, ServiceNow, Oracle, billing systems, order management systems, or product catalogues.
• Contributions to reusable data engineering components, internal accelerators, automation utilities, or open-source data tooling.

Why This Role Is Exciting

You will help build the data backbone of a new enterprise AI platform from the ground up. The work will involve real enterprise data challenges, giving you strong exposure to the practical foundations required for scalable AI adoption.

This role offers the opportunity to work closely with senior architects, AI/ML engineers, and product teams while building reusable data pipelines, quality controls, and AI-ready data assets for a venture-backed Infosys platform.