Data Engineer (Real‑Time Stream) (225-725)

As a Data Engineer specializing in Real-Time Stream processes, you will play a critical role in managing and optimizing the pipeline for live data processing. You will be responsible for architecting the data flow, ensuring that high-speed data arrives and is processed with minimal latency. You will work collaboratively with cross-functional teams including data scientists, software engineers, and business analysts to ensure that the streaming applications and data meet the organization's requirements. Your work will enable the business to leverage streaming data to make fast, informed decisions, enhancing overall business processes and efficiency. You will need to have a strong foundation in data engineering, be adept at working with real-time data processing tools, and possess a keen eye for detail to handle large volumes of data efficiently.

Responsibilities

• Design and develop real-time data streaming solutions for business requirements.
• Build and optimize data processing pipelines to ensure high availability.
• Collaborate with software engineers to integrate streaming systems with cloud services.
• Maintain and scale distributed data streaming architectures as needed.
• Monitor and troubleshoot performance issues within real-time data systems.
• Work closely with data scientists to facilitate analytics on streamed data.
• Develop robust data extraction, transformation, and loading (ETL) techniques.
• Implement data security measures to protect sensitive information in transit.
• Create automation scripts to streamline data pipeline management activities.
• Document processes and coded solutions for future reference and scalability.
• Ensure compliance with industry standards and data governance regulations.
• Provide training and support to team members on data streaming architecture.

Requirements

• Bachelor's degree in Computer Science, Information Technology, or related field.
• Strong experience with real-time data streaming frameworks and tools like Apache Kafka.
• Proven expertise in programming languages such as Python, Java, or Scala.
• Experience with cloud-based solutions such as AWS, Azure, or Google Cloud Platform.
• Understanding of distributed systems and parallel processing paradigms.
• Excellent problem-solving skills and the ability to troubleshoot complex issues.
• Familiarity with container technologies like Docker and orchestration tools like Kubernetes.
• Strong communication skills to collaborate effectively with cross-functional teams.