Senior Data Engineer

Collaborate with data scientists, product, and infrastructure teams to turn raw data into scalable, reliable assets.</p><h3>Key Responsibilities</h3><ul><li>Architect end to end batch and <strong>Spark Streaming</strong> pipelines on cloud (AWS/GCP/Azure).</li><li>Implement ML feature pipelines and Realtime inference services.</li><li>Optimize peta byte scale processing with Spark, Kafka, and Flink.</li><li>Build and maintain data warehouses/lakes (Redshift, BigQuery, Snowflake).</li><li>Enforce data quality, governance, and security.</li><li>Develop CI/CD, monitoring, and alerting for pipelines.</li><li>Mentor engineers and drive best practice documentation.</li></ul><h3>Required Experience</h3><ul><li>5+ years production grade data engineering.</li><li>Deep expertise with <strong>Apache Spark (batch & streaming)</strong>, Kafka, and distributed processing.</li><li>Handson ML pipeline experience (feature engineering, model training, deployment).</li><li>Cloud data platforms and warehousing.</li><li>Strong SQL + Python/Scala; familiar with Airflow, dbt, or similar.</li></ul><p>Originally posted on <a href="https://himalayas.app">Himalayas</a></p>

Design, build, and maintain high performance data pipelines that power analytics and machine learning products. Collaborate with data scientists, product, and infrastructure teams to turn raw data into scalable, reliable assets.

Key Responsibilities

• Architect end to end batch and Spark Streaming pipelines on cloud (AWS/GCP/Azure).
• Implement ML feature pipelines and Realtime inference services.
• Optimize peta byte scale processing with Spark, Kafka, and Flink.
• Build and maintain data warehouses/lakes (Redshift, BigQuery, Snowflake).
• Enforce data quality, governance, and security.
• Develop CI/CD, monitoring, and alerting for pipelines.
• Mentor engineers and drive best practice documentation.

Required Experience

• 5+ years production grade data engineering.
• Deep expertise with Apache Spark (batch & streaming), Kafka, and distributed processing.
• Handson ML pipeline experience (feature engineering, model training, deployment).
• Cloud data platforms and warehousing.
• Strong SQL + Python/Scala; familiar with Airflow, dbt, or similar.

Originally posted on Himalayas