Senior Data Engineer

Here is a concise summary of the key points from the Senior Data Engineer job posting at WorkGarden (formerly Astro Sirens): 1. Key technical requirements and skills: - Expertise in cloud technologies like AWS, Azure, and containerization (Docker, Kubernetes) - Proficiency in data engineering tools and frameworks such as Databricks, Kafka, Spark, dbt, and Python - Experience building and managing robust, scalable data pipelines and data warehousing solutions 2. Team/project information: - Work with a cross-functional team of data scientists, analysts, and engineers to ensure seamless data flow across the organization 3. Unique/notable aspects of the role: - Implement CI/CD pipelines for data workflow automation and deployment - Ensure compliance with data governance, security, and privacy regulations 4. Salary/benefits: -

About Us:

WorkGarden (formerly Astro Sirens) is a forward-thinking software consulting company specializing in innovative software and data solutions. We are looking for a senior Data Engineer to join our team. In this role, you'll work with cutting-edge cloud technologies like Databricks, Kafka, Spark, DBT and Python libraries to build robust data pipelines and scalable infrastructure.

Requirements

Responsibilities:

• Design, implement, and maintain robust and scalable data pipelines using AWS, Azure, and containerization technologies.
• Develop and maintain ETL/ELT processes to extract, transform, and load data from various sources into data warehouses and data lakes.
• Collaborate with data scientists, analysts, and other engineers to ensure seamless data flow and availability across the organization.
• Optimize data storage and retrieval performance by utilizing cloud services like AWS Redshift, Azure Synapse, or other relevant technologies.
• Work with containerization tools like Docker and Kubernetes to ensure smooth deployment, scalability, and management of data pipelines.
• Monitor, troubleshoot, and optimize data processing pipelines for performance, reliability, and cost-efficiency.
• Automate manual data processing tasks and improve data quality by implementing data validation and monitoring systems.
• Implement and maintain CI/CD pipelines for data workflow automation and deployment.
• Ensure compliance with data governance, security, and privacy regulations across all data systems.
• Participate in code reviews and ensure the use of best practices and documentation for data engineering solutions.
• Stay up-to-date with the latest data engineering trends, cloud services, and technologies to continuously improve system performance and capabilities.

Requirements:

• Excellent communication skill, specifically fluent verbal English is mandatory to explain complex technical concepts to non-technical stakeholders and collaborate across teams.
• Proven experience as a Data Engineer, with hands-on experience building and managing data pipelines.
• Strong proficiency in cloud technologies, specifically AWS (e.g., S3, Redshift, Glue) and Azure (e.g., Data Lake, Azure Synapse).
• Experience working with containerization and orchestration tools such as Docker and Kubernetes.
• Proficient in data engineering programming languages, such as Python, Java, or Scala.
• Solid experience with SQL and NoSQL databases (e.g., PostgreSQL, MongoDB, Cassandra).
• Familiarity with data processing frameworks like Apache Spark, Apache Kafka, or similar tools.
• Experience with workflow orchestration tools like Apache Airflow, DBT, or similar.
• Knowledge of data warehousing concepts and technologies (e.g., Snowflake, Amazon Redshift, or Google BigQuery).
• Strong understanding of ETL/ELT processes and best practices.
• Experience with version control systems like Git.
• Strong problem-solving skills and a proactive approach to troubleshooting and optimization.

Preferred Qualifications:

• Experience with data governance and security best practices in cloud environments.
• Familiarity with infrastructure-as-code tools such as Terraform or CloudFormation.
• Experience in working with machine learning and analytics tools for data analysis and reporting.
• Knowledge of data visualization tools (e.g., Power BI, Tableau) is a plus.
• Previous experience working in agile development teams.

Benefits

• Competitive salary and flexible payment method.
• Opportunities for growth and professional development.
• Flexible working hours and full remote work opportunity.
• Work in a collaborative, innovative and inclusive environment.
• Be a part of a data-driven culture that is at the forefront of innovation.

Originally posted on Himalayas