Senior Data Engineer Python/ Spark

SUMMARY:

As a data engineer, your heart will beat faster when you see a data pipeline that does its job - extracting data from a wide variety of systems, transforming it automatically and making it available to all stakeholders. It is your responsibility to determine how raw data is processed in the company and which systems play the decisive role.

In close coordination with the Data Lead, you design storage and access concepts and serve a wide variety of consumers. You will master challenges regarding different data types, data quality and data sources.

JOB RESPONSIBILITIES:

• Participate in a data engineering team which is responsible for POPS data landscape, including data architecture, data extraction and collection, data manipulation, feature engineering and other data-related product developments
• Proactively perform research on related and/or assigned technologies, domains and techniques that are relevant to the practice or assigned by analytical leaders
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build scalable and reliable ETL pipelines and processes to assemble data from a large number and variety of data sources
• Work with development team to enhance capabilities of internal analytics
• Implement solutions and processes for management and governance across data quality metrics, metadata, lineage, data access rights and business definitions

JOB REQUIREMENTS:

• At least 3 years of experience in handling large data sets and working with structured, unstructured and datasets
• Familiarity with database systems as Mongo, MySQL, PostgreSQL
• Experience with SQL, Python, bash shell scripts Experience with Spark and its features: Spark SQL, Spark streaming, structured streaming
• A degree or higher in Computer Science, Software Engineering, Information Technology or other related technical disciplines.
• Strong data engineering intuitive: understanding different data structure from abstract to physical view
• Experience in ML projects is a plus.

• Flexible working hour, 12-16-day annual leaves.
• Attractive annual bonus.
• Premium health insurance for Team Lead/Manager/Director level.
• Full insurances (health, social and unemployment) and competitive allowances (parking, transportation and telephone).
• Free GYM service, team building budget, Happy-hour events and monthly activities
• Company's congrats gifts on birthday, Wedding Day, Children's Day, Birth Deliver and other special occasions.
• Opportunity to work in exciting and innovative/creative media projects.