Data Engineer Lead

- Optimizing in-house big-data pipelines, architectures and data sets
- Design, develop, automate data pipeline, data ETL, etc. on distributed data processing platform Spark
- Prepare data inputs for machine learning processes: model training, testing and production deployment
- End-to-end data processing, troubleshoot data-related problems and authorize maintenance or modifications.
- Monitor and analyze information and data systems and evaluate their performance to discover ways of enhancing them (new technologies, upgrades etc.)
- Ensure digital databases and archives are protected from security breaches and data losses.
- Degree in Computer Science or other related fields
- +4 years of experience as Lead Data Engineer
- 2 - 4 years of using the programming languages including Go/Python, Java, Scala
- Excellent understanding of data administration and management functions (collection, analysis, distribution etc.)
- Familiarity with modern database and information system technologies.
- Big data tools: Kafka, Hadoop, Hive, Spark, Elastic Search
- Relational SQL and NoSQL databases: MySQL, MongoDB, Postgres, Cassandra
- Data pipeline and workflow management tools: Airflow
- Stream-processing systems: Spark-Streaming
- Familiar with Linux development environments and multi-threaded programming
- Have knowledge/Experience in crawl data is plus
- Monitor server capacity and resource usage for server capacity planning is plus
- Create customized alerts to more effectively track key server capacity metrics is plus
- Good logical thinking;
- Passionate about coding and programming, innovation, and solving challenging problems
- An analytical mindset with problem-solving skills.
- Excellent communication and collaboration skills.
- Proficient in Microsoft Office
- Good English communication skills
- A team player, with brave, enthusiastic cheerful attitude