Big Data Engineer -Python/Java Up to 4k

As a Hadoop big data engineer, you will develop, operate and drive scalable and resilient data platform based on Hadoop ecosystem to address the business requirements:

• Ensure industry best practices around data pipelines, metadata management, data quality, data governance and data privacy

• Design and implement business-specific large-scale data processing pipelines

• Work with complex data structures, manipulate, cleanse data, and perform transformations to make insights from data.

• Responsible to Ingest data from files, streams, and databases. Process the data with PySpark, Kafka, Hive, Hive LLAP...

• Develop efficient software code for multiple use cases leveraging Spark and Big Data Technologies for various use cases built on the platform

• Provide high operational excellence guaranteeing high availability and platform stability.

➢ Must requirements:

• Experience in Hadoop ecosystem including HDFS, MapReduce, YARN, HBase, Zookeeper, Pig, Hive...

• Experience in building large-scale data processing (batch-processing, stream processing)

• Experience with Apache Spark preferably in PySpark

• Understanding of SLA and meeting Timelines for support activities

➢ Good to Have:

• Experience with Hadoop distributions such as Cloudera, HortonWorks, comparison and feasibility

• Experience in Apache Kafka, Apache Beam

• Experience with Data warehouse

• Experience in ETL

• Experience in Data management: o Data Governance o Data Architecture o Data Modelling o Data Quality o Data integration

• Experience in SQL and NoSQL Database

• Good in programming language Python, Java

• Experience with SRE, Patching & Automation: Kubernetes or Docker & Containerization • Experience working with Big Data on Cloud environment

• Experience with Google Cloud

• Experience in Backend development using Java

• Experience in Data API

• Good to have Architecture knowledge or experience