Data Engineer

ABOUT THE ROLE
As part of Technology Innovations CoE - supervised by the Engineering Manager & advised by the firm's CTO, you'll also check in with the team daily on the project, collecting information from other products to remain in sync with the entire eco-system in an Agile mindset & strong-Tech Startup environment culture. (Silicon Valley style Tech-startup)
We are creating a comprehensive Cross-border eCommerce Platform that features solutions for Storefront, Payment Gateway, Inventory, Catalog, Logistics, Marketing Insights, CRM... Here are some examples:
- Develop a Storefront solution
- Centralize payment gateways for managing transactions and after-payment business (such as refunds, chargebacks, claims, inquiries,..)
- Integrate various 3rd services to operate a complete supply-chain system, including inventory, catalogs, logistics tracking, and more...
- Convert raw data from marketing campaigns, transactions, user behavior, etc., into insights beneficial and significant for business operations.
- Plus, numerous other opportunities await your input.
In addition to development tasks, collaboration with other firm's CoE is crucial to comprehend issues within their specific sectors. Thus, from a technical standpoint, you will propose solutions to help them improve consistently every day.
WHAT YOU WILL DO
- Participate in Requirements Analysis & Technical Specification in the Agile/Scrum development process.
- Design, develop, and maintain scalable data pipelines and ETL processes to collect, clean, and process large datasets. The system is built on the microservice architecture and deployed Kubernetes (use AWS EKS service) across 3 AWS regions globally (US, HK, SG).
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and develop data solutions that meet business needs. Use BI tools for data analytics, such as Grafana, AWS QuickSight, or Excel (Google Sheets).
- Develop and implement data integration strategies and data transformation logic, ensuring high-quality and accurate data in the data warehouse. Use Kafka and Nifi tech-stack & eco-system to build data flow.
- Optimize and improve the performance, efficiency, and reliability of data pipelines, data models, and data storage solutions. Use database or object storage services like PostgreSQL, Elasticsearch, Redis, and AWS S3.
- Monitor and troubleshoot data pipeline issues, ensuring data integrity and timely data delivery. Use tools for monitoring logs and issues such as Sentry, Grafana, or alerts Slack.
- Continuously evaluate and implement new technologies, tools, and techniques to enhance data engineering capabilities.
- Document data engineering processes and maintain data dictionaries, ensuring compliance with data governance policies and regulatory requirements. Use the Atlassian toolset (Jira, Confluence) or Slack to help teams organize, collaborate, and complete work together.