Data Engineer, Global E-commerce Data Platform

Tóm tắt công việc
ABOUT THE ROLE
Join our Technology Innovations Center of Excellence (CoE) at Crossian, a fast-growing startup that achieved a milestone of $100M in 2024. In this dynamic role, you'll check in daily with your team to stay aligned on project goals and collaborate across our ecosystem with an Agile mindset infused with a unicorn-worthy startup culture.
We operate a production E-commerce Data Platform on AWS Cloud, built on a modern lakehouse-style architecture with Airflow orchestration on Kubernetes (Amazon EKS), dbt, Apache Iceberg on Amazon S3, AWS Glue, and Athena. The platform has been running in production for over a year and is now in an active phase of expanding features and optimizing performance, reliability, and cost. You'll help us scale ingestion, harden pipelines, and apply best practices across data domains such as Storefront, Payment Gateway, Inventory, Catalog, Logistics, Marketing and Ads Performance, CRM, Finance, and Customer Support.
WHAT YOU WILL DO
Design, develop, and maintain scalable data pipelines and ETL/ELT processes to collect, clean, and process large-scale datasets in the Data Platform, with a strong focus on building new pipelines to ingest data from third-party sources.
Collaborate with Data Scientists, Data Analysts, and other stakeholders to understand data requirements and develop data solutions for analytics, reporting, and AI/ML.
Build and optimize the data architecture of the Data Platform on AWS S3 with Apache Iceberg, supporting both batch and streaming data processing, including opportunities to apply and tune streaming technologies such as RisingWave.
Monitor and troubleshoot data pipeline issues to ensure data integrity and timely delivery. Utilize tools such as Grafana, Prometheus, and AWS CloudWatch to track logs and send alerts via Slack.
Document technical processes, manage metadata with OpenMetadata, maintain data catalogs, and ensure compliance with data governance policies. Use Atlassian tools (Jira, Confluence) or Slack for work management and team collaboration.
Optimize the performance, reliability, and cost-efficiency of the Data Platform. Enhance the scalability and efficiency of all components, including dbt, Airflow, PostgreSQL, AWS S3, and orchestration frameworks.
Continuously research and implement new technologies to optimize the current system, applying best practices in Data Engineering and DataOps.
We're not just about building platforms; we're about creating a workplace where talent thrives. Here's what you'll gain:
13th-month salary + performance bonus.
15 paid leave days + health insurance.
Budget to keep learning and growing.
Growth Opportunities
Hands-on exposure to cutting-edge technologies and complex system architecture in a global-scale project.
Clear career advancement pathways and access to continuous professional development programs.
Global Vision
Contribute to projects that redefine how brands connect with consumers globally.
Opportunity to work on challenging problems in data analytics, customer engagement, and operational efficiency.
In addition to development tasks, you'll collaborate closely with other CoEs to understand their unique challenges. With your technical expertise, you'll continuously propose innovative solutions to elevate their performance and contribute to the overall success of the organization.