Tóm tắt công việc
ABOUT THE ROLE
Join our Technology Innovations Center of Excellence (CoE) at Crossian, a fast-growing startup that achieved a milestone of $100M in 2024. In this dynamic role, you'll check in daily with your team to stay aligned on project goals and collaborate across our ecosystem with an Agile mindset infused with a unicorn-worthy startup culture.
We operate a production E-commerce Data Platform on AWS Cloud, built on a modern lakehouse-style architecture with Airflow orchestration on Kubernetes (Amazon EKS), dbt, Apache Iceberg on Amazon S3, AWS Glue, and Athena. The platform has been running in production for over a year and is now in an active phase of expanding features and optimizing performance, reliability, and cost. You'll help us scale ingestion, harden pipelines, and apply best practices across data domains such as Storefront, Payment Gateway, Inventory, Catalog, Logistics, Marketing and Ads Performance, CRM, Finance, and Customer Support.
WHAT YOU WILL DO
Design, develop, and maintain scalable data pipelines and ETL/ELT processes to collect, clean, and process large-scale datasets in the Data Platform, with a strong focus on building new pipelines to ingest data from third-party sources.
Collaborate with
Data Scientists,
Data Analysts, and other stakeholders to understand data requirements and develop data solutions for analytics, reporting, and AI/ML.
Build and optimize the data architecture of the Data Platform on AWS S3 with Apache Iceberg, supporting both batch and streaming data processing, including opportunities to apply and tune streaming technologies such as RisingWave.
Monitor and troubleshoot data pipeline issues to ensure data integrity and timely delivery. Utilize tools such as Grafana, Prometheus, and AWS CloudWatch to track logs and send alerts via Slack.
Document technical processes, manage metadata with OpenMetadata, maintain data catalogs, and ensure compliance with data governance policies. Use Atlassian tools (Jira, Confluence) or Slack for work management and team collaboration.
Optimize the performance, reliability, and cost-efficiency of the Data Platform. Enhance the scalability and efficiency of all components, including dbt, Airflow, PostgreSQL, AWS S3, and orchestration frameworks.
Continuously research and implement new technologies to optimize the current system, applying best practices in Data Engineering and DataOps.
We're not just about building platforms; we're about creating a workplace where talent thrives. Here's what you'll gain:
13th-month salary + performance bonus.
15 paid leave days + health insurance.
Budget to keep learning and growing.
Growth Opportunities
Hands-on exposure to cutting-edge technologies and complex system architecture in a global-scale project.
Clear career advancement pathways and access to continuous professional development programs.
Global Vision
Contribute to projects that redefine how brands connect with consumers globally.
Opportunity to work on challenging problems in data analytics, customer engagement, and operational efficiency.
In addition to development tasks, you'll collaborate closely with other CoEs to understand their unique challenges. With your technical expertise, you'll continuously propose innovative solutions to elevate their performance and contribute to the overall success of the organization.
Bachelor's degree in Computer Science or a related field.
4+ years of experience in data engineering, ETL development, or a similar role.
Proficiency in using advanced SQL queries to extract valuable insights from datasets.
Strong proficiency in Python for data processing and scripting.
Hands-on experience with dbt and Airflow for data transformation and workflow orchestration, including running Airflow on Kubernetes (AWS EKS).
Experience with core AWS data and cloud services such as S3, Glue, Athena, and Lake Formation.
Knowledge of container orchestration with Kubernetes or AWS EKS, including deploying and operating containerized, microservice-based workloads.
Proficiency with Git and version-controlled, collaborative development workflows.
Experience with relational databases such as PostgreSQL, MySQL, Oracle, or MS SQL Server.
Knowledge of data architecture concepts and cloud-based storage solutions.
Strong problem-solving skills, attention to detail, and the ability to work independently or as part of a team.
Excellent communication and collaboration skills, particularly in an Agile environment.
Proficient in English, with the ability to communicate, read, and write effectively in a professional work environment.
Preferred (but not required)
Experience with Infrastructure as Code (IaC) and GitOps-based CI/CD workflows.
Experience with performance and cost optimization on AWS.
Experience with data streaming platforms. Java is a plus for streaming workloads.
Experience with microservice architecture.
Familiarity with data integration from various sources, such as web services, APIs, and file systems.
Familiarity with the Agile toolset, such as GitLab, Jira, Confluence, and Slack.