Experienced Data Engineer to design, build, and operate scalable data pipelines and platforms aligned with enterprise standards. The ideal candidate will translate business requirements into reliable, secure data solutions, owning delivery from ingestion and transformation through serving and production support for operational and analytical use cases.
Key Responsibilities
Design and implement ETL/ELT pipelines using Python, SQL, and Spark/PySpark for scalable batch and streaming data processing.
Build ingestion from databases, APIs, files, and event streams, using change data capture
(CDC) and managing schema evolution.
Develop Data Lake/Lakehouse layers and curated datasets using Delta Lake and Parquet,
aligned with enterprise data architecture.
Implement dimensional models, including star/snowflake schemas and slowly changing dimensions, to support BI and analytics requirements.
Orchestrate pipelines with Airflow or Databricks Workflows, managing dependencies,
scheduling, retries, and controlled backfills.
Implement data quality checks, reconciliation, and remediation; maintain metadata, data lineage, and catalog documentation.
Apply security and access controls, encryption, and secrets management in line with data classification and governance policies.
Optimize SQL queries, Spark workloads, storage layouts, and compute resources to improve performance and control platform costs.
Own production reliability against agreed SLAs through observability, alerting, incident resolution, and tested recovery procedures.
Use Git, CI/CD, automated testing, and infrastructure as code (IaC) to deliver repeatable deployments and maintainable engineering standards.
Collaborate with Data Architects, BI, and application teams to agree data contracts,
resolve integration issues, and communicate delivery trade-offs.
Required Skills & Experience
4 years of experience in data engineering or a related field, delivering and supporting production data pipelines.
Strong Python and SQL proficiency, with hands-on Spark/PySpark experience in distributed processing and performance tuning.
Practical experience with ETL/ELT, batch and streaming, and CDC, using Kafka, Azure
Event Hubs, Amazon Kinesis, or equivalent services.
Hands-on experience with at least one cloud (GCP or AWS) and data platform such as
Databricks, Snowflake, BigQuery, or Redshift.
• Databricks certification is required.
Strong understanding of Data Lake/Lakehouse patterns, Delta Lake, Parquet, dimensional modeling, and schema evolution.
Experience with Airflow or Databricks Workflows, pipeline observability, failure recovery, and meeting data freshness and availability SLAs.
Practical experience with Git, CI/CD, automated unit, integration, and data tests, and IaC tools such as Terraform.
Working knowledge of data quality, lineage, metadata/catalog tools, and security/access control; strong communication with business and technical stakeholders.
Preferred Qualifications
Relevant data engineering certifications in Databricks, Snowflake, Airflow, GCP, AWS,
or Azure.
Experience with dbt, reusable transformation frameworks, or Data Vault implementations.
Advanced streaming experience with event-time processing, checkpointing, and handling late or duplicate events.
Familiarity with data mesh, domain ownership, and implementing governed data products.
Background in a regulated industry (telecom, finance, healthcare, insurance) with strong compliance and data governance needs.
Thông tin chung