Role Description The Data Engineer is responsible for architecting, building, and optimizing centralized data infrastructure (Data Warehouse / Data Lake) to empower business intelligence, operational reporting, and data-driven executive decision-making. This role owns end-to-end data integration pipelines (ETL/ELT, CDC, Batch & Real-time) across enterprise operational platforms (SAP ERP, DMS, WMS, TMS, OCS/CRM, Logs, and 3rd-party services), guaranteeing data consistency, high availability, cost efficiency, and query performance.
Data Pipeline Architecture & Engineering: Design, construct, and optimize automated ETL/ELT pipelines ingesting heterogeneous data sources (core enterprise systems, OCS, DMS, CRM, SAP S/4HANA, WMS, TMS, Web/App Logs, 3rd-party APIs) into the centralized Data Warehouse.
Engineer enterprise-grade data integration solutions, Change Data Capture (CDC) pipelines, and high-throughput Batch and Real-time / Near Real-time processing flows.
Data Modeling & Architecture: Design and maintain robust data models on Data Warehouse platforms (Star Schema, Snowflake Schema) optimized for high-performance querying and analytical reporting.
Structure domain-specific Data Marts and analytical layers aligned with operational and strategic management needs.
DataOps & Data Quality Governance: Implement CI/CD best practices for data codebase management; build automated data testing and Data Quality checks to proactively detect and alert data anomalies.
Monitor pipeline reliability, execution SLAs, latency, and error rates to guarantee continuous system uptime.
Performance Tuning & Cloud Cost Optimization: Profile, troubleshoot, and optimize complex SQL queries; implement efficient partitioning, clustering, and data compression strategies to curtail cloud storage/compute expenditures.
Cross-Functional Collaboration & Project Management: Partner closely with
Data Analysts,
Business Analysts, and Product Teams to translate business requirements into clean, scalable data datasets and marts. Manage milestones for Data Warehouse enhancements, pipeline re-architecting, and BI reporting ecosystem initiatives.
Qualifications
Education: Bachelor's degree in Computer Science, Information Technology,
Software Engineering, Management Information Systems (MIS), or related fields.
Experience: Minimum 2-4 years of progressive experience as a Data Engineer, DWH Developer, or ETL Engineer.
Technical Qualifications:
Strong programming proficiency in Python or Java / Scala.
Expert-level SQL proficiency (advanced data manipulation, analytical functions, indexing, and execution plan optimization).
Direct hands-on experience or professional certification in modern Cloud Data Warehouses: Google BigQuery, Snowflake, AWS Redshift, or Databricks.
Practical experience with Change Data Capture (CDC) or equivalent real-time event streaming paradigms.
Proven competence in workflow orchestration and transformation tools: Apache Airflow and dbt (Data Build Tool).
Working knowledge of RESTful APIs, JSON/XML data serialization, file-based data ingestion, and enterprise integration patterns.
Proficient in Linux/Unix environments, Shell scripting, and task automation.
Comprehensive understanding of Data Lake, Data Warehouse, Lakehouse paradigms, and modern ETL/ELT architectures