Data EngineerDe Heus LLC

Mô tả Công việc
A. JOB PURPOSE:
• The Data Engineer is responsible for orchestrating data transformations and integrations using Microsoft Azure services to develop a comprehensive Data Lake.
• This role emphasizes advanced data engineering practices, including the utilization of Databricks and Azure Synapse Link, to support our operations across Vietnam, Cambodia, Indonesia, Myanmar, and other business units in Asia and other regions.
• This role enables advanced data analytics, reporting, and dashboard functionalities.
B. ACCOUNTABILITIES:
1. System Development and Maintenance:
• Design, develop, and maintain Azure Cloud data lakes, data warehouses, and both SQL and NoSQL databases.
• Implement and manage batch processing, streaming, and integration solutions using Databricks and Azure Synapse Link for data ingestion from various applications to data repositories.
• Optimize SQL queries and Databricks notebooks to reduce processing times across data lakes, data warehouses, and cloud data cubes.
• Automate data pipeline processes using Azure Data Factory, ensuring secure source data access and efficient data policy management.
• Collaborate with data visualization specialists, analysts, and IT teams to support backend data solutions that meet business reporting needs.
2. Performance and Quality Assurance:
• Ensure system stability and performance without impacting user experiences.
• Monitor databases and data processing activities to meet specific business scenarios.
• Ensure data accuracy and availability for ongoing data analytics projects.
• Manage and optimize costs related to data storage and processing services through efficient resource management and a pay-as-you-go approach.
• Maintain comprehensive documentation for data architecture, ingestion, processing, and operations.
• Manage source data access securely.
• Configure data access authentication and authorization.
• Manage and enforce data policies and procedure standard.
• Collaborate with Data Visualization and Analyst specialist, Data champion and IT team to ensure about data back-end can support building dashboard as business requested.
C. RESULTS:
• Ensure current systems can work smoothly and do not impact to performance.
• Data architect from design to real life operation.
• Ensure data availability and accuracy for further data analytics project.
• Optimize the cost of use for data storage and data processing services (impact to the amount of payment) on monthly basis by pay-as-you-go approach.
• Data catalogue documentation.
• Scripting and documentation for data ingestion, data processing, data serving.
• Data architect and operation can be extended to acquire other data in other BU, other division, new business...
• Data available timely and in good quality.
• Updated authorization matrix in quarterly basis.