Data Engineer

A. JOB PURPOSE:
• The Data Engineer is responsible for orchestrating data transformations and integrations using Microsoft Azure services to develop a comprehensive Data Lake.
• This role emphasizes advanced data engineering practices, including the utilization of Databricks and Azure Synapse Link, to support our operations across Vietnam & Asia.
• This role enables advanced data analytics, reporting, and dashboard functionalities.
B. ACCOUNTABILITIES:
1. System Development and Maintenance:
• Design, develop, and maintain Azure Cloud data lakes, data warehouses, and both SQL and NoSQL databases.
• Implement and manage batch processing, streaming, and integration solutions using Databricks and Azure Synapse Link for data ingestion from various applications to data repositories.
• Optimize SQL queries and Databricks notebooks to reduce processing times across data lakes, data warehouses, and cloud data cubes.
• Automate data pipeline processes using Azure Data Factory, ensuring secure source data access and efficient data policy management.
• Collaborate with data visualization specialists, analysts, and IT teams to support backend data solutions that meet business reporting needs.
2. Performance and Quality Assurance:
• Ensure system stability and performance without impacting user experiences.
• Monitor databases and data processing activities to meet specific business scenarios.
• Ensure data accuracy and availability for ongoing data analytics projects.
• Manage and optimize costs related to data storage and processing services through efficient resource management and a pay-as-you-go approach.
• Maintain comprehensive documentation for data architecture, ingestion, processing, and operations.
• Manage source data access securely.
• Configure data access authentication and authorization.
• Manage and enforce data policies and procedure standard.
• Collaborate with Data Visualization and Analyst specialist, Data champion and IT team to ensure about data back-end can support building dashboard as business requested.
C. RESULTS:
• Ensure current systems can work smoothly and do not impact to performance.
• Data architect from design to real life operation
• Ensure data availability and accuracy for further data analytics project.
• Optimize the cost of use for data storage and data processing services (impact to the amount of payment) on monthly basis by pay-as-you-go approach.
• Data catalogue documentation.
• Scripting and documentation for data ingestion, data processing, data serving.
• Data architect and operation can be extended to acquire other data in other BU, other division, new business...
• Data available timely and in good quality
• Updated authorization matrix in quarterly basis