Lead Data Engineer

ABOUT THE JOB:
Lead the development and ongoing management of the company's cloud-based data lake and common data platforms (CDP), ensuring they are scalable, secure, and optimized for business needs.
Oversee data ingestion processes from a variety of on-premise and cloud-based sources, utilizing batch and real-time methods for collecting and integrating data.
Utilize Azure Data Factory, Databricks, and other relevant tools to transform raw data according to business logic, ensuring it is stored and processed efficiently in multiple layers (bronze, silver, gold).
Implement and manage real-time data ingestion using Confluent Cloud (Kafka) and similar technologies to support business intelligence and decision-making.
Ensure data privacy protocols are followed by masking Personally Identifiable Information (PII) and safeguarding sensitive data throughout its lifecycle.
Set up and maintain CI/CD pipelines to automate data workflows using Azure DevOps, ensuring smooth and efficient deployments.
Continuously monitor and optimize ETL jobs, data pipelines, and cloud resources to improve system performance, minimize errors, and reduce costs.
Work closely with the data science team to prepare clean, stable, and high-quality data for AI/ML model development and other predictive analytics initiatives.
Lead the development of interactive Power BI dashboards and reports to deliver insights into business performance, sales behaviors, fraud detection, and more.
Establish efficient and secure methods for sharing data across teams using platforms like SharePoint and Databricks Unity Catalog.
Mentor junior data engineers, foster a collaborative team environment, and drive best practices in data engineering.