Build ingestion workflows for APIs, web data, and files.
Handle structured and semi-structured formats (JSON, XML, HTML, CSV).
Apply resilient scraping practices, including session handling, proxies, and request automation.
Ensure compliance with security and access restrictions when integrating external data sources.
Optimize SQL queries and transformations for analytics.
Design, build, and maintain robust Python-based data pipelines.
Implement testing frameworks and good engineering practices for reliability.
Ensure data quality, consistency, and scalability across workflows.
Expert-level Python for data engineering and workflow automation.
Knowledge of web scraping, ingestion workflows, and handling varied data formats.
Experience designing and maintaining production-grade data pipelines.
Familiarity with best practices in testing, monitoring, and pipeline reliability.
Proficiency with version control (Git/GitHub/Azure DevOps).
Strong SQL skills for ETL and analytics.
Qualifications
BSc or MSc in Computer Science, Data Engineering,
Software Engineering, or related fields.
Equivalent practical experience will also be considered.