Senior Software Engineer

Mô tả Công việc
Crawler Development and maintenance: Mentor junior developers in crawler development techniques and best practices, Provide guidance in architectural decisions for data crawling and management.
Technical Leadership: Analyze and improve the performance of crawlers, reducing latency and increasing data retrieval efficiency, Address challenges related to rate limiting and blocking mechanisms, Performance Optimization: Implement security measures to protect sensitive data and ensure compliance with data privacy laws, Monitor and adapt to changes in social media platforms's APIs, rules, and regulations
Security and Compliance: Integrate crawlers with existing systems and third-party tools to streamline data access and usage., Work closely with other engineering teams to incorporate new features and maintain compatibility across services.
Integration and collaboration: Build data pipelines to clean, process and store data effectively, Collaborate with data scientists and analysts to ensure data quality and availability for analysis.
Data Management: Design, develop and maintain robust web crawlers that can effectively collect data from Facebook, Tiktok, Threads, Linkedin, and other platforms, Optimize crawling algorithms to ensure data is gathered quickly and accurately while adhering to ethical standards and platforms policies