Data Engineer

• Build and maintain large-scale data platforms processing tens to hundreds of millions of
events per day.
• Design and operate real-time and batch pipelines handle 100k+ events/second using Kafka, Spark Streaming .
• Develop and scale multimodal pipelines.
• Work with vector databases (Milvus, Weaviate, or similar) to store and query millions of
embeddings with metadata filtering and hybrid search.
• Integrate LLMs and multimodal models (GPT-4, Llama-3, Claude, etc.) into data pipelines
for entity extraction, classification, enrichment, and summarization.
• Build and maintain a modern Lakehouse using Apache Iceberg or Delta Lake on S3/MinIO
with schema evolution and time-travel capabilities.
• Ensure good data quality, observability, and monitoring (lineage, basic data quality checks, dashboards, alerting).
• Optimize cost and performance of Spark/Flink jobs running on Kubernetes (autoscaling,
resource management, basic spot-instance usage).
• Collaborate closely with AI Engineers and Business Analysts to translate business needs into reliable data pipelines.
• Keeping real-time pipelines stable and accurate under high throughput (10k-100k+ events/sec).
• Handling noisy, conflicting, and rapidly changing data from many sources with dozens of entities, thousands of attributes, and hundreds of relationships.
• Achieving low-latency enrichment with LLMs and vector search in streaming workflows.
• Maintaining vector indexes with millions of new embeddings daily.
• Performing schema migrations and backfills on large datasets with minimal downtime.
• Ensuring good observability so the team can quickly spot and fix issues.