Research Engineer Large Language Model (LLM) Pretraining

Key Responsibilities
1. Data Engineering for Pretraining
• Build and maintain scalable pipelines for text collection, cleaning, deduplication, filtering, and quality scoring.
• Process large-scale Vietnamese and multilingual datasets.
• Implement tokenization workflows, corpus sharding, mixture sampling, and dataset balancing.
• Develop automated dataset validation and quality assurance tools.
2. Model Training & Optimization
• Support distributed training of LLMs using DeepSpeed, Megatron-LM, FSDP, or similar.
• Optimize throughput, memory efficiency, and multi-node GPU performance.
• Run full-scale LLM experiments and troubleshoot training issues.
• Conduct model fine-tuning, instruction tuning, and alignment if needed.
3. Infrastructure & Engineering
• Work with multi-GPU/multi-node clusters using Slurm, Docker/Singularity.
• Maintain experiment tracking pipelines.
• Develop reusable tools for logging, checkpointing, and evaluations.
4. Evaluation & Benchmarking
• Prepare and maintain Vietnamese and multilingual benchmark suites.
• Implement automated evaluation pipelines.
• Analyze results to guide improvements.