Job Description:
We are looking for a
Senior Inference Enginee r with a strong foundation in
software engineering, distributed systems, and performance optimization to build and optimize inference engines for large-scale LLM serving systems
. You will work across both research and production environments, ensuring our LLM serving systems are fast, scalable, and efficient. The role spans the entire inference stack - from kernel and runtime to scheduling, memory management, and distributed execution
Key Responsibilities:
Profile, benchmark, and analyze bottlenecks for LLM inference workloads across multiple layers: kernel, memory, networking, and scheduler
Optimize inference engines (vLLM, SGLang, TensorRT-LLM) for throughput, latency, memory efficiency, GPU utilization, and cost
Implement and fine-tune inference optimization techniques including batching, KV-cache management, quantization, speculative decoding, parallelism strategies, and disaggregated serving
Build instrumentation and profiling tools to identify bottlenecks
Ensure the reliability of the inference pipeline through A/B launches, rollback, model versioning, and fault tolerance
Collaborate with the Platform Engineering team to improve serving architecture based on performance findings
Document and share knowledge, contributing to internal best practices and AI open-source projects whenever possible
Job Requirements:
Minimum 5 years of experience as a Software Engineer, Performance Engineer, or equivalent roles
Strong foundation in software engineering, distributed systems, and performance optimization
Proficient in Python;
experience with C/C++, Go, or other low-level programming languages is a plus
Experience with high-performance systems such as high-throughput backends, distributed services, large-scale serving systems, or equivalent
Knowledge of or hands-on experience with AI/ML serving systems, LLM inference, or GPU workloads
Understanding of GPU architecture and the CUDA programming model
Hands-on experience with at least one inference engine (vLLM, SGLang, TensorRT-LLM, Triton Inference Server) or equivalent systems
Understanding of inference optimization techniques including batching, KV-cache optimization, quantization, speculative decoding, tensor/pipeline parallelism, or disaggregated serving
Experience with profiling, bottleneck analysis, and performance tuning for production systems
Strong systems thinking, a high sense of ownership, and willingness to dive deep into complex technical challenges
Nice to Have:
Hands-on production experience with CUDA programming or NVIDIA libraries such as cuBLAS, cuDNN, and NCCL
Open-source contributions to inference-related projects (vLLM, SGLang, TensorRT-LLM) or AI infrastructure projects
Deep experience with the NVIDIA inference stack including Triton, CUTLASS, TensorRT, or CUDA profiling tools (Nsight Systems / Nsight Compute)
Experience with distributed inference, request routing, or inference orchestra
Experience with LLMOps, RAG systems, or AI agtionents
Experience with observability stacks (Prometheus, Grafana, OpenTelemetry) for ML or distributed systems
Published research papers or open-source contributions in the field of ML systems or inference optimization