Senior ML Engineer - LLM Inference And Deployment

About Auric AI

Were a well-funded defence-tech team building a sovereign Multi-INT platform for Indian defence and intelligence customers. Our systems are migrating from frontier closed models to self-hosted open weights, deployed air-gapped on customer hardware. Youll own that migration, then own the inference layer.

Not a research role. Not an API-wrapping role.

What youll own

• The migration: 13 agents, real users, real eval set. Open models (Llama 3.3, Mistral, Gemma) with the smallest quality drop achievable within hardware budget.
• Tool-call and structured-output reliability: This is where the regression budget actually gets spent. Youll decide when the fix is prompting, constrained decoding, a model swap, or fine-tuning.
• The serving stack: vLLM first; possibly TensorRT-LLM or SGLang. Batching, KV cache, prefix caching, tensor parallelism, quantization.
• The eval harness: Per-agent, end-to-end. The highest-leverage part of the job. Design is yours.
• Heterogeneous serving: Not every agent needs 70B. You decide what runs where, across a multi-node H100/A100 cluster.
• Air-gapped production hardening: No internet, no phone-home, no apt-get during install.

Non-negotiables

• 4+ years ML systems, 2+ on LLM inference
• Production deployment of a 70B-class open model with real numbers
• Deep hands-on with vLLM, TGI, TensorRT-LLM, or SGLang
• Shipped tool-calling and structured-output reliability work. Constrained decoding (Outlines, XGrammar), prompt design across model families, or format-adherence fine-tuning
• Quantization tradeoffs measured, not read about (AWQ, GPTQ, FP8, INT8)
• Strong PyTorch, CUDA-level debugging
• Multi-node serving experience: tensor parallelism, pipeline parallelism, cluster-level batching

What we interview for

• Eval-first thinking: "How will we measure" before "which model."
• Knowing where open models fail: Specific failure modes (schema drift, format collapse, tokenizer-induced truncation) with opinions on root cause.
• Updates from evidence: Stories about being wrong.
• Systems intuition: Cost, latency, GPU utilization as connected.
• Comfort with the unglamorous: Migration is a thousand small regressions.

Nice to have

• vLLM/TGI/transformers PRs.
• SFT/DPO/LoRA.
• Restricted-environment deployment (defence, banking, telecom).
• Multi-agent systems.

Why this role

• You become the inference person for a sovereign Multi-INT platform while the architecture is still yours to shape. The serving stack, the heterogeneous design, the eval harness: decisions you make and live with across every customer deployment.
• Frontier-to-open migration is genuinely unsolved at the level of craft. No playbook. Youll measure tool-call drops on real agents, pick quantization schemes against real hardware, decide what gets a 70B versus a 12B across a multi-node cluster.
• Air-gapped, single-site, customer-controlled. If those constraints sound annoying, this isnt the role. If they sound like the interesting part, it probably is.
• Direct founder access. No committee culture. We dont care about degrees, we care about what youve shipped.

Apply with one of

• A project where you stood up an open model in production (with numbers)
• GitHub link to inference work. vLLM/TGI/transformers PRs are gold
• A paragraph on a non-trivial inference debug, bonus if its a tool-call regression