Senior Math Libraries Engineer - LLM Integration and Developer Experience

We are seeking a Senior Math Libraries Engineer to join NVIDIA, the company renowned for its GPU‑accelerated platforms that power AI, data analytics, scientific simulations, computer vision, and autonomous systems. In this role you will focus on extending the CUDA‑X math library stack so that its APIs can be consumed directly by large language models (LLMs) and AI agents, enabling developers to build high‑performance applications with minimal friction. Your daily work will involve shaping the architecture and implementing LLM‑first math library interfaces, creating documentation structures that are introspectable and explainable for AI agents, and collaborating with both internal teams and external partners to deliver timely LLM‑enhanced releases. You will also design metrics, tools, and processes to evaluate the impact and quality of AI‑generated code, and prototype solutions that translate existing CUDA‑X APIs into representations that work smoothly with popular code‑generation tools. The position requires a strong academic background and solid professional experience. Candidates should hold a PhD or MSc in Computational Science, Computer Science, Applied Mathematics, or a closely related discipline, and have at least three years of relevant work experience. You must possess deep knowledge of LLM technologies, including finetuning, reinforcement learning, retrieval‑augmented generation, model‑centric programming, and agent tooling, and a proven track record of designing clear, composable APIs together with high‑quality, well‑documented code for complex technical domains. The ability to juggle multiple projects, work autonomously with minimal direction, and maintain excellent collaboration, communication, and documentation habits is essential. Additional strengths that will set you apart include prior involvement in AI‑assisted software engineering, code generation, or programming language design, as well as familiarity with the specific CUDA‑X math libraries such as cuBLAS, cuFFT, cuSOLVER, and cuSPARSE. NVIDIA offers a dynamic environment where the most talented engineers push the boundaries of computing, and you will be part of a globally recognized team that values creativity, autonomy, and challenging problems. What the role asks for: - PhD or MSc in Computational/Computer Science, Applied Math, or related field - 3+ years relevant professional experience - Strong knowledge of LLMs, finetuning, RL, RAGs, MCP, agent tooling - Proven ability to design clear, composable APIs and write documented code - Ability to prioritize multiple projects and work independently - Excellent collaboration, communication, and documentation habits - Nice-to-have: Experience in AI‑assisted software engineering or code generation - Nice-to-have: Familiarity with CUDA‑X math APIs (cuBLAS, cuFFT, cuSOLVER, cuSPARSE)