Rithesh Kumar

About

I work on building audio interfaces to AGI at OpenAI. We are entering an era of increasingly intelligent, agentic models that can gather context at enormous scale asynchronously, while still needing to listen, perceive, and respond in real-time and present information in a human-like way.

Before OpenAI, I was a Senior Research Scientist at Adobe Research, where I led speech generation efforts across controllable text-to-speech, automatic dubbing, and speech editing. My work centered on scaling diffusion models and developing efficient distillation algorithms for multilingual audio generation.

Before Adobe, I was Technical Lead for Audio Research at Descript Inc., where I built and shipped 4+ text-to-speech models powering the flagship Overdub and Regenerate features—enabling ultra-realistic voice cloning and text-based audio corrections.

My work has been rooted in the fundamentals of generative modeling and deep learning since 2016. I had the privilege of completing my M.Sc. in Computer Science (2017–2019) at the Mila lab in Université de Montréal under the supervision of Prof. Yoshua Bengio.

Experience

Shipped Products

2025 Adobe Firefly Translate Video and Audio, Text-to-Avatar

Speaker-preserving voice translation for multilingual media workflows, plus voice generation for animated avatars.

2023 Descript Regenerate

Speech editing and audio repair that turns awkward cuts, gaps, and retakes into smooth generated speech.

2018+ Descript AI Voices

Voice cloning and text-to-speech systems behind Overdub and later AI voice creation workflows.

Selected Publications

1. ICASSP 2026

   Taming Audio VAEs via Target-KL Regularization

   Prem Seetharaman*, Rithesh Kumar*.

2. ICML 2026

   AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing

   William Chen, Prem Seetharaman, Rithesh Kumar, Oriol Nieto, Shinji Watanabe, Justin Salamon, Zeyu Jin.

3. ICLR 2026

   SpeechOp: Inference-Time Task Composition for Generative Speech Processing

   Justin Lovelace, Rithesh Kumar, Jiaqi Su, Ke Chen, Kilian Q. Weinberger, Zeyu Jin.

4. ICML 2025

   DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis

   Yingahao Aaron Li, Rithesh Kumar, Zeyu Jin.

5. NeurIPS 2023Spotlight

   High-Fidelity Audio Compression with Improved RVQGAN

   Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, Kundan Kumar.

6. ISMIR 2023

   VampNet: Music Generation via Masked Acoustic Token Modeling

   Hugo Flores Garcia, Prem Seetharaman, Rithesh Kumar, Bryan Pardo.

7. NeurIPS 2019

   MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

   Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, Aaron Courville.

8. M.Sc. Thesis2019

   Maximum Entropy Generators for Energy-Based Models

   Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, Yoshua Bengio.

9. ICLR 2017

   SampleRNN: An Unconditional End-to-End Neural Audio Generation Model

   Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, Yoshua Bengio.