00:00:03 · Molly Welch
[00:00:03] S0: Welcome to a special edition of Radical Talks. I'm Molly Welch, I'm a partner here at Radical Ventures. Today, we're taking you inside our annual Radical AI founders masterclass. It's an exclusive program where AI researchers and technical entrepreneurs learn directly from industry pioneers, drawing inspiration and practical insights from their journeys. In this mini series, we'll share these conversations from our twenty twenty five sessions, giving you access discussions helping AI founders build innovative companies that shape the future. Today, we are featuring a conversation between myself and NVIDIA's director of AI and distinguished scientist, Fan. This one was a lot of fun. So let's dive in. My name is Molly Welch, and I'm a partner here at Radical Ventures. We're an AI focused venture fund, and we're thrilled to have you all here for our annual master class. We have a special session today. We have the privilege of speaking with Jim Fan, a leading voice in AI agents and robotics research. Jim is fresh off of NVIDIA GTC. I think he's still in DC, so we're gonna have a lot of fun things to speak about today. Jim is a director and a distinguished research scientist at NVIDIA, where he co leads development of the Isaac GR00T platform for humanoid robotics and heads the GEAR or generalist embodied agent research team. Jim earned his PhD from the Stanford Vision Lab, where he was advised by Fei Fei Li. His groundbreaking work includes Voyager, the first AI agent to master Minecraft through lifelong learning, MineDojo, which won the NeurIPS twenty twenty two Outstanding Paper Award, and Eureka, a project teaching robot hands complex, dexterous tasks like pen spinning. Jim has also the unique distinction of being OpenAI's very first intern back in 2016 where he co developed World of Bits, and we're gonna hear a little bit more about that later. Jim, we're thrilled to have you. Welcome to the Radical AI Founders master Class.
00:02:02 · Jim Fan, Molly Welch
[00:02:02] S1: Thank you so much for having me, and thanks for the very kind intro, Molly. [00:02:07] S0: We're we're really pleased to have you. Jim, I wanna start with this incredible journey that you've been on from OpenAI, Stanford's Vision Lab, now leading embodied agents at NVIDIA. Walk us through your journey, your decision making processes at each transition. And in particular, we wanna hear about what drew you from academia to some of these roles in the industry.
00:02:31 · Jim Fan
[00:02:31] S1: Wow. That's a huge question. So let me break this out. I think, you know, in in retrospect, it's really serendipity and following some good vibes. I'm calling it vibe research that I did over the last ten years. I would say the vibe I was following is basically two things. One is picking hard problems, and the second is to seek simple solutions. And I feel like a lot of the opportunities that I hopped on, a lot of the projects are all following these two principles. You know, I got interested in deep learning first in 2012, and that's when AlexNet first came out. And I was really attracted by its elegance, right? Like, know, before 2012, computer vision is a very complex pipeline. You have hand engineered features, right, all the geometric analysis on pixels, and then SVM and a bunch of statistical methods to to reach a classification of the image. And all AlexNet did was mapping pixels directly to a category, Right? So, like, a thousand different objects, and then you pick one out of a thousand. That's it. Very simple, very elegant. So my first deployment research project was actually an astrophysics project at a Columbia computer vision lab. So the goal is to estimate gravitational lens parameters from astrophysics images, and that was around 2013, some of 2013 to 2014. And, you know, at that time, was still very kind of traditional pipelines. Right? Like, you have SIF features and SVM to do classification. I decided to implement something like a convolutional neural net, but at that time in MATLAB, and then I found out they actually blew everything out of the waters. It's a very simple method. If you have a small training set, you do gradient descent, and the system programs itself. So that was kind of my first contact with deep learning. And then in 2015, I joined Andrew Ng's lab to work on a project called Deep Speech, And, you know, as the name suggests, it's for speech recognition. And, again, at that time, there are a lot of traditional systems that first do audio segmentation and then phoneme classification, wave feature extraction, and finally a ton of probabilistic scoring models. But at that time, for deep speech, we believed in end to end. Basically, you take a raw audio wave in and then output text, a sequence of text, and that's it.
00:05:08 · Jim Fan
At that time, we didn't even have transformers, so it was like very old sequence models called LSTM, and we talked about having a data pipeline. We talked about scaling up data and the model back in 2015. So it was a very simple, elegant approach, and it turns out that it solved speech recognition really, really well, much better than traditional systems. And then 2016, that's when... That was the last summer internship before I started my Ph. D, and I heard about a new company, a new startup with a bunch of brilliant people, and that startup turns out to be OpenAI. And I reached out, and that they they said they don't have an intern program, but maybe it's time to create one. So I had the great honor to to join OpenAI in that summer, and the project I worked on was called World of Bits. And I was mentored by none other than the great Andrej Karpathy. So it was his idea, his vision, and I was very gladly part of part of that vision. So the idea of World of Bits was that we wanted to teach an AI agent to learn to use the web browser by observing the pixels rendered from the browser and then uploading keyboard and mouse controls directly. It's actually part of a much larger initiative at OpenAI at that time called Universe, and it's more general. Basically, anything that you do on a computer can, in principle, be this interface of mapping pixels from the screen to keyboard and mouse actions. So you can play games. You can use software like Photoshop, Microsoft Word in exactly the same way. So that project is really about building a general purpose interface for the agents. And once you have that general purpose interface, then you can enable end to end agents that maps from pixels all the way to actions. I think at that time, it was it was revolutionary. Like, I have never seen anything like that before. Right? Like, we were... You know, the AIs at that time were struggling to play very simple games, let alone solving general purpose computer use, and we're using reinforcement learning. There was no foundation models. Right? Everything was kind of trained from scratch. So in 2016, that was way, way ahead of its time. But, you know, like, this year, 2025, right, nine years later, we kind of... You know, history repeats itself. Or or really, it doesn't repeat exactly, but it rhymes. Right? We're starting...
00:07:48 · Jim Fan
We are starting to solve the world a bit using modern LLMs and also reinforcement learning, but this time, it's not from scratch. Right? It's aural reasoning on top of LLM, then we have things like computer using agents, MCPs that can use tools. We're finally starting to solve that. So even though we didn't... We were not able to completely solve in 2016, but I think that summer's internship really shaped my research taste, which is that, you know, the data pipeline can be very complex, but a model interface needs to be very simple because only simple and elegant methods can leverage all kinds of the data sources. Only the simple methods can scale the best with compute. Right? It's really a perspective on how to approach these hard problems. So that was the summer. And then I started my PhD with Professor Fei Fei Li at Stanford, and as everyone knows, Fei Fei doesn't need an intro. She's the world renowned scientist on computer vision. She was known for ImageNet, and at that time, computer vision was about image segmentation, benchmarks, classification, video video annotation, and so on. And we call those static computer vision. You have a data set, you have static images and videos, and you map that to some sort of labels. Now Fei Fei's vision was that static computer vision is hitting a wall, and the next big revolution would be embodied vision, meaning that computer vision should be solved with agency. So you can have visual agents in simulation and in the real world, and the visual agents in the real world is basically robotics. So from 2016 to 2021, over the five years I was at the lab, I basically witnessed Fei Fei's lab's transition from static to in body vision and then to have a new north star of solving these kind of robotics and then visual understanding, but in an active perceptive loop. So that was very inspiring, and again, that's kind of a second major reshaping of my research taste. And then graduated, went to NVIDIA. NVIDIA is still my first and only full time job, and I've been carrying on a lot of ideas from my PhD work into building general purpose robotics and robot foundation models. And sorry, that's a very, very long intro, but I do hope to look back and see kind of all, like, how all these threads tie together. You know, it's VIBE research.
00:10:27 · Molly Welch
[00:10:27] S0: What a wonderful framing, vibes based research, and so many great threads that I'm excited to pull on in our conversation today over the several exciting acts of of your career so far. And maybe to touch on this first act that your internship at OpenAI, which feels, you know, it it kind of insane to conceptualize a... You know, OpenAI in the earliest days, also this type of research, which as you said, is so relevant today, and so much in the zeitgeist. These same ambitions were... Are now possible, uniquely possible with frontier models and and, high compute RL, as you described. So take us back to this first internship at OpenAI twenty sixteen. This is World of Bits before LLMs become really central to the work at at OpenAI. What was that experience like, and how did it shape your perspective on AI research and the broader industry?