00:05:08 · Jim Fan
At that time, we didn't even have transformers, so it was like very old sequence models called LSTM, and we talked about having a data pipeline. We talked about scaling up data and the model back in 2015. So it was a very simple, elegant approach, and it turns out that it solved speech recognition really, really well, much better than traditional systems. And then 2016, that's when... That was the last summer internship before I started my Ph. D, and I heard about a new company, a new startup with a bunch of brilliant people, and that startup turns out to be OpenAI. And I reached out, and that they they said they don't have an intern program, but maybe it's time to create one. So I had the great honor to to join OpenAI in that summer, and the project I worked on was called World of Bits. And I was mentored by none other than the great Andrej Karpathy. So it was his idea, his vision, and I was very gladly part of part of that vision. So the idea of World of Bits was that we wanted to teach an AI agent to learn to use the web browser by observing the pixels rendered from the browser and then uploading keyboard and mouse controls directly. It's actually part of a much larger initiative at OpenAI at that time called Universe, and it's more general. Basically, anything that you do on a computer can, in principle, be this interface of mapping pixels from the screen to keyboard and mouse actions. So you can play games. You can use software like Photoshop, Microsoft Word in exactly the same way. So that project is really about building a general purpose interface for the agents. And once you have that general purpose interface, then you can enable end to end agents that maps from pixels all the way to actions. I think at that time, it was it was revolutionary. Like, I have never seen anything like that before. Right? Like, we were... You know, the AIs at that time were struggling to play very simple games, let alone solving general purpose computer use, and we're using reinforcement learning. There was no foundation models. Right? Everything was kind of trained from scratch. So in 2016, that was way, way ahead of its time. But, you know, like, this year, 2025, right, nine years later, we kind of... You know, history repeats itself. Or or really, it doesn't repeat exactly, but it rhymes. Right? We're starting...
00:07:48 · Jim Fan
We are starting to solve the world a bit using modern LLMs and also reinforcement learning, but this time, it's not from scratch. Right? It's aural reasoning on top of LLM, then we have things like computer using agents, MCPs that can use tools. We're finally starting to solve that. So even though we didn't... We were not able to completely solve in 2016, but I think that summer's internship really shaped my research taste, which is that, you know, the data pipeline can be very complex, but a model interface needs to be very simple because only simple and elegant methods can leverage all kinds of the data sources. Only the simple methods can scale the best with compute. Right? It's really a perspective on how to approach these hard problems. So that was the summer. And then I started my PhD with Professor Fei Fei Li at Stanford, and as everyone knows, Fei Fei doesn't need an intro. She's the world renowned scientist on computer vision. She was known for ImageNet, and at that time, computer vision was about image segmentation, benchmarks, classification, video video annotation, and so on. And we call those static computer vision. You have a data set, you have static images and videos, and you map that to some sort of labels. Now Fei Fei's vision was that static computer vision is hitting a wall, and the next big revolution would be embodied vision, meaning that computer vision should be solved with agency. So you can have visual agents in simulation and in the real world, and the visual agents in the real world is basically robotics. So from 2016 to 2021, over the five years I was at the lab, I basically witnessed Fei Fei's lab's transition from static to in body vision and then to have a new north star of solving these kind of robotics and then visual understanding, but in an active perceptive loop. So that was very inspiring, and again, that's kind of a second major reshaping of my research taste. And then graduated, went to NVIDIA. NVIDIA is still my first and only full time job, and I've been carrying on a lot of ideas from my PhD work into building general purpose robotics and robot foundation models. And sorry, that's a very, very long intro, but I do hope to look back and see kind of all, like, how all these threads tie together. You know, it's VIBE research.
00:10:27 · Molly Welch
[00:10:27] S0: What a wonderful framing, vibes based research, and so many great threads that I'm excited to pull on in our conversation today over the several exciting acts of of your career so far. And maybe to touch on this first act that your internship at OpenAI, which feels, you know, it it kind of insane to conceptualize a... You know, OpenAI in the earliest days, also this type of research, which as you said, is so relevant today, and so much in the zeitgeist. These same ambitions were... Are now possible, uniquely possible with frontier models and and, high compute RL, as you described. So take us back to this first internship at OpenAI twenty sixteen. This is World of Bits before LLMs become really central to the work at at OpenAI. What was that experience like, and how did it shape your perspective on AI research and the broader industry?
00:11:29 · Jim Fan
[00:11:29] S1: Absolutely. At that time, in OpenAI, there was the OpenAI universe that I worked on, and there was also a few other exciting projects. One is the DOTA project, so that was called OpenAI five, basically to train auto agents to solve games like DOTA, and there was also actually a robotics project at that time to use high degree of freedom, shadow hands, to solve Rubik's cube. So later, we're going to talk about some research like Eureka that actually reinstantiated some of these ideas and then trained the dexterous hands to do a lot more. So I think at that time, there were a lot of building blocks, even as early as 2016, and the most significant one being reinforcement learning and sequence models. But it wasn't until 2017 and 2018 where kind of the transformer architecture comes, and then it becomes this data sponge where you can really scale on the data and compute, and you will see magic happening from pretraining. So after that moment, we're starting to get into the pre training age of foundation models, and then RO comes back, and as I mentioned, history doesn't repeat, it rhymes. RO comes back and then really takes off from this strong foundation. The foundation needs to be able to do reasoning and needs to be able to do in context learning, all of those capabilities, and then RO can push it to the next frontier. So that experience at OpenAI also inspired one of my first projects at NVIDIA. So that was a pair of projects you mentioned at the beginning, MineDojo and Voyager, where we wanted to have these embodied agents to solve Minecraft. And I I think, you know, for that project, it was also one of my excuses to play games at work. It's all for science. Right? I'm collecting data. Right? I'm not stacking up. I'm collecting data.
00:13:29 · Molly Welch
[00:13:29] S0: All in the spirit of data collection.
00:13:31 · Jim Fan
[00:13:31] S1: It's it's all for AI. It's all for pushing the frontier of AI. So I think what was fun about that project was Minecraft is this open ended environment. It's also a simulation, meaning that you can basically generate infinite data of almost infinite diversity in Minecraft. Minecraft. I've seen people doing crazy things inside this sandbox game, right, like building the Hogwarts Castle brick by brick or even building a functioning CPU circuit using those, like, fundamental building blocks in Minecraft that actually makes the game Turing complete. So this game has infinite diversity, yet humans played it very well, but for AI, it was lagging way behind. It couldn't even do some basic things. So at a time when we started the Voyager project, that coincides with the introduction of g d four. That was the first foundation model that can do coding extremely well. So how we apply g d four to Minecraft is that we give it an API so that g d four can read the instructions of the Minecraft API and then write code, and then we execute that code in the Minecraft simulator to drive the agent to mine blocks, right, to explore, to travel, right, to dig, to craft new items. So, basically, we we we turned the game into a text environment, and it communicates by code. And then we found that g d four is not only good at in context learning, it's also very good at a capability called self reflection, so it can run through a box. No one can write perfect program at the beginning, like in one shot, not even experienced programmers. You've got to iterate, and that's what we observed with gmd4. It writes code, tests it, and the code breaks, it sees bug, it debugs, it tries again, and once it writes a good code, a good piece of code, it would commit to a memory, so we call this kind of an on the fly code repo. So g m d four will save some of the the code it wrote to this repo, so later it can retrieve it if it sees a similar situation. And in this way, we we couldn't actually fine tune the parameters of a GD four, but it still learns over time because it's committing code, retrieving code, and doing trial and error, and it accumulates the context as it goes. So it's a kind of in context lifelong learning.