Latent Space: The AI Engineer Podcast
00:10:36 · Shawn Wang
It's called Airbyte. It's still doing okay today, but nobody will call it a rocket ship. That's just... That's objectively true, and I will say that to the founders. But because I was like, okay. I have... I I did, like, some machine learning courses, but I'm not an AI expert. I am so interested in this field. There's there's only one way for me to get into the field, is to learn in public. And so that's what I did. I started a blog, and I started analyzing and learning and, like, writing up my learnings into a blog. And at first, nobody will notice, but then eventually, if you start talking about people's things, like, literally, Emad Mostaque, who is the CEO of Stability AI, which created Stable Diffusion, which is the the reason I started the blog, read the blog post analyzing stable diffusion and said this is good. And I think that work was basically me learning AI, like the past five years of just grinding and and and writing about that. And I think that is really the... You know, like, you are not qualified, and I'm still probably not that qualified today, but I'm a bit more qualified than I used to be four years ago. So that's good. And I think that's the only way you can learn. Like, I think, especially in Asian cultures, there's a lot of emphasis on credentialism. Yeah. Like,. Right? Like, you need that piece of paper. You need, like, some qualification. Like, if it's not if it's if it's not a degree, it's a certification. If it's not a certification, you must have had a job to do it. And how do get the job? You must have a degree. Like, it's just like a infinite loop of things that basically mean that when you're 16, you need to decide what you're do in But, like, I think life is a lot more flexible than that, and you can choose to pivot if you want to. You just have to be really persistent and also realize that just because you work very hard, you're not entitled to win. You have to work very hard and be good. I think a lot of people try, a lot of people fail, and they're like, why didn't they fail? Well, you're not good. Too bad. But some people try and they they worked and and they had judgments. They really did work on substantial things rather than insubstantial things, and that's what... That's generally what happens with learning public. So so, yeah, that start...
00:12:45 · Shawn Wang
This started Latent Space, and then very, very interestingly, Latent Space was just a blog. It was just Stars was just a blog. Yes. Just a blog, and I had no other no other idea apart from this is Swyx's writing about AI. That's it. But we happened to host some events, like, gatherings, like, oh, like, you know, ChatGPT came out, like, obviously, the most exciting thing of all time. And then we wanted to host some hackathon, so we hosted a hackathon. And at the hackathon, that's where the idea for AI Engineer came about, which is the realization... Actually, it was this woman that came up to me and said, I'm I'm all in on AI. I'm nontechnical. What can I do? And I realized that the simple answer is based... Compared to an engineer, she can do nothing. Engineers are about to be so much more powerful than the non engineer because engineers can control 100 AIs at once and program it. Whereas the the maximum that a non technical person can do is prompt better. Then I was like, oh, like, I think I have seen this before. So my my career... Throughout my career, I've seen, like, professionalization of roles that are nonprestigious. So what do I mean by nonprestigious? It is not prestigious to be a prompt engineer. The prestigious thing is you train large language models. You do pretraining. You do posttraining. You have to go through a PhD, you make fancy algorithms like GRPO, you know, and and you you do manifold hyperconnections, which is what DeepSeek does. Yeah. That's prestigious, and that continues to be prestigious. But I think the overcorrection that people have is that looking at everything else that is not that ML research, that is worthless. That actually was untrue. You actually do want to build harnesses, and harnesses do not require you to know advanced math or have a lot of experience in supercomputing. Like, actually, you just need to be a very good software engineer that is open minded enough to understand how to put AI into traditional software code, which is a harness. Mhmm. We didn't have the word for harness at the time, but it was very clear that engineers would be able to do things like this that normal people cannot. And they would be in demands, and there's not only that, companies are trying to hire this kind of people, and developers want to become this kind of people. They want they want a term to call themselves something.
00:14:57 · Shawn Wang, S1
And so all you have to do is figure out the right term, which is... The simplest one is AI engineer, and make it a industry, make it a discipline. And so that's what I did. [00:15:06] S1: Actually, I can share with you a side story in China is that at the time when you just mentioned advent of stable diffusion, the open source of stable diffusion, it was actually at the time that, right now, the largest AI community in China, which is called WaytoAGI, actually, that was also started from a bunch of people who are ambitious or lovers of those stable diffusion technology. I want to transform into different, like, images generation. And also it become the foundation of the ComfyUI. You're definitely familiar with that ComfyUI. [00:15:40] S0: Yes. Comfy is famous. We've had him on the podcast. [00:15:43] S1: Yeah. Yeah. Yeah. And in China as well. And then so [00:15:45] S0: Do they use ComfyUI? Yes. Oh, wow.
00:15:45 · Shawn Wang, S1
[00:15:45] S0: Do they use ComfyUI? Yes. Oh, wow. [00:15:47] S1: And also and also then they, like, gradually invite more people to learning public and contribute more contributions on online document, which was on the Lark, which was facial under Bai Dansba, which was on the Lark. So everybody contribute their knowledges, share their use cases about stable, efficient, ComfyUI, and then become larger and larger. Actually, think right now, the online community has around a million users learning together and several million users, like, learning together. So become the largest AI community in China as well. So that's also, I think, that happened exactly at the same time, I think you start to feel like you need a... We need a new word for this type of engineer, and then as... Which is you just published the essay of the AI engineer.
00:16:36 · Shawn Wang
[00:16:36] S0: Yeah. I think people... So here are some of the alternatives. I think you can best understand it by the alternatives that people were trying at the time. So there was AI Tinkerers, which is another significant community. I I know Joe and Alex well. And then there are people who are saying LLM engineers or applied AI engineer or all these all these other terminology. And I think, basically, you have to focus on the job title that people want to have. And I think that for us, that was very key. And I think I picked this up from the James Clear. Is it James Clear that wrote Atomic Habits? I think so. I I forget I forget I forget the authorship relationship, but I think it's James Clear. Basically, if you want to deeply understand or connect with somebody, you have to connect with them not on the level of their job, not on the level of their activities or hobbies or whatever, it is their identity. Like, who are you? Right. What what do you aspire to be? If you can give them the path towards being the better version of themselves, then you can build a community around that. And so a lot of non AI people wanted to be in AI. And what is the path for them? While you give them a job title that pays money, Like, then they will be loyal to you. Yeah. Yeah. Yeah. Right. And so so, Yeah. Now there's a lot more AI engineers in the world.
00:18:00 · S1
[00:18:00] S1: Yeah. And to follow-up just to my interest as well. So I know that there are just a lot of works or to do, all right, as an AI engineer. I know you mentioned harness, a of developments happen in the AI. So do you think the role of the world AI engineer has been evolved as well, or is different from what you just mentioned, like three years ago, once you started this