Y Combinator Startup Podcast
00:19:50 · Aaron Epstein, Marvin Schwaibold
[00:19:50] S0: Mhmm. [00:19:51] S1: Here's an example of a side project I've been working on over the weekend, which I can go into into detail a bit, but one of the things I wanted to build out for this for this project was a a two by three overview of these grass studies and these videos I took, and the interaction here is you can open up any one of these videos, and you can drag down to to close them, and it works works the same in the in the simulator. I I explained this... I I I described this to AI, and and and I was working in Codex, and Codex built this out for me, but the interaction wasn't interruptible. So every time I clicked on an asset and pulled it down, I had to wait until the asset was all the way back in its position Mhmm. [00:20:31] S0: And
00:20:31 · Marvin Schwaibold
[00:20:31] S1: sat down before I was able to tap on another image. And that's just not how you wanna use this pattern. You... This has to be very responsive very quickly. Otherwise, you'll get very annoyed with the interface very fast. Mhmm. So I just went into into Figma and kind of drew out very rudimentary simple boxes, like, when when you click on the box, red comes up, you can swipe it away. If as you're swiping away, you click on blue, blue now comes up. Mhmm. And you can click that away and you do it with green. And I think that's just like a a good example of of of how, at least myself personally, I'm I'm using I'm I'm using Figma more. It's more a scratch pad to, like, map out ideas Mhmm. And I give those ideas to Codex. And I and I use the Scratchpad a because it's easier for me to to decompile something, but also it's actually easy for Codex to decompile it if it sees it visually. It interprets that image, I think, to how I interpreted it. At least it understood what I was doing and how I was thinking about the motion. I couldn't articulate it via via text or voice, but I could draw it out. There's two parts of this that I think are important to call out. I think number one, Codex and all the current AI software is is... And the models are extremely capable at creating anything. You can use these tools to make anything that you want, but you have to know what you're doing, And you have to be able to take a a concept or or something more complicated like the shader and break it down into all its little components. You have to be able to understand those components, and then you can put those components back together.
00:21:55 · Aaron Epstein, Marvin Schwaibold
[00:21:55] S0: Yeah. Just show us how you did that here because, so to be clear, you had a bunch of images [00:22:00] S1: And videos. [00:22:01] S0: And videos of of actual grass. Yeah. And then you wanted to create a shader Yes. That looked identical to that, but you could control and function like a shader. And so talk about how... Like, the different elements and what you had to go through to actually get that to look lifelike. [00:22:16] S1: I have a few screenshots here. Again, this is a good example of how I use of how I use Figma. [00:22:20] S0: Mhmm.
00:22:20 · Marvin Schwaibold
[00:22:20] S1: So I fed the video into into into Codex for the first time and just explained it my... The the the project and the process that I was... I wanted you to go through. Take the video, turn this into a shader. I want the shader running running in the web, and I want you to be as as as meticulous as you can, be very pixel perfect, grab all the colors, make sure that you're using the same hex codes, understand each part of the image, and then try to recreate that. And it went off and said, no problem. I got you, and it came out with this, which obviously doesn't look at all where we landed... Where we ended up now. Yeah. But then the process from here was I would screenshot this and my original my original video. Mhmm. I would open up ChatGPT, and I gave it to another agent there. I explained that agent exactly what I was doing as well. I said, why is the other agent not understanding what I'm trying to get to? And that ChatGPT then helped me kind of, like, understand my own process. Like, well, you have to explain to it where the light is coming from and what what shadows you're using, and maybe break the image actually down into, like, six parts and have it do each one of those individually and put that back together. Mhmm. And as I was kind of thinking about all of these different things, like, I was trying to come up with how could I become very clear with the... With with AI and explain to it exactly what it was actually looking at. So one of the tricks I used was I went into ChatGPT, and I had it render basically a topographical map of the underlying structure. Mhmm. AI, as I was going through this, I was... It was kinda getting closer, you could see that, you know, it was painting more pixels into into the grass blades here, and these started to look like bushels a little bit, which which felt more realistic. And then over here, we... I was like, okay. We're getting somewhere here now. This doesn't look this doesn't look terrible anymore. But I kind of figured out it doesn't understand that there's... It's not a flat layer underneath. These are little hills. Right. Basically, I fed this back into into AI, and and and it understood, oh, okay. So there's there's a... An underlying structure in a map that has different depths and heights. Mhmm. And if I can rerender the grass on top of those...
00:24:12 · Marvin Schwaibold, Aaron Epstein
On top of that infrastructure or that structure in general, it'll be much easier for for me to to create the scene, and that kinda worked really well. And, you know, in this in this prototype, your friends, you can actually bind... There's a binary toggle to turn the grass on or off, and you can actually play with the terrain quite a bit here as well. [00:24:29] S0: Right.
00:24:29 · Marvin Schwaibold
[00:24:29] S1: This was all unlocked via Josh to me at a lunch the other day where he kind of, like... We installed DialKit into this demo Mhmm. Because he looked at this and he's like, this looks pretty good, but you should look at the length of the grass blades. I was like, woah. Okay. That's true. So I asked I asked Codex, hey, how many how many grass blade variations do we have? And it said, we have about 217,000 different grass blades, but they only differ by, like, point six, point seven, point eight. So we were like, okay. Let's just build a toggle for it so that we can see it ourselves. Yeah. I don't wanna have to go back and forth with the interface and and chat about this. Expose the underlying structure. Exposing the underlying structure would also show me where AI had made mistakes. The original version of the terrain map that it had implemented was completely wrong and and and skewed. Mhmm. So I couldn't get to my outcome anyway because AI had misinterpreted something five prompts ago that I didn't know about anyway. Dialkit kind of unpacked that for me. But then, in general, you can just mess with a few of these values, and they kind of help, like, bring bring some of this... Like, they they they help visualize Mhmm. What what I what I really wanted to see. And if if if, you know, the the length here is longer, then you you can you can introduce more wind. This was something I wanted to to test and see how how crazy I could get with the Yeah. With the wind movement. You can see it it gets a little wild. But, yeah, I think this is just a small example of a little bit of, like, a a fun project just to try and, like, see how far can you get with with the tools. And these are things that I think before AI, I I I couldn't do. I just... I I might have had these ideas, but I but but I but I wasn't allowed or or able to execute on them. And I think that's just in general what's changing with a lot of the... With with the design role right now. Any... Anyway, like, it's it's it's... Designers are becoming more full stack as are engineers and PMs. It goes it goes it goes in every which way.