Y Combinator Startup Podcast
00:18:24 · Marvin Schwaibold
[00:18:24] S1: In this specific case, admin has a a playground that we build. Instead of that playground, we test. That's not production code, and that wouldn't be allowed back into the main Yep. GitHub repo of Shopify into, like, our main repository. Having said that, though, AI has definitely gotten to the point now where, as designers, we are highly empowered Mhmm. And capable to to visualize any idea that we want to get our idea across. In a previous world, that that just wasn't possible. We we looked at the at at the Collins website earlier. A lot of that work in Figma, when we worked on it, still looks like still looks like this. Mhmm. You can zoom into any part here, and they're, like, highly detailed Figma screens that that that we're designing through, and this is an ideation process. I think this is a mixture of me, Marvin Schwaibold, and Jay Tull working on this project together. And you can just see the evolution of, like, as we're going through the inter... Like, as we're going through ideas and, like, trying out new things, these are... The the... These are too heavy and too expensive to test in code. We don't wanna test this in code right now because it would require an engineer to take a look at it and build it for us, so we tested all in Figma. Figma is the playground in the in the world we we were building everything in. We we were building it on desktop, and then we were translating it to mobile, and all of this had to be stress tested out in Figma first for a very long time before we would go and and bug the engineering team. A lot of the way I'm talking to AI or I'm still using Figma at the moment is I can get really far with a very broad sketch.
00:19:50 · Aaron Epstein, Marvin Schwaibold
[00:19:50] S0: Mhmm. [00:19:51] S1: Here's an example of a side project I've been working on over the weekend, which I can go into into detail a bit, but one of the things I wanted to build out for this for this project was a a two by three overview of these grass studies and these videos I took, and the interaction here is you can open up any one of these videos, and you can drag down to to close them, and it works works the same in the in the simulator. I I explained this... I I I described this to AI, and and and I was working in Codex, and Codex built this out for me, but the interaction wasn't interruptible. So every time I clicked on an asset and pulled it down, I had to wait until the asset was all the way back in its position Mhmm. [00:20:31] S0: And
00:20:31 · Marvin Schwaibold
[00:20:31] S1: sat down before I was able to tap on another image. And that's just not how you wanna use this pattern. You... This has to be very responsive very quickly. Otherwise, you'll get very annoyed with the interface very fast. Mhmm. So I just went into into Figma and kind of drew out very rudimentary simple boxes, like, when when you click on the box, red comes up, you can swipe it away. If as you're swiping away, you click on blue, blue now comes up. Mhmm. And you can click that away and you do it with green. And I think that's just like a a good example of of of how, at least myself personally, I'm I'm using I'm I'm using Figma more. It's more a scratch pad to, like, map out ideas Mhmm. And I give those ideas to Codex. And I and I use the Scratchpad a because it's easier for me to to decompile something, but also it's actually easy for Codex to decompile it if it sees it visually. It interprets that image, I think, to how I interpreted it. At least it understood what I was doing and how I was thinking about the motion. I couldn't articulate it via via text or voice, but I could draw it out. There's two parts of this that I think are important to call out. I think number one, Codex and all the current AI software is is... And the models are extremely capable at creating anything. You can use these tools to make anything that you want, but you have to know what you're doing, And you have to be able to take a a concept or or something more complicated like the shader and break it down into all its little components. You have to be able to understand those components, and then you can put those components back together.
00:21:55 · Aaron Epstein, Marvin Schwaibold
[00:21:55] S0: Yeah. Just show us how you did that here because, so to be clear, you had a bunch of images [00:22:00] S1: And videos. [00:22:01] S0: And videos of of actual grass. Yeah. And then you wanted to create a shader Yes. That looked identical to that, but you could control and function like a shader. And so talk about how... Like, the different elements and what you had to go through to actually get that to look lifelike. [00:22:16] S1: I have a few screenshots here. Again, this is a good example of how I use of how I use Figma. [00:22:20] S0: Mhmm.
00:22:20 · Marvin Schwaibold
[00:22:20] S1: So I fed the video into into into Codex for the first time and just explained it my... The the the project and the process that I was... I wanted you to go through. Take the video, turn this into a shader. I want the shader running running in the web, and I want you to be as as as meticulous as you can, be very pixel perfect, grab all the colors, make sure that you're using the same hex codes, understand each part of the image, and then try to recreate that. And it went off and said, no problem. I got you, and it came out with this, which obviously doesn't look at all where we landed... Where we ended up now. Yeah. But then the process from here was I would screenshot this and my original my original video. Mhmm. I would open up ChatGPT, and I gave it to another agent there. I explained that agent exactly what I was doing as well. I said, why is the other agent not understanding what I'm trying to get to? And that ChatGPT then helped me kind of, like, understand my own process. Like, well, you have to explain to it where the light is coming from and what what shadows you're using, and maybe break the image actually down into, like, six parts and have it do each one of those individually and put that back together. Mhmm. And as I was kind of thinking about all of these different things, like, I was trying to come up with how could I become very clear with the... With with AI and explain to it exactly what it was actually looking at. So one of the tricks I used was I went into ChatGPT, and I had it render basically a topographical map of the underlying structure. Mhmm. AI, as I was going through this, I was... It was kinda getting closer, you could see that, you know, it was painting more pixels into into the grass blades here, and these started to look like bushels a little bit, which which felt more realistic. And then over here, we... I was like, okay. We're getting somewhere here now. This doesn't look this doesn't look terrible anymore. But I kind of figured out it doesn't understand that there's... It's not a flat layer underneath. These are little hills. Right. Basically, I fed this back into into AI, and and and it understood, oh, okay. So there's there's a... An underlying structure in a map that has different depths and heights. Mhmm. And if I can rerender the grass on top of those...
00:24:12 · Marvin Schwaibold, Aaron Epstein
On top of that infrastructure or that structure in general, it'll be much easier for for me to to create the scene, and that kinda worked really well. And, you know, in this in this prototype, your friends, you can actually bind... There's a binary toggle to turn the grass on or off, and you can actually play with the terrain quite a bit here as well. [00:24:29] S0: Right.