Y Combinator Startup Podcast
00:17:28 · Aaron Epstein, Marvin Schwaibold
[00:17:28] S0: Mhmm. [00:17:29] S1: And it's it's become this beautiful process where once a demo is at a point where we kind of all align, everybody branches off and builds a version Mhmm. Of something that they think they... That brings value. And then we all come back together, and it actually lives all inside the same the same demo and the same link. Yeah. And it's just a beautiful way to to to to share ideas and and and context. [00:17:49] S0: You said designers are building this [00:17:52] S1: Yeah. [00:17:52] S0: In here. A lot of times, designers are designing. And they're not building, but it's... In in Shopify and your team and you, you are building a lot of the stuff that you are designing so that you can test it, so you can prototype it, you can see it on live live data, live experiences. Yeah. And that's a huge part of of nailing it and building something useful and making sure that it's great. What was your evolution from, you know, designing in Figma Yep. To now you are not just designing, but you're actually creating it out into the world to be able to test it and doing that full loop yourself.
00:18:24 · Marvin Schwaibold
[00:18:24] S1: In this specific case, admin has a a playground that we build. Instead of that playground, we test. That's not production code, and that wouldn't be allowed back into the main Yep. GitHub repo of Shopify into, like, our main repository. Having said that, though, AI has definitely gotten to the point now where, as designers, we are highly empowered Mhmm. And capable to to visualize any idea that we want to get our idea across. In a previous world, that that just wasn't possible. We we looked at the at at the Collins website earlier. A lot of that work in Figma, when we worked on it, still looks like still looks like this. Mhmm. You can zoom into any part here, and they're, like, highly detailed Figma screens that that that we're designing through, and this is an ideation process. I think this is a mixture of me, Marvin Schwaibold, and Jay Tull working on this project together. And you can just see the evolution of, like, as we're going through the inter... Like, as we're going through ideas and, like, trying out new things, these are... The the... These are too heavy and too expensive to test in code. We don't wanna test this in code right now because it would require an engineer to take a look at it and build it for us, so we tested all in Figma. Figma is the playground in the in the world we we were building everything in. We we were building it on desktop, and then we were translating it to mobile, and all of this had to be stress tested out in Figma first for a very long time before we would go and and bug the engineering team. A lot of the way I'm talking to AI or I'm still using Figma at the moment is I can get really far with a very broad sketch.
00:19:50 · Aaron Epstein, Marvin Schwaibold
[00:19:50] S0: Mhmm. [00:19:51] S1: Here's an example of a side project I've been working on over the weekend, which I can go into into detail a bit, but one of the things I wanted to build out for this for this project was a a two by three overview of these grass studies and these videos I took, and the interaction here is you can open up any one of these videos, and you can drag down to to close them, and it works works the same in the in the simulator. I I explained this... I I I described this to AI, and and and I was working in Codex, and Codex built this out for me, but the interaction wasn't interruptible. So every time I clicked on an asset and pulled it down, I had to wait until the asset was all the way back in its position Mhmm. [00:20:31] S0: And
00:20:31 · Marvin Schwaibold
[00:20:31] S1: sat down before I was able to tap on another image. And that's just not how you wanna use this pattern. You... This has to be very responsive very quickly. Otherwise, you'll get very annoyed with the interface very fast. Mhmm. So I just went into into Figma and kind of drew out very rudimentary simple boxes, like, when when you click on the box, red comes up, you can swipe it away. If as you're swiping away, you click on blue, blue now comes up. Mhmm. And you can click that away and you do it with green. And I think that's just like a a good example of of of how, at least myself personally, I'm I'm using I'm I'm using Figma more. It's more a scratch pad to, like, map out ideas Mhmm. And I give those ideas to Codex. And I and I use the Scratchpad a because it's easier for me to to decompile something, but also it's actually easy for Codex to decompile it if it sees it visually. It interprets that image, I think, to how I interpreted it. At least it understood what I was doing and how I was thinking about the motion. I couldn't articulate it via via text or voice, but I could draw it out. There's two parts of this that I think are important to call out. I think number one, Codex and all the current AI software is is... And the models are extremely capable at creating anything. You can use these tools to make anything that you want, but you have to know what you're doing, And you have to be able to take a a concept or or something more complicated like the shader and break it down into all its little components. You have to be able to understand those components, and then you can put those components back together.
00:21:55 · Aaron Epstein, Marvin Schwaibold
[00:21:55] S0: Yeah. Just show us how you did that here because, so to be clear, you had a bunch of images [00:22:00] S1: And videos. [00:22:01] S0: And videos of of actual grass. Yeah. And then you wanted to create a shader Yes. That looked identical to that, but you could control and function like a shader. And so talk about how... Like, the different elements and what you had to go through to actually get that to look lifelike. [00:22:16] S1: I have a few screenshots here. Again, this is a good example of how I use of how I use Figma. [00:22:20] S0: Mhmm.
00:22:20 · Marvin Schwaibold
[00:22:20] S1: So I fed the video into into into Codex for the first time and just explained it my... The the the project and the process that I was... I wanted you to go through. Take the video, turn this into a shader. I want the shader running running in the web, and I want you to be as as as meticulous as you can, be very pixel perfect, grab all the colors, make sure that you're using the same hex codes, understand each part of the image, and then try to recreate that. And it went off and said, no problem. I got you, and it came out with this, which obviously doesn't look at all where we landed... Where we ended up now. Yeah. But then the process from here was I would screenshot this and my original my original video. Mhmm. I would open up ChatGPT, and I gave it to another agent there. I explained that agent exactly what I was doing as well. I said, why is the other agent not understanding what I'm trying to get to? And that ChatGPT then helped me kind of, like, understand my own process. Like, well, you have to explain to it where the light is coming from and what what shadows you're using, and maybe break the image actually down into, like, six parts and have it do each one of those individually and put that back together. Mhmm. And as I was kind of thinking about all of these different things, like, I was trying to come up with how could I become very clear with the... With with AI and explain to it exactly what it was actually looking at. So one of the tricks I used was I went into ChatGPT, and I had it render basically a topographical map of the underlying structure. Mhmm. AI, as I was going through this, I was... It was kinda getting closer, you could see that, you know, it was painting more pixels into into the grass blades here, and these started to look like bushels a little bit, which which felt more realistic. And then over here, we... I was like, okay. We're getting somewhere here now. This doesn't look this doesn't look terrible anymore. But I kind of figured out it doesn't understand that there's... It's not a flat layer underneath. These are little hills. Right. Basically, I fed this back into into AI, and and and it understood, oh, okay. So there's there's a... An underlying structure in a map that has different depths and heights. Mhmm. And if I can rerender the grass on top of those...