The Pragmatic Engineer
01:22:40 · Titus Winters
[01:22:40] S1: Yeah. So, I mean, I think learn the basics is an entirely loaded question because nobody agrees on what the basics are. That's fair. Kent Beck said a couple months ago when we were last together something like, everyone that's currently working has grown up in an era where there was a playbook. Like, we know, at least in theory, the right way to do this. And there's no playbook right now. Right? So we all have to get a little comfortable of not knowing the answer. And we certainly don't know the answer on what is still the basics. The things that I think that the the juniors should be focusing on, one, the resilience to avoid just copying your ticket into AI prompts and then trying to check-in that result. Right? Like, if you're... If that is all that you are doing, you can be replaced with a very small shell script. Like, that is not a defensible advantage for you being on staff. I think it's really important to use AI to better yourself and to widen your inputs, but not to immediately produce all of your outputs. There's a metaphor that I really liked from about a year ago at a think tank. We called it Andrew's axe. And it's, if if I go chop down a tree with an axe, I do not say that it was the axe that chopped down the tree. It's me that chopped down the tree. The AI, as much as we occasionally anthropomorphize it, right, is still a tool. And 1979 IBM training manual, a computer will never be held accountable. Therefore, a computer should not be making management decisions. The AI is still not accountable. The AI labs are not going to indemnify us for, you know, the mistakes that these things produce. So you have to be able to stand up for whatever your name is on for those poll requests. And that requires a lot more rigor and not cutting those corners. That attention to detail, I think, is is really hard to train. Going back to the sort of the undergraduate version of this, the story... The scary story that I've been hearing from a lot of profs is the best students, like the b b plus and above students, they are better than they ever have been. Already? Yeah. Because instead of relying on a TA that is available, you know, a couple hours a week and is one or two grades above them, they have an infinitely patient, always available, incredibly well read, sometimes hallucinatory, tutor, right, at the drop of a hat. Right?
01:25:21 · Titus Winters, Gergely Orosz
They can always learn and explain and figure And [01:25:24] S0: they actually have the autonomy and the drive to use this. [01:25:28] S1: Right. And the and the metacognitive resources to avoid that, like, I'm just gonna copy and paste. Right? The very worst students are somewhat better because they used to not be able to do anything, but now they can produce slop. [01:25:41] S0: Yeah. They just throw in slop. [01:25:43] S1: Right. Which is a bit [01:25:43] S0: hard to detect, but I I get what But you're [01:25:46] S1: everyone in the middle has kinda collapsed. They don't have that resilience. They would rather go to the party or play video games than do the homework. And so they're like, well, just this one time, I'll just get in the habit of doing this thing. And that slippery slope just gets worse. Right? And so there are a lot of profs from decent schools that are like, yeah, I don't know, man. I'm not sure that anyone below the vast middle, everyone below a b, might not understand how code works at all. Like, the system is just not set up to be tracking that sort of thing. [01:26:21] S0: Or at least giving them feedback, for example, alerting Right. Those kind of things. [01:26:25] S1: Yeah. And so a lot of the discourse initially was how do we prevent students from using these AI tools? And I think that that's nonsense because it's echoes of, you we got high level languages, and there's compilers. [01:26:39] S0: Yeah. Like... [01:26:40] S1: But now you're not gonna learn how to program in assembly. [01:26:42] S0: Right? Like The industry is using it anyway.
01:26:42 · Gergely Orosz, Titus Winters
[01:26:42] S0: Right? Like The industry is using it anyway. [01:26:45] S1: Right. So They're gonna be asked to use... They're gonna be forced, pressured to use it harder in some places when they get a job. [01:26:52] S0: Well, in fact, it might be... It will for sure be a red flag if a new grad says, like, I have not used these tools. Like, come on. Right. Like, have you not... Yeah. Do you not wanna work in this industry? It's it's an open secret what we're doing. We're publishing it on blogs and everywhere.
01:27:05 · Titus Winters, Gergely Orosz
[01:27:05] S1: Yeah. Exactly. Right? And so the line of, well, we're gonna never let you use these tools cannot possibly hold. I co published a piece in communications of ACM with a bunch of academics this month That I think the opening line there is imagine forward to 2040, and we hold the line treating all use of AI as cheating. But a generation of students, AI... You know, they've grown up with AI in the same way that the Internet natives have grown up with the network. Right? If we keep... Continue asking them these traditional questions about, like, find a palindrome in this string or measure the volume of rainfall in this graph. Right? Like, are they going to learn anything from a traditional CS degree? I think the answer is obviously no. So, like, if we know that we cannot stay here, and there's plenty of things that need fixing, let's burn it down and do something that matters and figure out how to use this appropriately. And I think finding ways to assume that they're using AI, and what are the new learning objectives, and how do you make that still frictionful in the ways that cause them to think and learn about different things, like, almost certainly at a higher level. Right? But, like, that's actually the key to undergraduate education and curriculum design and assessment. [01:28:41] S0: Well, sounds like whoever's... So the the professors, the the TAs, the people who are designing this course, they also need to put a bunch of work because no one has us figure it out. You need to experiment. You need to first use the tools yourself. [01:28:54] S1: Yep.
01:28:54 · Titus Winters, Gergely Orosz
[01:28:54] S1: Yep. [01:28:54] S0: You need to come up with novel ways of creating an assignment which you need to use AI to solve, but AI itself is not enough. How how do you get the feedback? You might need to put in more time. Let's say, like, sit down with someone afterwards and have them explain to you because, you know, again, like, interviews are also changing. And one of the interviews that seem to be working well is when you're pairing with someone, you know, like video or in person, you're using AI, but you're explaining what you're doing and how. Right? Because now you you cannot just fully fake it and go away, give it to a friend or or just have Claude do it. You're you're now showing your cognitive ability. The interviewer can push you into uncomfortable, you know, they'll they'll get a gauge of like how expert you are and push you further and further
01:29:40 · Titus Winters
[01:29:40] S1: and so on. Yeah. Exactly. And at the undergraduate level, like, there is some move towards more oral exams and those sort of things. But also, I think a really simple solution is stop grading just the outcome. Focus on the process. Right? Like, even if you don't come up with wildly harder assess... Assignments for assessment, Stop grading just the code that they turned in. Grade instead, what are the prompts that you used to get here? Also, what are the tests that you provided? Like, how did you make sure that this thing is actually doing what you need it to do? And when you start focusing on the process that got you there instead of just the outcome, I think not only CS, but most disciplines. Right? We've been using... Go write a new essay on Catcher in the Rye. We don't need another essay on Catcher in the Rye. Go write a new linked list. We don't need another linked list. We've been using the artifact that's produced from that as a proxy for you put in the time and the struggle and faced the friction that we assume that you now have learned something. It might be more appropriate to just focus on, do we have evidence that you've struggled and learned?