You've Been a Bad Agent

Matthew Carey (00:00)
Yeah, we don't give you any time to warm up.

Wilhelm Klopp (00:02)
Hello, welcome to the Bad Agent Podcast

Thorsten Ball (00:03)
Not an issue, so

as I was saying...

Wilhelm Klopp (00:06)
Okay, wait, can I actually, I wonder if I can share this because I think it would be nice to play our new podcast jingle for Thorsten.

Thorsten Ball (00:16)
it as it were real instruments.

Matthew Carey (00:17)
Yeah,

Wait, I was gonna play it on my phone next to my microphone and see if that works. That was actually my general plan. Yeah.

Wilhelm Klopp (00:23)
Wait, if I share my screen, if I...

Thorsten Ball (00:25)
That's professional. That's how you recorded

it too.

Matthew Carey (00:30)
Well, we

tried. We did an interview with Sunil and he played this like jingle on his guitar and we tried.

Wilhelm Klopp (00:37)
Okay, are you ready for this? I hope this works.

Matthew Carey (00:39)
Nothing, nothing dude. Yeah, wait, gotta play it on my phone next to my mic. One second.

Thorsten Ball (00:41)
I can't hear anything.

Wilhelm Klopp (00:42)
Look.

Oh

my god, I just got the hurt the best jiggle of my life you guys. I was worried there was such a little reaction. Okay, all right. Oh wait, wait, I'll... Yeah, wait, I'll send it to you again. I'll send it to you again.

Thorsten Ball (00:50)
Did you?

Ha ha!

Matthew Carey (00:55)
Wait, wait. Which one was the one that you liked? Was it the last one you sent me?

Thorsten Ball (01:01)
Dash4final.aac. ⁓

Matthew Carey (01:04)
Wait, okay,

no. Is it this one?

Wilhelm Klopp (01:13)
and Torson.

Thorsten Ball (01:13)
That's cool.

That's cool. That's not bad at all.

Wilhelm Klopp (01:17)
Should

we hear it again? Just... ⁓

Matthew Carey (01:19)
Yeah, okay, okay.

just so you, mate, should we hear some of your bloopers? They're even better, I think. Yeah, Will sang this, can you not hear Will singing?

Wilhelm Klopp (01:25)
No, no, no, no, that's not...

Thorsten Ball (01:26)
Are we all

saying this?

Matthew Carey (01:38)
with Matt and ⁓

Thorsten Ball (01:41)
How did you, how

did you, is this, is it AI? Or you paid an orchestra for this?

Wilhelm Klopp (01:45)
This is a woof woof woof.

Matthew Carey (01:45)
It's Will. It's

Will's organic production. No, it's Eleven Labs.

Wilhelm Klopp (01:50)
This is the

latest 11 Labs music thing, which is actually really good. I mean, to be fair, it did take a couple of attempts. As I guess is just normal with doing this kind of creative stuff, you take a couple attempts. ⁓ But yeah, it's surprisingly, and a bit of creative brainstorming or whatever, but yeah, I thought it was decent as well.

Thorsten Ball (02:10)
How does it do the whole, you know, it has like the intro thing and then the name, you know, becomes like the main thing. can you say it's a jingle or something?

Wilhelm Klopp (02:18)
Yeah.

Yeah, exactly. You tell it it's a jingle. Let me see if I can get the chat. And it's like a chat interface as well, so you can do iterations. But yeah, think you can literally tell it it's a jingle.

Matthew Carey (02:34)
Yeah, it's pretty cool. I think they announced it like a week ago or a week and a half ago. As soon as they announced it, I messaged Will and was like, dude, we've been waiting for this. We need a jingle. Because I don't know if you've listened to any of the previous ones, but we just upload this. Yeah, we upload this pretty much raw, which is kind of funny. Anyway, I was going to say, I was going to intro us as welcome back to Will interviewing his former teammates.

Thorsten Ball (02:34)
I wanna try this.

I listen to snippets,

Wilhelm Klopp (02:53)
Yeah, I can't find it.

Thorsten Ball (02:58)
There you go. did you work with Lou Wilson before? No.

Wilhelm Klopp (02:58)
Hahaha.

Matthew Carey (02:58)
That

was the way I was going to introduce this.

No, we had Danilo.

Wilhelm Klopp (03:04)
⁓

Thorsten Ball (03:05)
⁓ yeah, yeah, there you go. Yeah, didn't it? yeah.

Wilhelm Klopp (03:07)
wait, as in sorry, I do Thorsten's intro. Is that what you... I was rehearsing that as well. Sorry, I thought that's what you meant. Because we've done that a few

Matthew Carey (03:10)
No, wait, well you can do if you want.

gone. No no no let's go let's

Wilhelm Klopp (03:17)
get there, actually, don't worry. All right, so here we go. Thorsten's, yeah, total legend. You've seen him on Twitter for ages. Very incredibly strong software engineer, but also a great writer. One of the things, actually, I think we should promote here is Thorsten's perfect, beautiful newsletter. That sounds a bit like a Trumpian description. ⁓ But it's... ⁓

Matthew Carey (03:18)
Yeah, eventually, eventually.

Thorsten Ball (03:37)
Hey, you're moving the hands!

Wilhelm Klopp (03:43)
But

it's called Joy and Curiosity and it's genuinely incredible. It's just like so many beautiful tidbits of what happened in the internet and it's always a joy to read. It's also always a reminder of how productive Thorsten is and how much he is able to read in a given week, which makes me feel a little bad. But no, it's full of amazing stuff and you're on over 50 issues now, is that right? So you've been going for...

Matthew Carey (03:44)
Big beautiful.

Thorsten Ball (04:06)
Yeah,

this is now. it's going to be 52 on Sunday. And this one is going to be packed. Usually I have like an end of week rush, but.

Wilhelm Klopp (04:10)
52 here.

⁓

I had another quick look this morning ⁓ and you had a section there about ⁓ bikes and cycling, which is an interest for both of us, so we can talk about that. ⁓ yeah, Thorsten, I you were one of the first engineers at Sourcegraph, I think is just, ⁓ is that not correct?

Thorsten Ball (04:25)
⁓ yeah, there you go.

Well, well, depends on how you define first, I employee 18. I was I joined in 2019, you know, so it was pretty early.

Wilhelm Klopp (04:43)
Okay.

2019, yeah.

And then you worked at Zed ⁓ for ⁓ around a year, where we very briefly overlapped. And then you rejoined Sourcegraph to work on Amp. I think we are building all kinds of really cool agent stuff and talking about it publicly a lot as well, which is obviously exciting for the rest of us trying to figure out all of the stuff.

Thorsten Ball (04:53)
Yes.

Wilhelm Klopp (05:07)
enjoyed hanging out in the past because I think we have some similar taste and some similar preferences for certain things. So ⁓ it's nice to finally have you on the pod. Matt, anything you'd add to Thorsten's intro?

Matthew Carey (05:18)
Yeah,

the newsletter's awesome. I mean, I'm a subscriber. Amps, sick. Yeah, I also, like, I guess I was first introduced to, because I read your book, the building a compiler in Go, an interpreter first and a compiler. I was kind of interested in that because I was learning Zig. And I was like, if I do this in Zig at the same time, then I'll have learned something because, yeah, get to learn how to build a compiler and I build it in Zig. So it was really good fun.

Thorsten Ball (05:35)
Yeah.

Matthew Carey (05:42)
I guess that was my first introduction, actually introduction way before the Zed stuff actually. ⁓ Yeah, and then I next saw you in Zed. I think when Will demoed at AI demo days, you demoed Zeta, your tab completion model, which I guess that was a big thing that you worked on. ⁓ So yeah, really cool. I think we have some questions, maybe.

Wilhelm Klopp (05:53)
yeah, which is... Yep.

Thorsten Ball (05:54)
Yeah,

I worked on this with Antonio, yeah.

Yeah.

Wilhelm Klopp (06:03)
Yeah, you've done so much, so much stuff, so much varied stuff as well.

Thorsten Ball (06:07)
Can I just say what an amazing, you know, set of intros here. Thank you guys for making me blush. I don't think I... Yeah, it's nice to hear that somebody... I always say this when somebody says they like the newsletter. think compared to, you know, say writing the books years ago or being on Twitter or whatnot, it's a much lonelier activity writing this newsletter.

Wilhelm Klopp (06:09)
Ha

Thorsten Ball (06:30)
I, it's, it's, it's not like people reply to you with every issue and say amazing issue. But then when I go on vacation, then people start messaging me and they're like, wait, no newsletter this week. I've, know, I've been missing this. read this every Sunday morning and it's, so it's nice to hear that people, you know, look forward to it.

Matthew Carey (06:49)
I love the tidbits on writing as well. I'm like actively trying to get better at writing and I think there was one about the, it's completely slipped my mind, the dictionary, but the dictionary from like 1914 or something. Yes, the Webster's Dictionary, yes. And I was reading that and I was like, is a beautiful description of a word. I want to save this as my dictionary on my Mac. So yeah, I did, it was very

Thorsten Ball (07:02)
Mmm, yeah, the web says dictionary James Somers. ⁓

Yeah.

Yeah, it's nice. There's a new James Somers. So this is I'm a big fan of his. So he he's also like me. He once was a Ruby on Rails programmer and he worked at it was once called RapGenius.com, I think. And now it's just called Genius.com, the website that has like the lyrics. And now he works at Jane Street.

Matthew Carey (07:14)
tidbits.

Wilhelm Klopp (07:14)
That's awesome.

Yeah.

Mmm.

Thorsten Ball (07:39)
but he also writes for the New Yorker magazine and writes for the Atlantic. that's a guy I'm really jealous of. he's a really good writer. So I don't know, you might have seen this. I'm sure you have actually. was a couple years back, there was this, oh, I'm sorry, I'm gonna mess this up.

Wilhelm Klopp (07:44)
What?

Matthew Carey (07:47)
That's a varied one.

Thorsten Ball (07:58)
But there was this New Yorker article about Jeff Dean at Google, and it basically says the friendship that shaped Google. And it was about Jeff Dean and his colleague whose name I cannot remember nor pronounce. And they were like really influential engineers. And this article described a lot of, you know, what they did together. And this was written by James Sommers. And he's a yeah, he's an incredible writer. And he also has a newsletter and he wrote it.

Wilhelm Klopp (08:20)
No way.

Matthew Carey (08:21)
soon.

Thorsten Ball (08:26)
really nice thing this week about, you know, how he does reporting based on what John McAfee does. And, you know, I'm going to put this in the news center, but I just wanted to give him a shot. I sent him an email once and then we got talking and blah, blah, blah. And he's like, if you're ever in New York, you know, hit me up. But I'm like, 100 percent now, I haven't been to New York ever, but, know, that would be nice.

Matthew Carey (08:37)
Amazing.

Wilhelm Klopp (08:47)
That is amazing. Yeah, I'd

never heard of this guy, that's, wow, yeah, definitely easy subscribe for me.

Matthew Carey (08:52)
Definitely gonna Google him up this.

That sounds like an amazing segue to go into maybe a little bit of your backstory. I mean, we talked just then, the intro gave some details after 2019, but I'd love to hear where you grew up, how you got started in programming, maybe some of your first jobs were.

Thorsten Ball (09:12)
Yeah, so I grew up looking outside the window. grew up basically around here. So I live back again in small town Germany and I grew up in like an even smaller town far even more in the woods. And I think I get kind of into programming as a teenager, you know, where

I like computers and then, know, in gym class in seventh grade, there was another guy who also liked computers. And that's when we stopped really participating in gym class and just starting walking around the track. And then there was one day where he's like, have you ever heard about Linux? You know, and it's like, no, no, no, tell me, tell me more. And then, you know, four weeks later, we had obviously by accident, wiped my hard drive and trying to install Red Hat, whatever it was.

Wilhelm Klopp (09:38)
Nice.

You

Thorsten Ball (09:58)
And I think that's, you know, I've been on, you know, used Windows before for gaming and like RSC channels. I was in online communities and I realized, I actually realized this a couple of weeks back, I've been writing online since probably 2001 or 2002. Like I've always had like a personal website with like a blog or back then you had like these, ⁓

What was it called? Like these, you would link to like the, is it called link chains? I don't know. You would link to your friends' websites and whatnot. And I've been doing this for long time. And then I got into programming and scripting and Python. But it was always websites. It was always this, ⁓ you know, I once wrote about this on a newsletter that, ⁓ you know, the Andy Warhol quote of,

Beautiful thing about Coca-Cola is there's only one Coca-Cola, even the richest people, they only drink Coca-Cola. There's no rich people Coca-Cola and poor people's Coca-Cola. And for me as a teenager, I had the same thing with the internet where somebody was then saying, well, you can get a URL and then you get web space and then you can have website. And I was like.

Wilhelm Klopp (10:49)
Mmm.

Mm-hmm

Thorsten Ball (11:02)
Wait a second, you're telling me I can have a website. If I type my name in the same address bar that you can type in coca-cola.com, I can type in thorstenball.com and it's that, you know, I can participate in this and I don't know. I got into, it is right? Like it's the same with software. Like you can play with the big boys. Like you can, there's no entry paths. You can build something, put it online and it can, it plays in the same league.

Wilhelm Klopp (11:17)
It's still mind-blowing in a way.

Mm-hmm.

Thorsten Ball (11:30)
as any other piece of software and it's beautiful.

Wilhelm Klopp (11:32)
Yeah,

it's so true. Although I feel like whenever my girlfriend looks something up, like we find out about a shop or whatever, she doesn't Google it, she looks on Instagram first. it's like, ⁓ which sure, you can access that through the web as well, but ⁓ it's like, yeah, everyone has an Instagram page, and now there's a bit more of a walled garden or whatever, and you look through it there. But also sometimes, I mean, this is a total derailment, but I feel like, ⁓

Thorsten Ball (11:41)
Which is, yeah, I'm too old for this. don't, yeah.

Yeah, that's true. That's true.

Wilhelm Klopp (12:01)
One of my hopes for this new brave world of AI is that if the stakes or the entry level expectations change, that everyone gets pulled up. I think if you look at the UX of something like Instagram, especially on their mobile app, it is so good and so optimized and so smooth and the transitions and everything are perfect. ⁓ Imagine if everyone, and I think that's obviously a team of hundreds of engineers

for years, like perfecting that and testing it and whatever. My hope is that like with AI, like any app that any of us build and put up on torstenball.com that is accessible to everyone can have the level of UX and smoothness as something like Instagram. I think that would be great. I think we're not there yet at all. In fact, the British Airways website barely works still. And I think that probably is an easier fix than making every app as great UX as ⁓ Instagram.

Thorsten Ball (12:31)
Yeah.

Yeah.

Yeah.

Wilhelm Klopp (12:58)
But hopefully, we'll get there.

Thorsten Ball (13:01)
Yeah.

Matthew Carey (13:01)
So many

websites don't work. I was just ordering some beers from Asda for a hackathon we're running tomorrow and the whole Asda search didn't work. Like nothing. Asda's like a massive supermarket. I'm pretty sure it's owned by Walmart. And like the whole search didn't work. Like you search for anything and there was just nothing, complete blank. The only thing that worked was the mini search before the search. So if you searched for something and the little drop down came down, that worked. So that was the only way I could navigate to a product. Anyway, we're massive. This is a massive detour.

Wilhelm Klopp (13:12)
Mm-hmm.

yeah, yeah.

Thorsten Ball (13:24)
Hmm.

Wilhelm Klopp (13:27)
the preview.

Matthew Carey (13:28)
⁓ I wish people fix actual things.

Wilhelm Klopp (13:29)
But we do also love detours on this book. I actually wanted to ask you about a German thing, Thorsten. ⁓

Thorsten Ball (13:33)
You're saying I

love teachers and it's like here's the next one

Wilhelm Klopp (13:38)
Here's the next one.

Here's the next one. ⁓ So I almost made a tweet about this ⁓ because it seems like I'm getting a lot of really, really great AI-generated content sent to me in German from German friends. It's like AI-generated videos. ⁓ Have you? ⁓ Like, they are hilarious.

Thorsten Ball (13:41)
Go for it.

Really? Really? What do mean?

What do you mean? Like, you mean shorts

or like personal videos or what?

Wilhelm Klopp (14:07)
Yeah, no,

no, shorts. Yeah, like an Instagram reel or like a TikTok style form, like a short form video. ⁓

Thorsten Ball (14:15)
But what is it? Like realistic looking

people? What is it?

Wilhelm Klopp (14:20)
So I think one example was...

There was some big like G7 conference recently, right? And someone took that like photo of all of the like world leaders standing next to each other and they're doing some kind of folk dance. So they took that, made it into a video and now they're all like folk dancing together and it's over some like strange German voiceover. But it's hilarious. Like it is actually so good. And like I think there's been truly like some kind of unlock on the German short form media creator side.

Thorsten Ball (14:32)
Yeah, yeah.

you

I

I know what you mean. I don't think it was German, but I saw a video where they took Trump at the summit and they did... Didn't they change his voice to talk about everybody else? They basically invented the scene as how it could have played out and whatnot. I didn't know about the German AI.

Wilhelm Klopp (15:06)
Mmm.

Yeah, this kind of stuff.

It's just surprisingly funny to me. ⁓ another one is, you know this format where people go...

take a microphone, interview people in the streets, and make them do puzzles or explain things or whatever. And a lot of people are really bad at it. Or you edit the clips to make it look like everyone in a certain town is absolutely stupid. It's very selective editing. There's a new AI format where...

Thorsten Ball (15:35)
Yeah.

Wilhelm Klopp (15:40)
Someone's being interviewed who looks like they wouldn't know anything at all, but they give you this highly technical definition of like what reality is in this like uber philosophical sense, but it's this like construction worker who's like barely wearing anything. So like it's just really, really, really good. ⁓ And then randomly yesterday I

Thorsten Ball (15:53)
Yeah, yeah.

Ha

Wilhelm Klopp (16:01)
or served some YouTube video where they're interviewing like a big German newspaper mogul guy and they're making him watch clips from... So, okay, this is a specific term. Have you heard of Italian Brain Rot? Okay, Matt, have you heard of Italian Brain Rot? Okay.

Thorsten Ball (16:15)
No. No.

Matthew Carey (16:18)
No, I've seen like,

is this like equivalent to like French Snapchat?

Wilhelm Klopp (16:22)
And maybe what's trash? Snapchat?

Thorsten Ball (16:23)
What?

I feel too old now guys.

Wilhelm Klopp (16:27)
I need a... I need a... Okay, Thorsten, I'll send you this video. It's in German for sure, but like... But there's these categories of Italian or... Also German brain rot, which crucially are not meant for the local audience. They're meant for a global audience. But basically it's some nonsensical AI-generated video, like a cactus wearing shoes, and then it's some like Italian words spoken alongside it that...

Matthew Carey (16:27)
Ha ha!

Wilhelm Klopp (16:50)
sound beautiful because the Italian language sounds beautiful like they're talking about like capaccio and saying some swear words or whatever and it just sounds really pleasant but it

Thorsten Ball (16:51)
Yeah.

my god.

And people watch this?

Wilhelm Klopp (17:01)
People watch it and it's especially popular not in the region it's from, right? And then there's also German brain rot, which is just like long German words to like a vertically rotating fish. Anyway, we should probably end this detour, but it's just like a hilarious current cultural phenomenon.

Thorsten Ball (17:04)
my god.

Yeah. ⁓

Matthew Carey (17:25)
so weird. French Snapchat is completely different. It's like ⁓ very raw, ⁓ like very real, except it feels fake. ⁓ So it's just crazy things that are happening in France. The one that I'm thinking of is a guy who makes friends with a pigeon and the pigeon lands on his windowsill like every day. And then he's like rapping in like Marseille slang to the pigeon and the pigeon's like dancing to him.

and he's like giving the pigeon like a hat made of like tinfoil. I don't know. Anyway, this is a ridiculous subculture.

Wilhelm Klopp (17:57)
incredible.

Thorsten Ball (17:59)
How, but wait, but Snapchat, this is how out of the loop I am. Snapchat is still just a messaging app or do people...

Matthew Carey (18:05)
Nah, they're like,

Snapchat had stories before Instagram did, and then I think these ended up on Snapchat stories, and then they got circulated. But in France, Snapchat was like a massive deal, which I don't think it was, I don't think it carried on being as much in the UK.

Thorsten Ball (18:10)
Oh, there you go. Yeah. Then they get posted. OK, OK.

Wilhelm Klopp (18:21)
Hmm, interesting. I think it's... Yeah, go on.

Matthew Carey (18:23)
Anyway, can we get back to Thorsten's Ruby on Rails? Because we didn't even quite get there.

Thorsten Ball (18:24)
Yeah.

Where were we? yeah, okay, where were we? Okay,

teenager learning program. think, ⁓ yeah, websites. But then crucially, I don't know, I don't know how was for you Will, but I think I wasn't really good at math in school and people, there was like this, if you wanna be...

Wilhelm Klopp (18:42)
same.

Thorsten Ball (18:45)
you know, work as a programmer, you got to study computer science, which is informatics in German. And if you want to do this, you've got to be good at math. And I kind of checked this off in my head. And at the same time, this was when I was 15, 16 or something, I wanted to be a journalist, you know, like that was my, I wanted to be a writer and, ⁓ and also then later on a musician. I kind of

I've always and have, you I've never stopped being a nerd who, you know, when he had his girlfriend over was like, well, you want to watch a movie? How about, you know, the stuff that I download here is on my Linux Ubuntu machine actually, you know, like awkward moments where you're like, let me find this via SSH on this server or something. ⁓ But I kind of stopped programming and then, you know, try to become a musician, move to Berlin.

Wilhelm Klopp (19:21)
That's awesome.

Thorsten Ball (19:33)
went for a year to Australia to do like work and travel, moved to Berlin, played guitar, tried to make it in music, studied philosophy and then, yeah. And funnily enough, so I've always, this is gonna sound even nerdier than anything else, but I.

read philosophy books even before I studied it as a teenager. And I think me and my friend, were always trying to one up each other. Where it's like, have you read this guy? I'm reading Marx and it's like, ⁓ you know. But so I got into this, but then while doing that.

Wilhelm Klopp (20:06)
Hahaha.

Thorsten Ball (20:10)
I think one thing I realized is that I didn't like studying in Berlin. It's too big of a city. You would never bump into anyone that you were studying with because you were so strewn about. I didn't like the bureaucracy of it. Like I didn't like. I could get points from basically nearly anything, but you have to then ask this person to give you a sign up so you can go to this seminar and then you have to do this and this and this. And I really didn't like that. And I was studying.

Wilhelm Klopp (20:16)
Mm-hmm. Mm. Yeah, I know. Mm-hmm.

Thorsten Ball (20:38)
not philosophy, but it was called culture and technology with like a focus on philosophy. And so lot of the stuff that I read back then keeps popping up again. And I'll get to this in a second. Talking about LLMs and everybody building websites. ⁓

And you know, then I was sitting there and I actually was in like a in like a local club here from my hometown and they asked me like, can you build us a new homepage? You know, call me in Berlin. Can you fix up the old homepage? And I was like, yeah. And I kind of got back into it and I realized, oh, there's HTML five now there's Ruby on Rails. And one thing that I realized. And I keep bringing this up still in conversations today with other people is that I've realized that

I have no problem sitting in front of the computer for eight, 10 hours every day, trying to get something to work. Even if it's not working, you know, but I like this. I like sitting in front of the computer. like reading, trying to figure stuff out, understanding how it works. It's like an endless thing where you can always learn more and learning more is always what has driven me. And then I compared it to playing guitar and practicing guitar, which

Wilhelm Klopp (21:44)
Mm-hmm.

Thorsten Ball (21:49)
I never enjoyed, I like playing guitar, but I could never practice for eight hours a day. And then you start to read about, know, Eddie Van Halen, he would sit at home and practice guitar instead of going to a party or something. And I realized that, you know, it sounds dumb when you say it out loud, but I realized for some people, some things...

that are really hard for others are maybe not as hard or they can tolerate it for longer. You know how some people talking about bikes and whatnot, some people have this thing for endurance sports where they have, it's not like they feel less pain when they run a marathon, but they kind of wire different in the sense that they can tolerate the pain for longer or this specific kind of pain. And Jerry Seinfeld has a good quote about this where,

Wilhelm Klopp (22:34)
Yep, yep, yep, yep.

Thorsten Ball (22:38)
And he's like, yeah, being a comedian is hard and bombing is hard, you know, in front of an audience and not having a set that works. But you have to find your pain, like your the pain that you're comfortable with, you know, like you have to find one thing that you can do even though it sucks, but you can still do. And even though you don't enjoy it, but you can still do it. And I realized that, you know, computers is this for me. So I hit up an old friend of mine who I knew from the Internet over the Internet.

Wilhelm Klopp (23:01)
That's great.

Thorsten Ball (23:04)
And yes, side tangent note, I have in the back of my head, I should write a blog post about that my life as it is today would be 90 % different if it wasn't for the internet. Like it's so much in my life has been influenced by the internet. So I reached out to my friend and I was like, hey, I'm getting back into programming. And he knew me for my IRC days and like PHP program, whatnot. And he's like, yeah, you can, you can switch. Like the only thing that counts is

Wilhelm Klopp (23:15)
Mm-hmm.

Thorsten Ball (23:30)
you don't have to have a degree, which was news to me. And he's like, the only thing that counts is you have to be able to show that you can do it as well. Like to kind of convey to the other person, you can do this job. And that, you know, lit a fire under me where I was like, okay, there's nobody like I don't have to do like arbitrary tests and get grades and jump through hoops. I just have to be good at this.

Wilhelm Klopp (23:33)
Mm.

Thorsten Ball (23:57)
And then I realized it's a lot of work. It sucks at certain points, but I can I like reading books. I like sitting in front of the computer. I like getting better at this stuff. So at that point I was, you know, bought all of the books every morning. And, you know, back then I was working at UPS night shift and morning shift, unloading trucks and studying philosophy. And so in my mornings or evenings.

I would sit in front of my computer and do this like every day, some Ruby on Rails. And then he hired me as an intern and I kept doing this on the side. And I actually held up this practice for the next 10 years that every morning I would spend like an hour at least to work on my stuff, know, like side projects, learning a new language or whatever it was. And then he hired me as a full-time like a year later or half a year later.

Wilhelm Klopp (24:25)
That's awesome.

Thorsten Ball (24:45)
hired me as a full-time program. We did like some, ⁓ I don't wanna say classic, but like classic German consulting where you're like the freelancer, like he was the senior and I was the junior or intern. And we would do like contract work for like a big company like Audi or something and build websites, build Ruby on Rails, JavaScript. And then we started to work for a startup, ⁓ Flink it was called, which did ride sharing. So basically before,

Wilhelm Klopp (25:12)
Hmm.

Thorsten Ball (25:13)
I don't want to say before Uber, but before Uber was as big as it then became, this was like an app where the idea was, you you've seen the meme by now where, these are all the cars on the street, and this is what all the cars would be if we wouldn't have three empty seats in every car, you know? Like, that was the thing. Like, hey, if you're going the same way that I am, why can't I pick you up? How about I put this right in? This was the big, you know,

Wilhelm Klopp (25:18)
Sure, yeah.

Yeah, yeah, yeah.

Thorsten Ball (25:38)
2011, 12, 13 with like, yeah, apps, like we can turn apps into everything into an app. And this was exciting. And so I started, you know, we started contracting for that startup and it turns out that startup is located near where I grew up. And I was still living in Berlin, but in the back of my head, I always had this feeling of, you know, this doesn't feel like home to me.

Wilhelm Klopp (25:40)
I remember this. Yeah.

Thorsten Ball (26:03)
like Berlin, there was always a feeling I gotta move away at some point. And the other thing was, there was like this little thing of like, wouldn't it be mad if I could work for really good company remotely while working from small town Germany? Back then, GitHub was like the, that was the number one company where I thought this would be the dream job, like to work at GitHub. ⁓ And yeah, funnily enough,

Wilhelm Klopp (26:20)
Mm-hmm.

Thorsten Ball (26:29)
Super little side note, I met my wife in Australia while I was backpacking there, but she's also from around here. So by chance, we met and we were like, what's that accent? Like, why are you? And I was like, where are you from? And this woman, she was like, I'm from so and so. And I'm like, no way. That's like a 10 minute drive from where I grew up. And she's like, I'm going to get, let me get my friend. And her friend is now my wife. And we

Wilhelm Klopp (26:35)
No way.

Thorsten Ball (26:57)
we got to talk and then she's like, what are you going to do after Australia? I'm like, I'm going to move to Berlin, become a musician. She's like, oh, I'm also going to move to Berlin. And I was like, what? And then we remat in Berlin, became couple. And then we were like, yeah, at some point we want to move back. then move back, work for startup, then that startup, worked there for three years, kind of acqui hired by the Deutsche Bahn, the German railway company. On the side, I wrote the books that you mentioned, Matt.

Wilhelm Klopp (27:20)
Nice.

Thorsten Ball (27:23)
always in my mornings, right on the train on commuting to work, I would write these books or on the on the way back and kind of became more interested in languages, know, systems programming ever deeper in the stack. And if you look at my first two or three jobs, it was always like I had a job and on the side, I was kind of preparing to get the next job, not specifically like this, but like in hindsight, it looks like this. So

In 2019, that's when I then joined Sourcegraph because you know, they do language server stuff. They do search. They do like all the cool compiler stuff and remote company. Yeah. And then worked at Sourcegraph and then after four and a half years, I was like, I gotta try something else. want to, I've been hacking on my compiler and Rust for three years. I was like, I want to do Rust, try this. Then I went to Zed and then realized after a year, yeah, this is not, I'm not, you know, for whatever reason, super happy here, feeling fulfilled and

Then looked around and then talked to a bunch of different companies and then also talked to Quinn again from Sourcegraph CEO and realized he sees things the same way that I do. And there was this feeling of like, yeah, what the hell? Like I don't want to go to big tech and I don't want to go to a super, super small startup again. And he gets me, I get him. Would be nice to work together again. So went back and yeah, that's it.

Wilhelm Klopp (28:44)
That's awesome. There's so, so, much in there I had no idea about. Really, really, really cool.

Thorsten Ball (28:50)
It's, I don't know. I don't know. It feels awkward to talk.

Wilhelm Klopp (28:54)
Alright, well, ⁓ let's go on some more tangents then. heard, ⁓ so something I've been meaning to ask you is, I think I've heard a few times someone saying, without you in the room, ⁓ Thorsten hates emojis. ⁓

Thorsten Ball (29:12)
yeah.

Wilhelm Klopp (29:13)
And I don't think there is on the internet available today a ⁓ kind of like first principles, Thorsten's perspective argument for ⁓ this, not for her, for this perspective. Because I can for sure see the like, you have some AI agent that's like vomiting emojis at you. Like nobody wants that obviously. But it seems like there's more here than just wanting to prevent that issue.

Thorsten Ball (29:23)
emojis.

Yeah.

I think it's not that I don't like emojis. ⁓ So it started as, well, I don't like it. I like emojis. I think they're funny. And I think I'm also not, you know, the old man screaming at clouds. I like that they add this funny stuff. Like there's so many good jokes you can make with just emojis. And it's, the...

There's like emoji reactions in Slack that have made me laugh so hard. It's really good. I love emojis, but I hate emojis in UIs. When you put them, when there's like a button that says submit and it has like some rocket on it or, know, when you ask any LLM, generate me some markdown or like a summary and it's like, how to install it rocket ship.

Wilhelm Klopp (30:08)
Yeah.

Mmm.

Thorsten Ball (30:31)
Set up and then it's like the little two gears or whatnot and it's like come on dude. That's like that to me is just like cliches and So I don't like them in UI and I don't like them in terminal UIs. I think it's just a cheap It's it's cheap. That's that's why I don't like it. You can do a lot more with bold, know, like just it's like writing where people put like Memes into their blog post instead of having like witty writing

Wilhelm Klopp (30:47)
Totally, yeah, yeah, yeah.

Yeah, yeah, yeah, yeah.

Mm-hmm.

Thorsten Ball (30:58)
And that's not

just, I love memes as much as anybody, but I think, you know, it can be cheap.

Wilhelm Klopp (31:04)
Totally, yeah, no, that makes sense. So you use emojis day to day. You would put an emoji in a message you send someone. Okay, okay, okay, Yeah, the...

Thorsten Ball (31:08)
Yeah, yeah, yeah, I would, yeah. I even use

the ⁓ slightly smiling emoji, you know, like the plain old slightly smiling, yeah, yeah.

Wilhelm Klopp (31:17)
⁓ god. ⁓ You're...

Matthew Carey (31:20)
That's

actually my, that's actually my, the thing I hate about Slack is, you know, it auto corrects like the colon bracket to slightly smiling. That is the worst auto correct ever. Like it's actually horrific.

Wilhelm Klopp (31:20)
That's violent.

Thorsten Ball (31:28)
Yeah. Yeah.

Wilhelm Klopp (31:32)
Yeah, that's pretty bad.

Thorsten Ball (31:33)
Yeah, I sometimes explicitly

undo it. think there's an art to it, right? So you can type colon closing parentheses. That's like the classic old school smiling. That means more and is realer than the slightly smiling emoji.

Matthew Carey (31:40)
Hmm.

Wilhelm Klopp (31:44)
Yes.

Matthew Carey (31:44)
Definitely.

Sometimes I do double parentheses, and then it doesn't autocorrect.

Wilhelm Klopp (31:48)
to 100%.

Thorsten Ball (31:52)
I like the doubles one, yeah. That's like it. Yeah, I think you can. Yeah, I think you can.

Wilhelm Klopp (31:52)
Cool, yeah. You can also turn this off, pretty sure, in the Slack settings. ⁓

Matthew Carey (31:56)
I have no idea how to turn this off. I can't use Slack at

the moment. I just get bombed with AI ads on Slack. Upgrade to the business plan with AI extra-ness. I can't use Slack at

Wilhelm Klopp (32:01)
That is also a thing. Yeah, that's quite

a lot. Are you familiar with the like, tail of how Germans use the like, what's that emoji called where you like close one eye? The like, what's that called? Wink face, yeah, yeah.

Thorsten Ball (32:18)
wink, wink emoji? Yeah.

Wilhelm Klopp (32:21)
There was a beautiful...

Thorsten Ball (32:21)
No.

Wilhelm Klopp (32:22)
You're not from here? okay, this is great. So this is a... I'll dig up the blog post for this as well. ⁓ But someone wrote about this. Someone not from Germany, moving to Germany. But the long and short of it is that in Germany, this is a really popular emoji to use, like Wink Face. Like really popular. Like you use it all the time. Like, I'll see you later. ⁓ Wink Face. And... And... ⁓

Thorsten Ball (32:37)
Yeah, yeah. Yeah, yeah.

Is it?

Matthew Carey (32:46)
That's so sus.

Wilhelm Klopp (32:51)
and someone was trying

Matthew Carey (32:54)
Yes, sus.

Wilhelm Klopp (32:54)
to figure out, like, I think this person had also moved to Berlin or something and was in like a chat with their elderly female landlord. there was a... ⁓

Thorsten Ball (32:57)
You

That's a good album name,

Wilhelm Klopp (33:09)
And she would just constantly use these wink face emojis. And I think one time she sent a message like, there's also a spare bedroom in case you have any guests over, wink face. anyway, then the author ended up digging into it. And it's just like kind of a German idiosyncratic thing. Like there's no ⁓ sexual connotation. It's just like a...

Thorsten Ball (33:11)
You

We didn't get air.

Yeah.

Wilhelm Klopp (33:33)
It's kind of like a subtly helpful or like like we both know something about this or

Thorsten Ball (33:38)
It's

like, I want to be friendly, this is not that serious. I think that's what this is. It's to lighten up things. I don't know. Maybe that says a lot about Germans and humor and whatnot.

Wilhelm Klopp (33:43)
Right. Yes, yes, yes, yes, yes.

Right, exactly.

So this is called ⁓ the Wink Face emoji. Are Germans flirting with you or not from 2018?

Thorsten Ball (33:58)
Also, I love, this is a whole nother meta layer. I love that, how interesting it is that different operating systems have different emojis. So you know the salute emoji that Apple has where you see like half the face? It's different on any other platform where you see the full face, right? So the salute guy on like Apple, it's like fading out like saying, you know.

Wilhelm Klopp (34:07)
Mm-hmm.

Mm-hmm

Yeah.

Thorsten Ball (34:24)
you go on without me or whatever. Good luck. And then you see it on a different platform. Looks completely dumb. And so you have like this thing now, this translation layer where some people know I'm talking to somebody in Android. I cannot use this emoji because it's going to look dumb and whatnot. It's interesting.

Wilhelm Klopp (34:26)
Mm-hmm.

you

Mm-hmm.

That's, yeah. I ⁓ feel like they've converged a little bit over time, but there was some awful stuff, I think. Like, I think there was a time when the gun emoji, Apple switched to, like, a water pistol, but everyone else still had the gun. So it's just, it can get brutal out there. Be careful with your emojis, folks. ⁓

Thorsten Ball (34:54)
yeah, the water press, yeah.

Yeah, yeah. Yeah, yeah.

Wilhelm Klopp (35:06)
Okay, here's another thing I was quite keen to ask you. ⁓ I think one of the things we've talked about often on this podcast, and I think you've talked about this often in your writing, is how we're in this ⁓ beautiful new world of AI agents, but you really have to invest in learning this stuff well.

Thorsten Ball (35:25)
Yeah.

Wilhelm Klopp (35:25)
Like, you really have to put it, it's like a new skill. Like, prompting while using something like Amp while using something like Cloud Code while you, it's not just drop in whatever, like people can have different skill levels at this and you can get better at this. ⁓ And it reminds me of like an age old debate.

⁓ or an age-old belief that like in tech we just like don't really have this concept much of like training like you didn't really go away for a week to do your training like you're kind of expected to like learn on the job which is very very different from a lot of other industries I think a lot of other companies ⁓ and for me it's particularly salient because the first ever like tech adjacent job I had was like a little mini internship at Bloomberg and all the interns are given this like ⁓

Thorsten Ball (35:57)
Hmm.

Wilhelm Klopp (36:14)
⁓ biography about Michael Bloomberg like an autobiography which by the way He published I think when he was like in his late 50s or whatever So he's lived a whole life since then but I was like a very keen intern so I read the whole thing and He one thing he keeps going on about is how everything in blue or like a lot of things in Bloomberg are really set up to invest in training So any employee can get training even on things that are unrelated Enables a lot of great internal mobility obviously Bloomberg's huge tens of thousands of employees, but you go visit the

office

and they have all these training rooms. They have full-time people on staff that are just there for training and they teach everything from like finance stuff to like programming stuff to like internal systems. So they always seemed like they really really had this figured out but it always shocked me that no other ⁓ like no other company I'd ever like worked at or seen that there's no like massive internal training team. There's no courses you go on or whatever. So I'm curious. Well, I guess that's that that old debate has been happening forever. But especially now it feels like we're really at a time

Thorsten Ball (36:53)
there.

Wilhelm Klopp (37:14)
where maybe you should invest like lesson tools and just use the tools you have already and just figure out how to do like internal training. Curious what you think about this or like how you do this internally like how do you teach each other how to use the stuff.

Thorsten Ball (37:27)
⁓ I think, yeah,

I think...

So I do think there is training. Like if you sell to large enterprises, you will encounter this where they, ⁓ know, ⁓ William Kennedy, like I'm sure you know him, he's been in the go scene and ⁓ like he trains people. He goes and does workshops for them and he gets hired for large enterprises to teach people how to do test from development and go or something. that exists, I do think.

Wilhelm Klopp (37:50)
Mm-hmm.

Thorsten Ball (37:58)
This is also tangent, sorry, but you were saying, you know, in other industries, it's different. They have more training. And I would argue that that's because the software industry is a little bit special in the sense that basically anybody can practice at home for free with endless resources. And if you go in any industry, say banking,

nobody can practice banking at home. So they all like, if you hire a bunch of interns, they're all on the same level, they get out of school, but then you pick the ones that are from better schools. So you select based on this, and then you train them up. But in software, you have is like this asymmetry of the market where, say you wanna hire 10 engineers and you're in London or in San Francisco, you're looking at the candidates.

Do you hire the candidate? You will have candidates that don't need any training. Like you will find people that have already done what you're hiring for at home for free for the last five years. So that skews the market because now you can select people that don't need training. And obviously like there's unfair advantages and because it leaves people behind that don't have these advantages at home. But I think that kinda explains what's going on. You can hire a 15 year old kid off the street

Wilhelm Klopp (38:52)
Right. Yep.

Yep, yep, yep. Totally.

Thorsten Ball (39:16)
that's better at programming than say a 30 year old that you send to a workshop for half a year, you know, like that messes with the market. ⁓

Wilhelm Klopp (39:22)
100%. Yeah, yeah, yeah, yeah.

And you have this

wild skill differences as well, right? I think we learned this a little bit from like the bootcamp era, right? Even if you do like hardcore Ruby on Rails for three months, it's just hard to compete with someone who's loved to tinker since they were like 14 and like ran into all these failure cases, deleted their hard drive while trying to set up like Red Hat or like what, like all the stuff, right? You learn a lot of stuff just by like messing around as a kid. That's like hard to teach in like three months or whatever. So yeah, that makes sense.

Thorsten Ball (39:27)
Yeah.

Yeah.

Yeah.

Yeah, exactly.

And not

just the actual skills, I think what you also learn is, and I thought about it this morning, I wanted to send like a snarky tweet, but then didn't do it by like, you know, I think there's a lot of, so this is the snarky tweet. The snarky tweet is that I think there's a lot of software engineers who don't care much about software, don't know much about software. Like they can program, but...

they don't have like this thing that you really want in, for example, a product manager. They know all of the patterns. They know why does this button, this button doesn't look right because usually the pattern is this, I've seen this in this other thing. I've used this before, I've used this. we're shifting away from, ⁓ you know, blah, blah, blah, login screens to this other thing. I've seen this here. And there's a lot of engineers who don't care. Like they don't use software that much. They can write software, they're programmers, but they don't have this thing.

Wilhelm Klopp (40:39)
Mm.

Thorsten Ball (40:44)
And that's also a little bit interesting about software that, you know, it's kind of like, okay, maybe this analogy I won't hold up, but, you know, imagine a Lewis Hamilton or somebody, these race car drivers, they know a lot about cars. Like they really care about cars and know about them and the race and the history of the sport and whatnot. And then sure, yes, you can train up somebody else who wants to be a race driver.

But how do you distill everything else into them? Good training will do this, but yeah, I'm rambling, but a bunch of thoughts.

Wilhelm Klopp (41:15)
Mm-hmm.

Matthew Carey (41:21)
That's cool.

Wilhelm Klopp (41:21)
Yeah,

no, that, that, that, sorry, go on Matt.

Matthew Carey (41:23)
No, no, no, I going say, I'm not sure it's a mega snarky tweet that many software engineers don't care about software. Because I think we all see that Yeah.

Thorsten Ball (41:37)
Yeah, I mean,

I guess it depends on what your expectation is, right? The other snarky tweet I did two weeks ago was ⁓ some people, software engineers, they don't understand the difference between a program, software, a product and a business. They think it's the same and it's not.

Wilhelm Klopp (41:48)
Hahaha.

Thorsten Ball (42:03)
Like you can have a really good program, you can have really good software, doesn't make it a good product. And you can have a really good product and an amazing business without having good software. And the job of software engineers in the business is to use their skills to serve the business by building software, but it's not the job to build software. To build the best possible software, it's to help the business.

Wilhelm Klopp (42:14)
Yeah, yeah, yeah, for sure.

Yes.

Totally. Yeah, yeah. 100%. Yep.

Matthew Carey (42:30)
Do you think that changes?

Wilhelm Klopp (42:31)
I think I know what this is

about. ⁓

Thorsten Ball (42:34)
No, no, no, no, no.

Matthew Carey (42:34)
No, no, No, this is good. This

is good. But do you think that this changes ⁓ more as like AI gets better and we have to, do you think we have to fulfill more of a stack? Like more of that breadth? Yeah.

Thorsten Ball (42:42)
Yeah. Yes.

Yeah, yeah, yeah,

think 100%. I think, I don't know where this is gonna go. I don't think anybody knows, but I was asked when we met the last time in SF, I think well, where not no, sorry, before that. ⁓ Somebody was saying like, well, what do you think? This is gonna evolve with like this triangle of designer, PM and engineer. And I didn't want to say

This is all gonna go away. But now looking back, I think this is gonna morph a lot because I think, you know, what I also said, I said this public on our podcast that some programmers, it turns out, they just like typing characters into text editors and they like building programs, one character at the time. And I think that has become less valuable.

And I think that means you have to complement this skill with knowing what the program should do. And the other thing that we say in the Amp team is that, know, prototypes and experiments over discussions or documents. And that just, it's a little bit flippant, but I think it just shows that, well, a lot of the stuff...

Has now changed and dynamics have changed you can produce a lot maybe cheaper or dumber or more broken stuff But the balance has shifted and you can produce more and it's more important to know what you should build and it's the same with You know I can I can whip up a website, you know with Instagram clone, right? I did this as a this was one of the first and demos that I gave a large company I built like an Instagram clone and it nailed it like 100 percent

Wilhelm Klopp (44:22)
Okay.

Thorsten Ball (44:23)
Yes, I'll do this. Here's the pictures and you know, like the web version. But then you have to apply two things. You have to apply, well, is it, did it do the good job of the software, like of the engineering side? Like, is this good, maintainable, scalable, whatever code? Does it fulfill its purpose now and in the future? Or on the other side, is this a good product? Like, does this make sense? You know, I just hit the button and the thing did it in five minutes.

Wilhelm Klopp (44:27)
Mm-hmm.

Thorsten Ball (44:50)
You know, like, what's next? Like, is this good? Is the button right? can I judge whether the output of the thing is good? And, you know, that's now part of your skill set, I think, or required skill set.

Wilhelm Klopp (44:54)
Yeah.

Matthew Carey (44:57)
Yeah, yeah.

Yeah, like.

Thorsten Ball (45:02)
How

do we start down this tangent? Is snarky to me?

Matthew Carey (45:05)
Yeah, something about moving between business... Yeah, okay. Training, training.

Wilhelm Klopp (45:06)
Haha

Thorsten Ball (45:07)
training.

Wilhelm Klopp (45:09)
Training, yeah. All right, let me

ask one last thing on the training thing before we can move on from that. So I think everything you've said about this makes sense. What you've just said now makes a lot of sense. So there is a big skill shift happening. Or there is a big, there's a new domain that is unlocked. As an engineer, maybe you now need to think more about whether what you're building is actually useful or have more ownership on the product side. It feels like... ⁓

Thorsten Ball (45:14)
Yeah.

Wilhelm Klopp (45:36)
there is more need for training because of this. But I don't think we're really seeing it, right? We're seeing a lot of new tools built. maybe that's, I don't know, think if you work at a bigger tech company, I think there's a lot of questions and budget available for ⁓ new AI exploration and tokens and new tools. But I think what will be way more impactful is a week-long course, everyone hears how you really structure a rules file.

whatever, right? Or like, or here is how to think about ⁓ prompting, like you want to plan first or, know, the oldest. And again, maybe this is just like, I've just given two very straightforward examples in a very complex domain that is actually like, there are no trainers yet and the rules aren't written yet. But like, I don't know, like.

Thorsten Ball (46:11)
video.

Wilhelm Klopp (46:23)
Maybe a specific question would be like, if you had to make a course right now, like next month is just you making a course or not a course or some training program, like a one day workshop. Do you think that training, if everyone consumed it, would move things forward or do you think it's just not, is something wrong in my mental model here?

Thorsten Ball (46:29)
Mm-hmm.

no, no, I agree. think, so I wrote about this on the Ampcode.com website, how I use Amp. This was a couple of months back. because people were saying like, I was like, this is just obvious stuff to me. And people are like, this is good. We need more of this. But I noticed that I'm holding back with a bunch of this stuff because I know that I haven't figured it out yet. And this stuff is changing.

It's really, know, we keep saying this, everything is changing, but it's this, how do you stand in front of a room and train people and say, well, this has worked for me. And yet if you adopt these practices, it might only work 70 % of the time, you know, that's the reality of it. And so then you start.

Wilhelm Klopp (47:24)
Sure.

Matthew Carey (47:26)
Yeah.

Thorsten Ball (47:29)
down this rabbit hole where you're like, okay, you have to understand how these tools work, how these models work and blah, blah, blah. And that's a big challenge. And I think the other big challenge that we're seeing is that, okay, there's two. I think the first one is with the whole consumer side of AI, let's say JAT, GPT and whatnot, and the talk about AGI, I think the expectations have not just expectations, but the framing

is in such a way that even engineers anthropomorphize AI and they think of it as like these things that you talk to, which it's gonna sound dumb because that's what you're doing, but you know, they're like, ⁓ this thing then helps me do this. And I think for software engineers, the much more productive and healthier way to look at things would be to think of them as pipes where you throw text in and it kind of does stuff and text comes out and the rest is just UI on top.

but then you have this expectation setting, it can now one-shot apps. And then you have this combination of, they think it's like an all-knowing entity and people saying it will replace programmers. So then they sit down and they go, build me a feature that lets me blah. And you're like, when in reality what I do most of the time is, I call it paint by numbers programming is, I think about how would I build this?

Wilhelm Klopp (48:24)
Mm-hmm.

Thorsten Ball (48:49)
what should it look like? What are the trade-offs? What are the trap doors? What are the details, you know, if necessary? And then I use basically the agent as, you know, like a 20th century scientist assistant where you're like, and now type this up, you know, like go and run the experiment and come back to me with the results. But I know, I know what the experiment is and what the ingredients should be and not just figure out how to make gold out of this. You know, it's instead, let's try this. Let's take this.

Wilhelm Klopp (48:55)
Mm-hmm.

That's cool. Yeah.

Mm-hmm.

Thorsten Ball (49:18)
and put this coconut in there and let's shake this and then you can come back to me and tell me whether it's gold or not. so the other challenge that I'm now seeing is, and I did tweet about it and wasn't snarky, is I mean, I would say I was mad. I think when we first had a call, this was in February or March or something. And I would say back then, if you look at the curve of agentic in programming, I think this was at the bottom of the curve still. I would say that's pretty early and

Matthew Carey (49:43)
Yeah, yeah, I agree.

Thorsten Ball (49:44)
So back then I was like, I've never done Coke, but I imagine this is what it must feel like where you're like, the agent can do anything and just, you know, the agent does this and try again, try again, and it works. this worked and it did this. And you have a new code base. And then you have the hangover where you're like, ugh, I cannot now not debug this and it doesn't work. And I don't understand the code. And then you kind of adjust. So now I feel like I'm like one or two months ahead of the like phase shifted.

Wilhelm Klopp (49:49)
You

You

Mm-hmm.

Thorsten Ball (50:11)
And so in the last two or three months, people were like, you can do this. And I have like eight agents running and blah, blah. I'm like, what am I missing? Like, I don't think this is going to work. And now in the last couple of weeks, people have been saying, well, I went back to do this. So now I'm thinking this is, it's like in all of these communities, you have like this collective experience that people have to go through where, we all tried microservices and now we all have a head like this and now let's move on. And now with the Genti coding.

Wilhelm Klopp (50:16)
Right, yeah, yeah.

Matthew Carey (50:17)
Yeah

Wilhelm Klopp (50:34)
Yeah, yeah, yeah.

Thorsten Ball (50:39)
We all went through this and now let's figure out what the actual productive way to use these tools are.

Matthew Carey (50:45)
So I'd love to jump in. yeah, so our best performing episode is about structuring code bases with AI. And you briefly just touched on that just then. But how do you feel about this? How do you feel about structuring code bases to use Amp on the most helpfully? How do you feel about those trade-offs about how you write that up and express that to an agent?

Wilhelm Klopp (50:45)
Wait, yeah, yeah, yeah, we've got one. This is so good.

Thorsten Ball (50:54)
Yeah.

Yeah.

Good question. I don't have a solution yet. A bunch of people asked me this. I don't have a solution. I have a couple of thoughts. One is, and I want to write this up. think if nothing else, these agents are now, and I think Will, I you about this. think these agents, they're buttons that you can press and outcome 300 lines of working JavaScript.

Wilhelm Klopp (51:27)
Yeah.

Thorsten Ball (51:36)
Python or Ruby code, like as a standalone program, right? They might not be able to go into Chrome code base and say, we have a branch prediction miss in the JIT compiler, can you figure out what the issue is? We might not be there yet, but can you build like a small script in 300 lines of Python that can analyze JSON and build your dashboard for it? 100%, like it can do this really well. So now you have a button that you compress and in 10 seconds, 500 lines of code come out.

So now my question is, as an engineer, how are you gonna use that button? There has to be a way to use this button because if, can you structure your code base in a way so you can leverage this? Can you structure your code base in a way that lets you say,

You know, I don't have to know about the rest of the code base. I only have like small parts. And this is maybe slightly different than decoupled or microservices or modular, whatever you want to call it. But can you build your code base in a way that, you know, you can use like a test script that the agent can build to give you like a feedback loop about something else. Can you build your code base in a way that, you know, this is what everybody should do. Everything in the code base should be easily

exercisable and be able so you can put it in a feedback loop. So it's not like the agent goes off and then you have to go, now I have to click four times and it is gonna take me 30 seconds. If it takes you four clicks to get to this point in application, you might wanna build it so that the agent can access this part of the code base on its own. I, like there's a bunch of these thoughts, but it's, I keep coming back to dumb code is cheap. How are you gonna use it? And one,

Wilhelm Klopp (53:13)
I really like that.

Thorsten Ball (53:19)
specific example that I used in production and how it changed how I approach some things. We had a bug in Amp where sometimes the edits were failing for a reason, like the agent would do something and it would fail to edit. And I was like, okay, how do I debug this? Like, I cannot easily reproduce it. It shows up sometimes. And I think in the old world, what I would have done is put like specific logging in and try and log a bunch of these parameters. ⁓

and try and hone in or zone in on the problem, or I would use like sentry or error reporting and get like something up there. And what I then thought was, well,

Data and going through data is not an issue. So what I did was I hard coded in a code base, you know, just to save everybody's hard drive. I was like, if you want Thorsten's hard drive, take all of the data, put it in one JSON file and dump it in this directory. And then I had like a direct, after like a couple of days, I had like a directory full of JSON files with error cases. And it would have taken me too long. I would have given up because I hate string escaping in JavaScript. It drives me mad.

And you have like strings and whitespace and like escaped strings and JSON payloads in JSON and whatnot. and I was like, I just want to see what the data is. But then guess what? You have the data on disk and JSON files. So I started Amp inside this directory and I said, here's a bunch of files that contain edits with a before and after thing and like an edit, blah, blah, Build me a data viewer that lets me look at these files. And it went off and did it in five minutes and it added syntax highlighting and it added like a

the navigation, so I then went through the data and on the basically sixth thing I clicked there I realized, it's because of some white space tabs versus spaces thing that we were missing. And it didn't take me any effort at all. I just collected dumb data and then had the agent write code for me. I swear to you, if I had to log this stuff, I don't think I would have found the tabs versus spaces. so this is not applicable to anything, but I mean, it's just this, okay,

Wilhelm Klopp (55:05)
Mm-hmm.

Thorsten Ball (55:21)
what things would I have previously not done because it would have required me to write some code and we are all inherently lazy, but now we don't have to write it ourselves. So how does this change? Think of engineering. It's like we've discovered a new material where it's like you don't have to use screws and bolts for everything anymore. You now have glue. And now the question is, what do you build? You have glue and now it's like we have a dumb

Matthew Carey (55:28)
Yeah.

Wilhelm Klopp (55:44)
That's...

Thorsten Ball (55:49)
code generation machine that can put out 500 lines on the press of a button. How are going to use this?

Wilhelm Klopp (55:54)
That is amazing. That's really, really good. Yeah, I like that. And I think you've described that incredibly well. ⁓ And I will say, I have a friend in SF who I think has kind of also shifted his startup around mostly this idea of giving the most senior engineers at companies the special new button and thinking about how to empower them the most, as opposed to like building an agent that does stuff completely autonomously or any of the many other things you could apply this stuff with. So I like this.

Thorsten Ball (56:17)
Yeah

Matthew Carey (56:21)
I thought

what was really interesting there was the moment when you decided, and now I go by hand. And you actually built the viewer to show the data. Instead of having that JSON file and being like, right, now I have a big JSON file. And someone naively might have gone, LLM, tell me what's the problem.

Thorsten Ball (56:42)
I see what you're saying. Yeah, okay. Yeah, interesting. Yeah, yeah. Yeah.

Matthew Carey (56:43)
But you actually took the initiative there to build a view of, to help

you, rather than to ask the model the problem.

Thorsten Ball (56:50)
Yeah, Yeah,

I think that's a good point. think you have... How do I...

Matthew Carey (57:00)
Do you think you realized

that that was beyond the boundaries of like how to best use a model? You were dumping in something there that wasn't inside of any sort of like training space.

Thorsten Ball (57:06)
Yeah, yeah, Yeah,

I'll give you another example. So my wife, I've been pushing her to get more into chat, GPT, and she's a trained, ⁓ she doesn't do this, and she works in school, but she's a trained tax accountant, you know, which is why I fell in love with her. But she does, she does help me with the taxes. because I sell millions of these books constantly. And, you know, you have

Wilhelm Klopp (57:23)
That's amazing.

Thorsten Ball (57:32)
do bookkeeping at the end of the year. And she has like these CSV files that I have to download from the bank. And then she has to look at them and kind of consolidate stuff. And then she's like, should I use ChatGPT and drop the files in? And to this point, maybe it could do it, like whatever you ask, like remove the invalid stuff that has nothing to do with the book. But I don't know what the specific issue was, but I said, she knows a little bit about

know, web programming and stuff like this. And I said, you can have it, you can give it the first 10 lines of the CSV and have it generate like an HTML website in which you drop three CSV files and it shows you to them visualized. And I think that's the same point where you know like these are fallible tools, but they're really good at producing something that you can then validate for yourself. So if you can build a visualizer or a tool to help you do your job,

You know, like that's the thing where you're like, you know, instead of, I don't know, I don't know, like trusting the LLM to, I don't know what it is, what's an example, like trust it to sort your bank statements. Maybe that's a good use, who knows, but you know, don't trust it to do the job for you in all the cases, but can you.

Wilhelm Klopp (58:40)
Mm-hmm.

Matthew Carey (58:42)
Thank thee.

Thorsten Ball (58:46)
have it be an assistant and build tooling for you that makes the job easy for you. And I think it's the same in these code bases where to give you another example in our code base. ⁓ So Amp has like a CLI and it has like a web UI and in the web what we found out and you'll notice is these agents are really good with you know looking at browsers and they can open the browser and they're really good with feedback loops. So one of the first things that Quinn did was he created a storybook.

which is I think at once upon a time this was the name of an open source project, but the idea is you have a website in your dev server that showcases all of your components in different configurations, right? And each component has a separate URL, right? This is important. And then what you can say is,

Matthew Carey (59:25)
Yeah, I think we use it. Yeah.

Thorsten Ball (59:34)
You know, when you build a component and UI components, say I'm looking at Riverside here, like the thing on the right where it says like Thorsten Boyle guest, 1080p, 70 % uploaded audio levels. And if you want to really build this, you can click through it and test it manually. Or you go and say to DLLM, can you create me all permutations of all of the data that can go into this component and render me like this component, 50 different variants, and then I can scroll through and check.

Wilhelm Klopp (59:53)
Yep.

Thorsten Ball (1:00:02)
And it can do this in five seconds, or let's make it 15. And a bunch of these will be dumb data. I never will have 0 % uploaded and whatever. But it helps you do your job better. And then you can use that URL and feed it back to the agent, and it gets feedback from this.

Wilhelm Klopp (1:00:10)
Mm-hmm.

Yeah. It's like

a shortcut for invoking the right code in isolation, which is kind crucial for the feedback loop closing. I'd never quite abstracted that in this way. Okay. Interesting.

Thorsten Ball (1:00:25)
Yeah. Yeah.

And then

we did storybooks for our CLI. So now you can render components in the CLI and you can render the whole state of the CLI. then, ⁓ the orb, yeah, the orb is nice. The orb, yeah. Yeah, so I think it's little things like this where you're like, it's again, the button of 500 lines of code where can it build like a nice little component? In most of the cases, it's pretty good at this.

Wilhelm Klopp (1:00:38)
and

Matthew Carey (1:00:38)
wow.

I liked your orb by the way. I liked your orb. That was really cool.

Thorsten Ball (1:00:59)
but it's even better at generating your 15 or sample data or whatever it is or test data and just generating stuff and that helps you a lot. And the question is how can you leverage this?

Matthew Carey (1:01:11)
think the ML guys ⁓ like Hamil Hussain and those type of people, have their massive proponents of getting LLMs to generate data viewing tools. And so just hearing what you're saying is like, it gives me really big echoes of what they're saying all the time is like, use the model to make the data markup tool that works for your exact use case, because why not? Like, why not?

Thorsten Ball (1:01:33)
Yeah, yeah, yeah, yeah.

And it's also, know, ⁓ say you have a CLI tool and you want to test certain things. For example, I'm hacking on like ⁓ a JSON input and output mode for the amp CLI. And so it can output JSON on the standard out and it can also read JSON on standard in.

And I was like, I know how to test this if I can, blah, blah, blah. And then I was like, hey, generate me a script that launches this binary and feeds stuff on the blah, blah, blah every five seconds and done. You know, like it's, and it did it for me and that stuff I would have done by hand or not done at all. And it's not smart code. It's like a little helper script. And then you start to think, sorry, I'm going on. I'm going over time, sorry guys. But it's like, how can I structure my code base so that

Wilhelm Klopp (1:02:06)
Mm-hmm.

No, no, no.

Matthew Carey (1:02:18)
No, no, it's good,

it's good, it's good.

Thorsten Ball (1:02:22)
A lot of this stuff can be dumb little scripts and that's different than decoupling. I don't know how to articulate it yet, but I think there's something to this where you can, like, okay, how can I build decoupled components that in itself can do something but they may be dumb? it makes me think of, this was long time ago, this is 2012, I think.

Chad Fowler was talking about microservices and he was saying, back then the original idea for microservices, they did this at Wunderlist, was he had this talk and he talked about, ⁓ you know, biological systems and basically the biological system is constantly being replaced, like you're losing whatever cells and you can regenerate it and whatnot. And he thinks about big systems.

you know, in the same way, and you've got to make them resilient to this. So he thought about microservice in the sense that the stuff should be so small that you can replace one microservices in two days of rewriting with another implementation, and it doesn't hurt the overall system. And I'm doing the talk in justice, but it's this idea stuck with me of how can you structure your system in the way that, you know, it's anti-fragile, where a small part can be destroyed or rewritten in a day.

Matthew Carey (1:03:35)
Mm-hmm.

Thorsten Ball (1:03:38)
And it's easy to understand in isolation, you know?

Matthew Carey (1:03:41)
How do you find for this for like compiled languages where you might want to compile the whole binary? Or, yeah, how do you think about that?

Wilhelm Klopp (1:03:41)
Yeah.

Thorsten Ball (1:03:46)
Yeah, I mean,

I guess for compile languages, would, that's what I meant, right? Where I guess you want to have multiple entry points. You want to have multiple main functions that exercise different parts of the binary, right? If we think of like a GUI application like Zed, right? Like, I don't know, if I was still working on it and had an agent, think what I would try to figure out is, and they do have this, they do have the, I don't know what it was called.

but like how can I just have like one command that renders one widget or something? Like instead of opening the whole thing and you have to click two buttons to see the settings screen, how can I get to the settings screen? Like how can I isolate this so it's a single thing? And that's, know, with CLI subcommands and websites and whatnot.

Wilhelm Klopp (1:04:32)
mean, this feels like it should become a product, right? Like something that just makes like, because it makes so much sense that you need, like what's the alternative, right? That your agent tests something out by like signing into your application and clicking through in the browser. It feels, I mean, yeah.

Thorsten Ball (1:04:50)
Yeah.

It's really hard,

it's hard though, right? Because the models get better and then you get different feedback loops and the feedback loops change again. And I mean, if you would have told me, here go, can send screenshots to my LLM and say, flip these two buttons, you know, I would have said, that's not gonna work, but it does now. And so that stuff is gonna change. And that's also with, you know, coming back to the training stuff, like.

Wilhelm Klopp (1:05:01)
Ahem.

Hehehe.

Thorsten Ball (1:05:21)
Do you want to do a workshop and then say it might half of it might be useless in half a year when you know.

Wilhelm Klopp (1:05:24)
Yeah, for sure.

Although, mean, it does feel like there are some fundamental things which are like, I think you just have to decide to believe in them and it feels very believable, right? Like this idea of like, okay, closing the feedback loop, that's, I'm going to just say it as always kind of be necessary, right? Or it's always going to be useful. And this is just like this multiple entry points, lots of dumb scripts. All this is kind of describing as an easy way to close the feedback loop, right? Because, even if the LLM gets way, way, way better, right?

Thorsten Ball (1:05:35)
yeah. Yeah.

Yeah, that's

Wilhelm Klopp (1:05:56)
I think an LLM that can actually call the code it just wrote or run it or send the right data to it is going to outperform a much smarter one that doesn't know how to do that. I think this is still, I obviously we didn't talk that much about Amp and how you encourage this kind of stuff, like, ⁓ but you still have a lot of agents that even Claude Code sometimes, right? It's like it writes the code and then it forgets to run it or it doesn't, it decides not to run it or whatever, right? It's like, that's crazy. Like,

Thorsten Ball (1:06:01)
Yeah, yeah, yeah.

Yeah.

Yeah,

yeah, Yeah.

Wilhelm Klopp (1:06:23)
Whereas if you have the dumb scripts integrated well

or whatever you want to call this pattern then hopefully that happens way less and yeah

Thorsten Ball (1:06:29)
Yeah,

but it's the tricky thing is I find myself it's like a mental game you have to with yourself where you're like if I have to give it a feedback loop it might take work, you know, and then I have to prompt in this way. Let's just try. Let's just try what it can do it without it. And then sometimes it nails it and other times it doesn't. And then you fall into the, you know, prey to the sunken cause fallacy like it just one more.

Wilhelm Klopp (1:06:56)
Yep.

Thorsten Ball (1:06:56)
just

try it again, dude. And it doesn't do it. I had some, mean, yesterday was, first half of yesterday was really bad. nothing worked. It messed everything up. But over the weekend, I had some nice things where I played around with speech attacks. I was like, I'm going to write the longest prompts, you know? And so I sat down.

And I tested the Amp CLI I was wearing the Amp CLI, and I said, I want you to do the following, and I described what I wanted, and I said, here's how to test it, and I said, I have a TMUX session running in which you can run the following command to see the CLI, and then you can take the output of this TMUX pane and read it, and it's all text. And then I allowed all of the tool calls and everything, and it just went for a really long time, and it did it. And it was mostly this, it had perfect like,

it could send all of the keystrokes to the application. could see basically the screen, which is the T-Mux pane, and it had perfect visibility into whether it works or not. And then the next day you try it and you come back to your computer and it says, perfect, the feature is working exactly how I want it. And you try it out and it doesn't compile. you're like, dude, you didn't even run this, you know?

Wilhelm Klopp (1:08:02)
Hahaha.

Matthew Carey (1:08:06)
That's tough. Well, I think that is actually a really wonderful place to end. On that bombshell. I feel like a few episodes ago Will said, Will said he'd read a blog post which, how did it end? Something like, what the fuck is happening?

Thorsten Ball (1:08:10)
Yeah.

Wilhelm Klopp (1:08:20)
⁓

Yeah, yeah, yeah. I'm pretty sure this was, you covered this blog post as well, Thorsten. It was a blog post of someone trying to figure out how to work with LLMs. And my point was just like, usually you have blog posts that very neatly talk about how they solve the problem in their big company, and it's like, here's the recipe. And this blog post ended with like,

Thorsten Ball (1:08:37)
Yeah.

Wilhelm Klopp (1:08:41)
What the fuck is happening? Why does this shit not work half the time? It's just like a, I think an indication of the times that like you have blog posts that that end in like frustration and question marks and like, please like, why, what's going

Thorsten Ball (1:08:47)
Yeah

if it is the one, if we're thinking of the same block post, there was one by Orta, I don't know, but basically it was, this did some amazing things and I don't know why, and it did some bad things and I'm not sure why. And then I started to figure stuff out, but it basically ended with like, this is crazy than I thought it would be and I don't know how to explain it yet and we're in a new phase of this.

And it's strange, you know? I don't know. I don't know if it's the same blog post, but it's yeah, not a lot of people have figured out what's going on.

Wilhelm Klopp (1:09:26)
And there is reason to think we will. But yeah, it's a fun road while we're still figuring it out.

Matthew Carey (1:09:26)
And that's a good ending.

Thorsten Ball (1:09:29)
Yeah.

What I do know is stuff is changing. Yeah, know, like that's that's yeah. And it's exciting and fun.

Matthew Carey (1:09:33)
Thank you so much.

Wilhelm Klopp (1:09:38)
That's

where we have that everything is changing in our own jingle. we play it again? No, no, it's all right. It's all right. Thorsten, it's been an absolute pleasure. Thanks for dropping some wisdom on us and stopping by. Really, really, really nice.

Thorsten Ball (1:09:41)
There you go. There you go. All right.

Matthew Carey (1:09:43)
You

Thorsten Ball (1:09:49)
Thanks

for having me guys and the nice double intro. That was nice. Thank you. Nice to see you.

Matthew Carey (1:09:53)
Yeah, thanks dude. It was really good. was really good. Nice to see you again.

Wilhelm Klopp (1:09:53)
Hahaha.

So much fun.

Matthew Carey (1:09:58)
Before we leave, do you want to shout out where people can find you or do just Google you? It's quite easy.

Wilhelm Klopp (1:10:03)
yeah.

Thorsten Ball (1:10:05)
can find me, I'm thorstenball.com, my name, although it doesn't matter, .com. I have a newsletter, register, spill.

Since yesterday I'm Twitter famous. I ran it on Twitter about Microsoft Minecraft accounts and it's my most popular tweet of all time with like 17,000 likes or whatever it is. And yeah, it's strange. But yeah, that's it.

Wilhelm Klopp (1:10:28)
That's amazing.

Matthew Carey (1:10:29)
is change. Internet things.

Anyway, thank you so much. Bye!

Thorsten Ball (1:10:33)
Thank you.

Bye bye.