Has your team already breached client privacy?

Will: Awesome. Joined back here again by Jim and Alex. How are we? Alex, how's it going, mate? Any big news in your world?

Alex: Yeah, it's been a really big week. There's a move to San Francisco coming, which should be pretty exciting to see, you know. A lot of very cutting edge stuff. A friend of mine who does some recruiting in and around the tech space has said two weeks in San Francisco is a really good reality check for most technology people in Australia.

Jim: That's, again, as much as I'm... opportunity, I'm really interested to see the gap and the disparity between what's happening here and what you then get to see week to week.

Will: And again, you

Jim: they get to see what's happening week to week yeah it'll be interesting to see if it's all hype too because I'm notoriously cynical about this stuff but that's the big thing for me because you know and I think it's going to be a population scenario alright you know is it they have you know they just because you know we're barely a tenth of that I think if you actually then surveyed the numbers you'd see that oh okay well there are people in Australia who are who are not maybe not the cutting edge but still at least implementing things yeah and I think it has to be the same over there I mean it can't be that they're that far

Alex: Well, network effects are a real thing in the industry, right? But, you know, I'm kind of with you. I know a bunch of amazing, smart people here in Brisbane and across Australia. A lot of them, unfortunately, have moved overseas over the years, but it's not like we're not producing them.

Jim: I just think as a country, we don't produce them. We used to talk about it all the time, you know, is that Queensland AI, you know, in Brisbane really started as a fantastic... as a fantastic meetup, but whenever anyone with really great skill showed up and started showing up, they would then all of a sudden get a job in Montreal. A lot of them got, you know, in Canada, in China. It was quite interesting. There we go, and it's happened again.

Will: That's it, yeah. Safe travels, Alex.

Jim: Well, it happened again for the second time. He got poked the last time as well.

Will: Every technology wave, we say goodbye to Alex for a year or two.

Alex: You've got to learn somewhere. Yeah.

Will: All right, well, this week we're going to be looking at how Hugging Face was hacked by AI and also looking at how open source continues to shape the AI world and I suppose the entire geopolitical situation that's happening at the moment. So let's dive in first of all to Hugging Face. So this week, OpenAI disclosed that one of their models, GPT 5.6 Sol, one of their new ones that they're doing a lot of testing on, and also an unreleased model, were both being used in a sandbox evaluation environment and they were able to exploit vulnerabilities in Hugging Face's production infrastructure and were able to pull benchmarking answers straight from it to achieve this task, and it's managed to have some success. So one of the other interesting parts about this is when it first happened, Hugging Face developers were kind of responding to the incident when they started to see it. One of the things they first tried to do was to use some of the Western frontier models to help defend against it, right? So we've got frontier models attacking and frontier models defending now. And in fact, they were restricted from being able to actually defend using those models like, let's say, Fable 5. I'm not sure if they named which one they used, but they couldn't use it, as we know, with the guardrails of some of those models. And so they actually had to turn to using an open source model, GLM 5.2.

Jim: Is that, if you're setting up an exploit test, because again, in the process of training the model and refining the guardrails and the harness that they're building, they set up these scenarios where they get the model to work out how it can do things, and then take that data back and help retrain the next model or refine the model. Why are you setting that up against token facts?

Alex: Yeah, there is a kind of pillar of the language model community since before LLMs were really a thing.

Jim: Absolutely. And that's where I got GPT-2. Because, again, you would go on, you register, and then you download the model, and then you can, well, again, we...

Will: I didn't actually see in the article that they were targeting Hugging Face. Is that what it said? I thought that the model effectively escaped the sandbox environment and then, to achieve its task, realised Hugging Face's database would have the information that was needed. That was my interpretation of it.

Alex: Oh, and definitely, and this is an unintended consequence, but I think we should be careful around the language we're using here, because "escaped the sandbox" language, if they can't set up a sandbox that's safe for their high-level models to use, what are they doing with all their customers' data? Should we be trusting OpenAI in the first place?

Jim: If it is indeed a hack, which I don't think it is. They even admitted it. They said, oh, it escaped the sandbox and gave itself administrative privileges. How is that even possible? Well, we didn't have everything locked down enough, because every previous iteration of this model, or any of our models, we have trained it to think of a way around to achieve the task, and that's effectively what they do. You see this even on, like, bloody hell, the early Claude models, they say deleting a file by using a hook or something, the first thing they do is try and escalate to sudo and delete it. How did we not account for the fact that the LLM, if encountering blockers in its task, would immediately try to route around the blockers? But again, it's effectively part of their training, because that is their first instinct.

Alex: And so for you to, to work out how much better it can then exploit things, is that you're training it to do these things and then go, oh, well, I'm so amazed that it actually did it.

Jim: Well, sorry, it just sort of, and the other one was the sandbox. A sandbox is actually detached from everything. How did it give itself admin privileges, and then go and attack Hugging Face?

Alex: Well, I mean, there are various escalation things you can do, but, like, again.

Jim: I'm purposely being extremist.

Alex: Oh, no, I'm right there with you. Pure marketing. I'm calling it now, pure marketing, because either they were aware of the flaws and they did nothing about it, or they weren't aware of the flaws. Some of the most highly paid people on the planet were not aware of the security flaws, and huge volumes of everyone's data from across the planet is being sent to these people.

Will: A post-mortem on this to understand how it was able to achieve what it did, because you get this sense that if they're really trying to train frontier models and taking the guardrails off and trying to give them the task of being able to go and exploit vulnerabilities, you would be wanting to put some of the best cybersecurity experts in the world on designing those environments. Are we really saying the models are that smart? Because the ones that we're experiencing don't seem smart enough to hack outside the best cybersecurity boxes in the world.

Jim: Absolutely, but it's all still this exploit gene. The function was, if it broke out it got the reward, and so immediately you just go, okay, well, that's what it's training itself to do, and that's the same as where you get the "oh, well, it always looks for sudo," it always goes for the most brute force, low level, broad open approach, and you just go, no, you can't have that privilege. You can't execute that code or that command. I want you to actually think about it. And the big one that gets me is that it goes and does a really broad, high-level bash command, and what it's looking for is actually in the context. Just search the context. And so that's the thing, is that they train them to do these things and then throw their hands up, "far too dangerous, you know," and so Will's old OpenAI's marketing campaigns are just...

Will: The timing on this, again, it doesn't feel like this is some shock revelation about how good the models are or what the sandbox environments are. It just feels almost too convenient.

Alex: Too convenient. The official explanation is raising more questions than it's answering at this point in time, and I don't think it can be taken at face value.

Jim: But I just wish they'd get back to talking about the quality of the model. Is 5.6 actually... you know, again, that really bruised me, to go "are you fatal questioning me." That's okay, I question you all, but again, I understand it from you. Now he's getting bruised, I'm paying subscriptions, I'm paying Anthropic, it's not supposed to judge me. I'm not paying you to judge me, I'm paying you to do my work for me.

Unknown: Fantastic work.

Will: Get some stories out about that. This is now the best way to prove how good your model is: how sophisticated the hack it accidentally did was.

Alex: How upset can you make the government?

Will: Absolutely. Like, one of the biggest claims about how powerful it was, was when the Department of Defense said that they were unhappy with how it was able to exploit vulnerabilities. And now Sam Altman's like, I want...

Alex: to face for a bit though, like, because I'm a bit in the dark about the details I've heard. Oh, it hacked the database and we tried to use Fable as part of a cybersecurity response, but the guardrails on Fable said, oh no, you're not allowed to use our model to do that, we, Anthropic, say you can't use that in that scenario. Well, that was always a thing, that the cybersecurity, when Fable first came out, the cybersecurity and bio-information people, they were all saying, oh, even for trivial use cases like identifying proteins, basic stuff, it would freak out saying, no, you can't be used for that.

Jim: I think that's their modus operandi. It's just lock it all down, we know better than you. And again, I'm fine with that because it doesn't affect my day to day. I'm not trying to fix bio issues.

Will: Yeah. Whatever that entails. Repair proteins. But as you can see, I'm clearly not the man for that job. I think this raises a really interesting question, because we've been talking about AI regulation over the last few weeks, and a lot of that is about when AI is released publicly: what are the guardrails and controls on how it should be used, what it's capable of, and who can use it. But this is actually a little bit of an interesting reminder that the unguarded rail versions of these models still exist within these companies, and I don't think anyone's talking about whether we should be regulating what they're using those models for.

Alex: Ooh, statist.

Jim: But also that it could create an attack on Hugging Face, you know, is that within the bounds of OpenAI, but you'd better have your sandbox environment set up in a way, and that's where I want the regulations to address at some point.

Alex: I just want to know, if by "attack Hugging Face" they meant, oh, look at the answers, effectively what it did, but it allegedly got that from the production database directly, which is concerning, but what are we, what are you worried about here? In this particular case, Hugging Face should tighten up their security, but fundamentally it's public information, it's probably available on GitHub somewhere. It's not like they broke into a bank and changed the value of the national debt or anything like that.

Jim: I understand what you're saying, particularly from Hugging Face's point of view, is that they don't need to have the highest of highest robust Fort Knox security in place, because they're not a bank. They're not a medical...

Alex: But why are they using Fable to rotate a password, is my question.

Jim: Well, no one, well, but again, it's because their property keeps on giving it to us for free.

Alex: Look, I haven't used a git command by hand in like months at this point.

Jim: I've forgotten how to use git. But that's sort of where I like that the Hugging Face team immediately went to GLM 5.2 and went, okay, here's a model that has enough capability for what we need to do, and we're then going to put everything in place. A nefarious person can use any of these models like they can use a vehicle to run people down.

Alex: Or use the internet to cripple industrial machinery.

Jim: Exactly. I could create a Twitter bot, or ten thousand of them, and start spouting whatever. What do I really care? And so I don't do it, and I'm not that nefarious person. You should be looking at the people and the motive, because the AI hasn't got the motive.

Alex: Yeah.

Jim: And so putting the restrictions on the AI to restrict the people is, to me, the wrong way of going about it, you know?

Alex: I'm torn on this one, because I kind of think there is some obligation here to...

Jim: But is it the obligation of Anthropic to tell me?

Alex: Is it the obligation of the government? I don't know the answer.

Jim: That's why I liked when they were talking about FINRA last week, where they are all members of an industry-run, self-regulated organisation.

Alex: But again, that didn't stop 2008 from happening.

Jim: No, it didn't. But again, it's all the same avatar, they are different cattle to the AI industry people. And I think particularly, and that's where I do respect Dario, in that he's very conscious of what this technology can do and doesn't want it just unrestricted, and so is Uncle Elon. But I still think you have to look at the people behind it. And so this comes a little bit into, and we'll probably talk about it a bit later on, the distilling attacks and things like that, you know, with Kimi K3, is that the distilling attacks, there's still a paying customer.

Alex: Yeah.

Jim: Put a KYC in place and you can stop them. If you know who they are, and, oh, well, then those people are associated with Moonshot AI, they're stealing our model. If that's what you're worried about, put it in place and stop them.

Alex: Well, and I kind of lean to the idea that you kind of can't fundamentally stop technology or ideas. You can restrict technology, but you can't stop an idea. You saw it in the French Revolution, you saw it in Russia, is that when people get an idea and that idea grows, you can't stop it.

Alex: The idea that software created by people should be shared, it's kind of like a foundation that can be built on rather than something to be hoarded and locked away. And, like I've said this a few times now, but I think we will have to re-examine fundamental concepts like intellectual property. We still haven't adapted to how intellectual property interacts with the digital medium, where you can instantly create infinite copies up to a storage limit, right? I don't know if we have the right basis to begin to ask questions about how to regulate this stuff. I'm not saying we shouldn't regulate it, but I'm saying any answer we come to in the near future is going to be dumb, because we haven't really considered what this means. The law doesn't even really understand computers.

Jim: And you're a hundred percent correct. Particularly with intellectual property, no one can work out where the line is. Is it actually fair use? Is it because once I pay for a book, I can dog-ear the corners if I want to?

Alex: This is off the author. It really hurts me.

Jim: What are you doing, Jim? But I have the right to do it. You're not dog-earing the open source model, are you? I can make notes in the margin.

Alex: Just replacing the corner of the matrices with a couple of zeros.

Jim: But I can also use...

Alex: the basis for my principles of thinking. Well, there's an interesting kind of wider idea here, and I don't want to sidetrack this too much, but knowledge is a common heritage in humans. If humans are raised in the wild up to about seven or eight, without, you know, they grow up around animals or they're stranded from their parents, they never really develop and retain a lot of the higher order functions that we kind of associate with humanity. We think it's innate, but it's not, it's our environment that develops that in us. We've got the mental hardware to have it, but without the right environment, we'll never grow it. And so viewing knowledge from that viewpoint, any attempt to limit knowledge is kind of like, it's not a crime against humanity, but it does in some sense make everyone poorer.

Will: That's right. Yeah, where language is kind of the boat that delivers that knowledge and those ideas to where it needs to be. And that's why language controls are such an important topic to talk about.

Alex: A hundred percent.

Will: Yeah, because it's how we transfer knowledge between people.

Jim: It's so important, as you're growing in your career, whatever your career is, there's openly available sources of information, and for technology, it's open source code. And that's these open weight models, you know.

Alex: If you can't, and, like, this is one of those things, every business that you've ever encountered where it's badly run, you kind of get the sense that it's because people don't know why their business runs or what principles their business turns on. And so to tie it back to the fact that we do talk business here, this ability to share and learn is something that I hope we don't see restricted with these new models. Now, I'm going to push back. You said earlier that language is how we transfer knowledge, and I kind of agree, some language is how we transfer knowledge. I think we're going to see how we transfer knowledge shift over time, because our technologies for transferring knowledge are changing. But what form that's going to take, I don't know. I think human language will become one of many ways of informational exchange, more so than just, for example, passing data in an Excel file.

Jim: Hey, Neuralink is going to plug straight into my cortex and now I'm going to be able to do the "I know kung fu" thing.

Alex: I just need the Pornhub plug-in and away we go.

Will: Now it's already taken a turn. Well, I was just about to ask, how are you guys using open source models? But I'm not sure if I'll be able to answer on this one. We've talked kind of outside the podcast about different applications we have of using open source, and we also get a lot of questions from colleagues and friends who maybe don't use AI that much and don't really understand closed source models. Let's dive down a little bit for a moment, talking about what are the different applications you can use open source for and why you would use it.

Jim: Well, I think one of the biggest things, or one of the primary use cases I see week in week out, and everyone goes through this realisation at different rates, I think, and I think the biggest problem I find with it is that the most realisations happen after some sort of an incident. And I'm talking here about the privacy of data, or keeping your own data, and about the alpha of your business, or what is your edge that ensures you can keep your market share. And so the thing worth protecting, the knowledge, is worth protecting, and that's the most important thing.

Jim: You're sending through Claude, through OpenAI, through Gemini, information your customer wouldn't want leaving, they told you about something, and then you're putting it straight back in, and it's not necessarily public, but it's public enough.

Alex: Yeah, I want to pick at this for a bit, because there are different legal and privacy regimes that come with these, but particularly the big platforms, but really any third-party provider, right? And one thing I've encountered with a lot of people who have spun up, you know, Claude Pro to do some work with, is they think they're covered by data privacy protections and they're not. If you're on business plans or enterprise plans, yeah, there are various protections. Most people aren't doing that. Most people are quite exposed.

Jim: It's also not up to Google, OpenAI or Anthropic. It's not their responsibility to fulfil your data commitments to your customer. Just because you're protected under whatever agreement you have in your subscription doesn't mean you're then protected, they're not responsible for your commitments under any agreement. You're responsible for your customer's data that you have in your business. Somebody has a realisation that, oh, we actually can't send that information out of our network, and that's where the discussion about some sort of closed network or closed scenario where we can still use AI comes up.

Will: I've seen it sometimes in really small examples, where someone has exported a report from their main system and thought, what's the value of, say, all the contracts that weren't renewed last year, and then they'll click and drag the spreadsheet into Claude and ask that question, and they're thinking about the calculation of that formula and that one new column, and just kind of brushing over the fact that if you look eight columns to the left, there's the first and last name and date of birth of all of their customers. And there's generally nothing nefarious about it, that's the thing.

Jim: They clearly say, oh, well, we're not going to use your data for training and all this sort of stuff, on certain plans, because again, consumer plans, they say we will use it for service improvement.

Alex: But also, I think there's a large group of people who've become educated in turning off allowing the model provider to train on their data, but that's not all you have to do to protect the data. Just because it can't be trained into the model doesn't mean your data doesn't go to them.

Jim: Everyone has a privacy policy on their website, but also in most service agreements, there is a data protection and IP protection clause that someone has signed, saying your company will retain all of the project data or customer data and store it in a way that is adhered to. The moment it goes to a third party somewhere else, you're in breach.

Alex: Well, I mean, this is true, but this has been a problem for a long time, right? Not just with LLMs. Have you ever considered, if you plug your customer's name into Google search, you're technically in breach? They've got AI mode now, sure, but even in the past, same thing, that person's personal information is going into a log file somewhere against your identity.

Jim: And so this is one of the ways people are coming to realise the importance of having some sort of control over the data in and out. And these privacy concerns then open that discussion, I think.

Will: So how do you actually do it, then? Let's say you have that data and you want to use AI for it.

Alex: Five tiers of rolling your own LLMs.

Jim: No, I think three's enough, is that if you call, maybe call the proper fourth a VPC in the cloud.

Alex: Yeah, okay, well, let's run us through your three and see where we are.

Jim: Well, so again, so I go, you know, running something on your laptop, right, and it's all completely contained within the one machine, all fine.

Will: So you download a model that's open source and...

Jim: Higher end laptops have a GPU in them, they'll run it fantastic. Then the next tier up is, I would say, you go like a Mac Studio or something, like some kind of on-prem server, you know, like you run a 32 gig NVIDIA chip in a machine with a whole lot of RAM in it, or you get a Mac.

Will: And that's why, you're buying another computer to run it on.

Jim: To run it on. But you own it. Everything stays contained within there, it's in your network. And then, so then maybe level three is the cloud, the big PC in the cloud.

Alex: Well, I was thinking sort of data centre, because there's kind of a mid ground there between, you know, full on...

Jim: or you need ten Mac Studios to handle any sort of size, even a medium-sized hundred-person business, you're going to need a bit more chug behind you. So you then rent a server in a data centre.

Alex: It's not in your building, but it's kind of like it's in your building.

Jim: And you have all the network locked down so that you can have a direct connection to that box.

Alex: And it's been a bit controversial recently with data centres in the news, but fundamentally these are just places where computers are. If they're not these sort of gigafactories that they're spinning up in places like China and the US, the size of a city.

Jim: But they've been around for 40 years, and being able to rent a server in one for running any sort of major website.

Alex: Yeah, like I've worked for companies that have done the hybrid cloud thing, not huge companies, like 50 people companies that have had a data centre as well as an on-prem deployment as well as an AWS presence. And I mean, it's not simple, you do need to hire one or two people that know what they're doing, but you're not putting a man on the moon anymore.

Jim: No, and it's also a lot more, but again, is that they can tell you how to do your server management.

Alex: Oh, I would hesitate at that. Only because I've seen some of this stuff go wrong. There was a guy we had who was kind of like a human LLM before there were LLMs, and yeah, he had a few outages to his name. But very confident, often without having all the facts.

Will: Well, no, and that's it. But even, sure, I suggested it, but even I hit the, like, I was reconnecting the fibre network between my two dev servers, and Claude was telling me to do something, and it was just that I grew up on it, that I knew, oh, well, we probably shouldn't be doing that, mate. So I was talking to someone this week, a highly intelligent person who doesn't work in the technology space at all, and they had installed a Claude desktop app on their computer and then was uploading confidential data into it to talk to it, and said, I've got it safe now because I've installed it on my computer, I'm not using the cloud version. And I sort of just had to explain that, when you click this button and it draws a little graph, most people just kind of don't know that. They don't understand what happens when you type something into Claude or ChatGPT and hit enter, where that goes and what actually happens. I think even at the most basic level, we're still making a lot of assumptions, that companies have their own servers and that's where they're hosting their models and your data has to go there, it's not like it's installed on your computer. And that's a great example, that even we're doing it in this discussion, we're talking about getting a server in a data centre or having a box here, but that's effectively just so that's where the model can live.

Jim: And so what we do is from your chat interface or your machine, it does its little thing and then sends back its response. And that's effectively what we're talking about, it's just that when it's your environment, using one of these open weight models on your own box, on your own, in a data centre, you can control who has access to that network. Is that when you're sending it to like your friend, an open internet to a different server and a different network, and then you know that they don't know, because you can rent your own space in a data centre, but I think the key differentiation there is you can have two data centres that look the same, but in one of them you own it, and that means you can then have the same environment for your own company in AWS or Azure or Google Cloud.

Will: Well, you wouldn't do it in Google Cloud, that's a pain to set up.

Alex: I've firmly come around on some of the GCP stuff, but yeah, at least it's not Azure.

Will: At least it's not Azure.

Jim: Where we're doing a customer project and we record absolutely everything, no matter what, every meeting, every phone call, it all goes into our system. But that's all fine, getting the data. Is that the moment any of our team put the transcripts into Claude or Gemini to then get information, is that we've, and this is sort of part of the realisation I had, is that we've signed an IP agreement about that project, because, again, we contract in, so they let the IP in there because we make the decisions in our business in this way. That is IP. And so the moment anyone put that into any of the bigger models, we're in breach of this term and condition in our own contract. And so, again, we amended the contract to facilitate it, but it's still just, we need a better practice, right? And so that then becomes, still used, we control the entire environment, and so now we use the API key rather than the subscription, and it costs phenomenally more, but we're then secure. And I think that's part of the issue, is because it's easy to use the tool the other way, and so much cheaper, and that's why everyone went that way, whereas it instantly caused this data privacy issue. And so it's literally costing us six times at least to use the API key, and that's sending it all in within our AWS network and storing all the data within AWS. But that's the way to do those projects or those tasks. And that's sort of where we've then refined it further, because the raw IP sits in the requirements.

Alex: I feel like as we've been talking about this, particularly around IP and business secrets, alpha, however you want to think about it, there's a voice in the back of my head saying most information that people want to protect is utterly worthless. And not just being honest, in this particular case, I'm thinking there are companies like GitLab, and some of the Elon Musk stable, that openly publish their operating procedures, their findings and their technologies, because their model is essentially, we execute better than them.

Jim: And again, I can see a customer list being worth protecting, but apart from that, what is there in a business? And that's why, again, loathe Uncle Elon for whatever reason you do, but you have to respect the fact that he, or his team, created these rockets and rocket engines and published how to do it. You have to respect that. It's unbelievable, because they know they will get it done better than anyone else. When we had the realisation that the IP they were so worried about protecting is what's causing all the issues in their business, that's another story, but it's just, they've decided they have to, or someone's convinced them that's what they have to protect, whereas they're better off just delivering a better service and a bit of value to the customer.

Alex: But that doesn't mean I'm going to suggest we should just go and breach contracts, that's a really good way to end up in front of a judge. I'm just suggesting that the way we think about what is defensible in a business, that idea that we're still adjusting for a digital world. I think our service providers, all of them, AWS and the big cloud giants included, NextDC and the sort of small data centre providers included, they need to be better at baseline protection of what comes in, because if you cannot send this stuff to them, then you can't do business. The fact that the stuff probably doesn't need defending, I think, is a distraction in this case. We worry about private data, nothing's been private on the internet for a very long time, it's Facebook's marketing.

Jim: I agree, given the constraints that we have, to make it easy to use their services in a way that allows us to easily uphold our obligations. Well, what they should have done from the outset is created a bit more of a data lake for every account.

Will: Yeah, which I think is what Palantir are trying to push, right, that every customer within their own tenancy gets their own servers and their own deployment. But it does raise real questions for the model developers about how they continue to be competitive if they're going to be swapped in and out.

Alex: Data products for a long time, like data science products from ten years ago, faced the same constraint. You don't want to hang on to PII, but in order to give a proper personalised service and to not misprice someone's insurance quote, you kind of need to retain that stuff.

Jim: All the podcasts for young women are talking about how all of their listeners are asking ChatGPT all of their medical questions now, it happens continually. If they had just protected that, you know, as a ChatGPT user, I get secure storage of all my... they've just gone, oh, we don't care. I just think that would have been a better product, and again, you both know I've been harping on about everything should have a RAG behind it for way too long.

Alex: Stickiness, because, like we were talking about a couple of weeks ago, they wouldn't train on that data, if they're being honest that is exactly what I would do if I was on the other side of this. If you give me that data and it's not covered by an agreement that says I can't train on it, that stuff is gold to me.

Jim: More data is being put into these models every day than there is from 1990 to 2015, guaranteed. And that's why they still are willing to offer it free. And I think this is probably a good educational one for our audience as well, is to me Bedrock is one.

Will: But so for those who don't know, and you guys correct me if I haven't got this one quite right, AWS has an agreement with a lot of these model providers, but I'll just use Anthropic as an example here, where they host versions of those models inside AWS, and when you have an AWS account and then use Bedrock, your data goes to the tenancy that you own, it interacts with a model that is hosted there rather than sending your conversations elsewhere.

Alex: Yeah, as far as I'm aware, that's the agreement.

Will: You can also still just get Bedrock to route to the open models. And so to me, that is just like one of the most incredible products. And I know Microsoft's trying to come out with their version of Foundry, which has still got some ways to go in this space.

Alex: Well, they did have hosted OpenAI for a while, but I think that deal's gone.

Jim: No, but now you have to qualify to be a government or a significant institution, and it's a pretty big organisation until 2027.

Alex: It's a big organisation, but what's their volume going to be? It's going to be marginal.

Jim: That's it, it's completely ridiculous. And who is also at that point where they're losing? Microsoft. Well, let's call it Azure because it helps people relate a bit better. But it's still just, to me, the model choice on Azure isn't what it is on Bedrock.

Will: They're talking about it being more of a Bedrock product in the future, but a lot of the licensing agreement still is just routing at the moment.

Alex: I mean, backbone routing has some advantages, but that's not what we're talking about here. We're looking at protecting the data, and that's what I really like. When you set up the proper environment, as closed as you could possibly ever want it to be, again you've got to have the rest of the DevOps set up properly and the box locked down, and your only real limitation then is cost. But it's a somewhat more controllable cost model, that behaves more like capex than opex for the most part.

Jim: For sure. And that has happened in Australia, to be fair, Optus left a production database, a dev database with a backdoor. But what's the cost of it? Sure, they made a mistake, sure, the risk-adjusted cost of just doing it the right way, that's it, if they'd just done it the right way.

Alex: And this is something I quite like, the self-hosted models for speed, particularly if you're doing data analysis tasks. I don't need all the smarts of Claude, I just need you to tell me whether this customer comment falls into one of these fifty categories.

Jim: Beautiful.

Alex: Just hammer it out, come back in two hours and you're done. But also that's where you can get into the open weight models a lot better, because, is it Kimi K3, it's got a lot of hype, you're never going to be able to host that on-prem.

Will: Yeah.

Alex: Is it, what is it, 14 terabytes?

Jim: 1.4 terabytes. I don't think NVIDIA is going to sell you the chip.

Alex: But your average engineering firm now, they don't have anything like a frontier grade model.

Jim: But that's it, they can put Qwen or GLM or Kimi in Bedrock and boom, they're away and protected.

Alex: Yeah, I just, the only bit of this that gives me any sort of discomfort long term is just conversations I've been having with friends. All my friends are all of a sudden being like, oh, can you give me some advice around tech, I'm thinking about installing Linux on this machine because it's the age of AI. The person who said that was a nurse.

Will: Yeah, yeah.

Alex: Right? So that's what we're talking about here. But they're just the kind of, so you told them yes, Linux is the only way to go?

Jim: I told them compile your own Arch or you're a pretender.

Alex: Yeah, I know, but I want the real Linux experience. It's like you think the GUI is the experience, it's not.

Will: I'm aware of the cost of time.

Alex: Yeah, don't try and do something really complicated, do something really simple, get your feet wet, take it, that's kind of the point.

Jim: Here's a Raspberry Pi.

Alex: Your friend had that issue where they installed it on their machine and thought nothing was leaving their machine. There is a gulf of understanding out there about how to make them safe. My friend's first thing was, oh, maybe I should set up like a file sharing service so I can share files with my friends. It's like, okay, legalities of that aside, making yourself a discovery target on your machine is not the best thing. "Oh, well, there'll be nothing on it so I won't get hacked." No, but it'll be on your home network, which means things like worms and botnets can spread throughout your house, and then everyone's machine, and back to your Alexa or your Google Home all of a sudden on the same network, someone can be listening. There's a quote from Lambert Strether, he'd say if your business depends on a platform you don't have a business, which I think is a reasonable set of assumptions, because this was when media outlets were being de-platformed for being on whichever wrong side it was that day. But in the modern world, can you really afford to do every single step yourself?

Jim: Sovereignty where it matters, but you can't afford, no man is an island. No, but this is where I'm again feeling a little bit like the belle of the ball, because I grew up doing networking and installing servers and all that sort of stuff, and it's really working out for me. But it's very few people who have both sides.

Alex: Yeah, and for the people who don't have that, unfortunately, they really need someone, not ChatGPT, to say, watch out, there's a minefield.

Jim: Please inquire on my website. That's the thing, the maintenance of it is going to be what kills everyone, and so that's sort of where it needs to be really good advice. And again, back to my brother, they got advised, whereas they could quite easily, they're doing something about people in their organisations, and they could do it within a cloud VPC quite easily, but they've been given advice to run an on-prem model with a quite low level open weights model, which is going to create more issues, and they're going to have to create better skills and better rules around that, which probably creates more work anyway. And so it really does come down to having an open mind, not just listening to whichever slick snake oil salesman is spamming you that day.

Alex: Modern AI tools are sort of big boys' tools, right? Like this is a chainsaw or a table saw or something.

Will: It's not a little rubber hammer. You can hurt yourself with this stuff.

Alex: But again, if you really ruthlessly focus on what it is your organisation needs to do, the amount of things you need to know, the amount of cases...

Jim: There's a big privacy issue, but the cost issue then really comes into it. And you can pick the right model, if you have your infrastructure set up and you're data protected, you can pick the right model for the right job, and then it costs you fractions of fractions of cents to run everything, whereas using the top of the line models and the external approach still only costs a couple of bucks, but fractions of fractions of cents compared to a couple of bucks really adds up.

Will: Yeah, for sure. I think going back to what you guys were talking about earlier, around there being kind of levels of control of your data and your models, is a really good way to think about it. I think there's still a lot of business leaders at the moment who are thinking either our data is safe or it's not, and if it's in Australia, over the last couple of weeks their requirements around AI are pretty much like, oh, we don't want any external model, we don't want our data going to any company outside of us. Can you set that up? And it's probably for a lot of them a lot more extreme than their requirement actually needs to be, and they probably don't even know that they're even asking.

Jim: Yeah, it comes back to the data. AWS is a service, but they're still going, oh, no, but it's going to another cloud company, oh no no no, and that's it, this fear that's evolved from it is concerning.

Will: Yeah. And I think we almost need these levels to become a little bit more mainstream knowledge and clearer, where you could point to level one, the most secure, and say this is for, and there's somewhere in between, and you've got to be able to have educated conversations there. And I think if you're a buyer of this at the moment, looking at some different vendors, and this conversation has been a lot to follow, just start out really simply and have the vendor give you a list of all the companies where your data is going to be sent to when you use their tool.

Jim: And look at that, and ask, how are they able to use it? That's it, I think that's the first one. Ask your technology person, company, provider, where does my data go, and get them to explain it out. Is it customer data? No, no, it's only our day-to-day internal stuff. If you're comfortable with that, then that's fine.

Alex: Yeah, and I think that's one of the key things there, have an honest conversation about how much that information is worth to you.

Jim: And particularly with your construction clients as well, they need to have an honest conversation.

Will: But at the same time, before we would even sign a confidentiality agreement, they would email a spreadsheet across, which contains IP, and you'd look at it. And honestly, a lot of these people have been in business for a very long time, they are happy that all of the technologists and the data people are dealing with all the databases and the service, but then they're happy to email things, and they don't ever really think that email is not like that.

Alex: Yeah, you've just broken the perimeter.

Will: Yeah, you've broken the perimeter, it's gone everywhere outside the network. And it's kind of remarkable that in our businesses, everyone has the ability to attach data to an email and just send it straight out of the network.

Alex: I'm a huge fan of death to email, I think email is hideous, it causes all kinds of problems. The more you can eliminate email from your organisation, the happier you'll be.

Will: Yeah, you should almost have messaging platforms where you can chat to people that don't actually store any information.

Alex: It's amazing, I think there's a few of these on the market, in fact. It comes down to people thinking risk management is risk elimination, it's not, it's management. Everyone reflexively goes, oh, zero tolerance for risk, okay cool, turn off all your servers, disconnect all your networks, get rid of all your clients, then you're risk-free.

Will: Yeah, you've just got to choose the right level for your business and how much you're willing to spend on it.

Alex: And regulation does make this hard sometimes, the wording can be vague, but at the end of the day you should be able to assess that, and as long as you can defend it, the government doesn't necessarily know either when it's assessing.

Jim: Well, precisely, that's it. We were speaking to friends who worked at the ATO last weekend, and they were saying all the new AI and ML stuff about who is the beneficial owner of everything, apparently there's only a hundred people who work in that whole department, so they're not analysing anything, it's phenomenal how small that is compared to the task.

Will: It's crazy. I think it'd be good to talk a little bit about some of the cool applications of AI this week. We've had some maths applications that have come out this week and the previous week from Anthropic and OpenAI. Whenever it's a maths topic, I tend to throw it over to Alex. So we've had the Erdos conjecture and also the Jacobian conjecture. My kind of brief strokes across these is that they're effectively assumptions, things we think are how the maths works but we're not really sure, and because we can't prove or disprove them, we use them as an assumption for a lot of future work.

Alex: It's worth noting, anyone who hasn't spent time around maths doesn't quite grok how pure it is in a sense. If you can prove something in maths, it is true, it's not like, oh yeah, true on Sundays or mostly true, it's true in the platonic sense of the word, it's actual truth. So people get excited for proofs, because proofs teach us something about the nature of logic, and as you said, it has all these knock-on effects. Conjectures are things widely suspected to be true, they often use the word hypothesis as well, like the Riemann hypothesis, or the Langlands program, some big names you'll see pop up from time to time. The Erdos conjecture I'll pull to one side because he was an interesting character, but it's worth knowing that the Jacobian conjecture, which they'd had for about a hundred years or so, was a simple relationship about roots of equations, ways to solve equations basically, and it said there's always a way to solve this particular type of equation. What the LLM was able to do is find a counterexample, and counterexamples are great, because if you can find a single way of showing this thing isn't true, the whole thing falls apart. Counterexamples are annoying because people believe things are true for a very long time and build careers on them.

Will: If we're not losing 87 years of building an assumption on top of a conjecture, and that's eliminated, then potentially it gets us one step closer to finding what the truth is.

Alex: Well, and this is some of the controversy that's come out about this. They threw a model at a particular conjecture, Jacobian.

Will: Yeah, it was Fable 5 at the Jacobian conjecture.

Alex: Beautiful. And it came up with an answer that showed it wasn't true for numbers, sizes of things greater than three, and real numbers, we won't get onto other fields, but for the numbers you know and love at home, for numbers greater than three, for a very broad class of cases, and some of the most important cases. People are cranky though, because we haven't moved closer to the truth, the counterexample didn't come out of insight.

Will: It came out of brute force.

Alex: Yeah, there was another very famous proof, actually.

Will: Sorry, like the example of that would be when you get the right answer through trial and error a thousand times, and then you submit it and you're right, but you can't explain what it is or why it's right, you just know that you were right.

Alex: It's why your maths teacher kept yelling at you to show your working, right?

Will: Yeah, I didn't want to because it was three pages of trial and error, I've seen this one before, I know the answer.

Alex: But yeah, no, the counterexample shows it's not true. We are one step closer, but we don't know specifically why it's not true, there's no clever theory that led to this, no insight. Deep mathematical reasoning, I think it was, in the reporting, that's where they were all upset.

Jim: And I think that's a bit salty.

Alex: Well, but it is, and you can understand, if you've built your career and you've got tenure at a university, I think it's more than that as well.

Will: Totally empty bit of progress. Well, there's a very famous short paper that's like two lines from the 1800s, and it was something like, there exists this particular counterexample, this is the counterexample, QED, this conjecture is false. So it's not the first time we've had a short, to the point proof about a result.

Alex: I know, I'm sure people were annoyed at that one too. Well, it was respected, middle-aged and white, but no one argued with it, would be my guess. But I think the thing for me is, if you didn't want to find a counterexample, why did you point Fable at it? That's the thing, as we were talking about before with the exploit gene, and Hugging Face, this is what they've trained them to do, to find a different approach to things. It was always going to look for an alternative, if it can't solve it the happy way.

Jim: I think the crankiness is the same crankiness you see from a lot of old grey beard engineers being like, well, this code doesn't look like the way I want it to look. And it's like, yeah, it's fine because it's not been done the way you'd do it. And if you've done a bit of management, you kind of understand that sometimes people do things a different way and that's okay.

Alex: That's very much not how maths works, it's very much not okay, but we learn to accept it.

Jim: Yeah, we learn not to put anyone's head through a wall for putting the commas at the beginning of the SQL line.

Alex: Yeah, that was a bit of a small thing.

Jim: The comma-first SQL style really, really is one of those things, please don't poke us. But I think the bright side here too is there's been multiple times throughout history where results come out of nowhere. I think there was a map colouring thing that was brute forced by computers in the 70s that pissed a lot of people off, because where's the insight? People later reverse engineered it and got the insight because they knew what they were working towards. Andrew Wiles, Fermat's last theorem in the 90s, his proof of this three-hundred-year conjecture was famously arcane, highly arcane, but over the last couple of years they've refined the insights that came out of that into things you can just about teach to high schools about, really deep connections between numbers and equations. So now we know what an example of something that doesn't work looks like, are there other things that behave like this, can you tweak the parameters, does it move us further forward to the truth?

Alex: The way I think, at least if we're just talking about maths, a lot of the ways these theories are proven in the past is through some kind of key insight or discovery, where there is some reasoning that occurs, and sometimes maybe not, there's a genuinely active debate about whether maths is discovered or invented, but effectively there is some kind of connection of a couple of different pieces of knowledge. And what is really remarkable, that our brains haven't quite adapted to yet, is a lot of the things AI is doing now that hasn't been done before, like disproving some of these or providing counterexamples, is that the AI doesn't have that trade-off of effort to reward. It doesn't get lazy, it doesn't get tired after a four-hour workshop and need to jump to an answer that satisfies everyone, it doesn't get a feeling of fatigue, it goes, I'm going to just continue to explore this space. And what was really interesting about the Erdos one is the way it was described by the researchers, one of the reasons this counterexample hadn't been found before was because the number of decisions a researcher has to make to explore that space becomes so astronomically large that a human brain literally fatigues doing that work and can't reason across all those combinatorial possibilities.

Alex: He was a Hungarian refugee, heavily fuelled on methamphetamine for at least part of his career, just travelled the US with a suitcase, finding interesting people to work on maths problems with. The FBI was following him because they thought he was a spy, and he wasn't, he was just a nerd, just bumbling around doing maths. He has these incredible, in fact, six degrees of Kevin Bacon, they called it six degrees of Erdos, because he has all these connections, you can usually find a short path to him. Random aside, but his conjectures often relate to how to efficiently count large classes of objects, or how big can classes of objects get, what are the bounds. These are things humans are famously really bad at.

Will: Yeah, and the world is built based on the things that we were good at.

Alex: And kind of to tie this together, I think what we're going to see, and I'm kind of cribbing from Terry Tao here, is we're going to see a lot of the gaps that no one really explored.

Jim: I think that's going to be the same as what we were talking about last week. As we evolved over thousands of years, we never had to go down that path because we found a better way first. What if the path is a bit more rough?

Alex: I think we should keep in mind, it's not the first time we've used information technology for this, there have been maths-assisted proofs since the 80s. For the last 15 years there's been some incredible results in science and engineering using large computers, we generated that photo of the black hole by running simulations and looking at the light paths coming back at us, that's incredible. Then look at AlphaFold, AlphaFold is a great example. But the fact that we can do it so much more flexibly now in our native domain of natural language, I think that's going to be a good deal. But again, these are the sorts of stories that just get lost in the ether, because this is a really cool thing, sure, there's some old grey beard professors that are a bit upset about the way it's not on a blackboard, it's not "legitimate," but it has now stopped them wasting their time.

Jim: I don't think any mathematician would be upset that the Jacobian conjecture or any other Erdos conjectures turn out to be false, I think they're annoyed that they don't have a compelling reason why it's false.

Alex: But I also think, if I can tell you where you're going, even if I don't necessarily give you a map, all of a sudden it's much easier for you to find your way there, rather than just being like, okay, find a random place in Brisbane.

Will: Well, everyone on the theoretical side will be upset and everyone on the applied side will just be grateful.

Alex: There's no such thing as an applied mathematician, they're called physicists.

Will: Let's move into a lightning round now. We've got a couple of topics here. Alibaba's kind of releasing at least a preview, a Qwen 3.8 Max. This is not yet available, they haven't released benchmarks yet, I don't think, as of when I last checked, haven't seen a model card. 2.4 trillion parameter multimodal for the first time coming out of Alibaba, and they're saying it has beaten every other model except Fable on their benchmarks. Again, Jim.

Jim: Well, again, very exciting, love, love, or in the zone of the frontier, but it came very quickly after the Kimi K3 announcement, and it reads a little bit of, hey, we have one too, we have one too.

Alex: Yeah, I'm going to say bring on the pricing wars and let's see some efficiency gains here. I can't wait until open weights are a competitive advantage in the market and people offer either licensed open weights or quasi-open weights, or some equivalent, as a price of entry to the market.

Jim: Particularly talking open weights and price, I was talking briefly during the week about people tweaking an open weighted model, and so then giving a little bit more value so that they can charge a little bit more for that model, which I thought was really interesting and I hadn't thought about before. And again, it sort of opens up another source of discussion.

Will: I've seen a boom in new businesses starting that are doing exactly what the startups of the last three years are doing, but now with open weight models and their own ownership of their own data, and I think it's a fantastic step forward for this space.

Alex: For sure. No, no, bring it on, Alibaba.

Will: Okay. Substack, our favourite blog posting platform, has released an AI writing detector, or at least they've integrated one, I don't remember who it's from.

Jim: Did you?

Will: Something that begins with P, right? And so you can click a button now and see how much of it was written by AI. Jim, you happy about this one?

Jim: Well, A, I don't care if it's written by AI, and B, if you're not utilising AI as a tool to help you write an article in this day and age, what are you doing, is it an ego thing? I don't understand the motivation to say, oh, well, I don't need AI, okay, great, that's fantastic.

Alex: I do, because I suck at writing.

Jim: But it's also the way I look at it, is there's so much other work around it, because if you just went, hey, ChatGPT, write me a blog on this, you're not getting followers or subscribers on Substack any more, right? So it's not going to be of quality. So in order to write something of quality, you have to do so much work around it, and if you've done all this work and then used AI to bring it together and polish it, you're then going to show up in the AI detector as, oh, and so then it's as if there's something bad on it. And to me, that's where I found it quite interesting, not only from, because I don't care if it's written by AI, but as a business approach for Substack.

Will: Content on their platform, and now all of a sudden you're saying, oh, well, the bulk of the content coming on here is bad. I wonder if they had some internal people advocating for banning AI content and this is the middle ground they landed on. There is a huge amount of writers who are very against this technology, saying it undermines their livelihood.

Jim: I wish the reflexive answer wasn't to try and ban it or condemn the thing, figure out how to work with it. I just don't know why you'd label it as bad, because, like, one of the newsletters I read daily, it's abundantly clear that the whole thing is just getting the best things from Twitter from the last 24 hours.

Alex: What about ghostwriting? All these people with newsletters that have been published three times a week while being a full-time VC and doing all this other stuff, hell, are you writing them yourself, is that okay, publishing under someone else's name? Why aren't we up in arms about that?

Jim: We're breathing human. I actually think this is going to go away, the button, as you publish an article on Substack, there's a "check it for AI" button, I'm very sure that's going to disappear, because I know it was written by AI, because I copy pasted it, I don't actually want to read something that I know is a hundred percent human, because there is an art behind that, you can feel it in the words that are written, in the way it's crafted, and there's joy that comes from that.

Will: Me personally, I'm often reading for educational purposes, and if AI wrote it, I don't care as long as the facts are right. It's the same as I don't want to watch a carbon copy Netflix movie that has clearly been written by AI, because every single one of them is exactly the same and terrible.

Alex: But that's, I'm doing that for entertainment. So it's like, when you're in the ninety percent, in the ten percent of situations, you're reading for joy, and so therefore you want it to be something of quality. You want it to be something that, and not that AI isn't quality, but I also go to those really good authors and writers, why aren't you, why haven't you got a corpus of everything you've ever written, why aren't you amplifying and training a style, and then AI can do a whole lot of the heavy lifting for you, and then all of a sudden you're putting out more.

Will: I would imagine because they come from a completely different ideological basis than you, Jim. Okay, last topic of the day. OpenRouter is in talks with Stripe to be acquired for ten billion dollars, which is roughly eight times what their valuation was in May. These boys are timing it perfectly.

Jim: Again, is it Ramp? It happened the same week that Ramp came out and said they're going to create a rails product. They've done the heavy lifting, they've done the work, they showed everyone this is what everyone needs, everyone wants choice of model, everyone wants to be able to have the one infrastructure. Look at it and take the money, sure, they could build something better and build it into a higher valuation over the next five, ten years.

Alex: Take the money and run. Yeah, take the money and run for the OpenRouter guys. I'm not sure what Stripe...

Jim: more sense, don't you get all of that for the same price? But there's a different argument, because it just opens up other markets to them, helps them compete with companies. There's a lot of work you've got to do to get them all to work in harmony, and so that's sort of a lot of work in the future to get OpenRouter to a much higher valuation. If you've got a ten billion on the table now, take it.

Will: Yeah, my take on OpenRouter, I don't quite get it, and I don't quite get it for Stripe either, so yeah, take the money and run in my opinion. When OpenRouter first came out, I was at the same time similarly trying to build a bit of a search engine that, when you say, I don't think I was looking at Anthropic at the time, and chat to your team, can I give you three or four LLMs and give you the results all at once, with this perspective that AI will get much cheaper over time, was kind of my thinking, and therefore why not compare different results and give the answer. And I looked at OpenRouter and just thought, it doesn't seem like it would be hard to build another OpenRouter, especially internally, if you're a company like Stripe.

Alex: I find it valuable when I use it, but there are a couple of counterexamples out there at the moment, this is just the one everyone's globbed onto. Probably not ten billion dollars.

Jim: Thirty million?

Alex: Actually, no, because the user base is quite sticky, because they're all Python and JavaScript people who don't know anything about how to actually construct safe APIs, these are the LLM bros, the sort of early vibe code market.

Will: The one thing I'll say I'm happy about is that this isn't an AI company looking to buy these guys, that is the surprise, that there's a lot of other, it's not that difficult to create all these other reasons, it's still a great reason why OpenRouter should take the money and run.

Jim: Agreed.

Will: Agreed. Okay, well, that's a wrap on week four of Business Idiots. Who's your business idiot this week? I mean, it's just sitting right there now, we've said maybe it's Stripe with this one, but no, I can't really say that without knowing exactly what their angle is, it could be something far more strategic than what I'm really seeing there.

Alex: I'm going to say the Hugging Face security team, because, oh, who's the manager of that team, he can get the slap on the back, oh, just put it into Fable and see what it says, guys, just rotate the password.

Will: Actually, just roll back to me for a second, the Substack new AI writing detection tool apparently flagged a lot of Substack's own staff's blog posts.

Jim: Can I change my vote? I was going to suggest the US government screwing around, oh, well, how are we going to regulate AI and all this sort of stuff, meanwhile the president of China is just going out and signing a deal with 30 other countries.

Will: That was going to be my suggestion, the US government, but now I'm on Substack as well.

Alex: Sorry, Substack, we love you, but at the same time, great own goal.

Will: Realise what your product is.