“I cannot say that there is a divine truth of the universe that explains why matrix multiplies seem to be the atomic unit of computation, but one way I’ve heard it described to me is, ‘Well, it’s a really succinct way to mix two blocks of numbers together and have them interact in some interesting way.’ It is really convenient that linear algebra turns out to be a very compact representation of arbitrary relationships and data.”

“The speed of light is the edge of what’s possible. If we think the chip can run at this frequency and produce this many multiplies per cycle, we’re gonna get there. To this day, I tell all my engineers, ‘We’re chasing a hundred percent speed of light.’ I don’t care about relative numbers versus the competition. I only care about absolute numbers.”

“Cerebras, Groq, and a couple others that are coming out of stealth now, I think have made a very interesting bet on not just building another GPU, but actually building a different kind of accelerator that focuses on a different memory hierarchy.”

“We’re not gonna limit ourselves to the eight hundred square millimetre reticle limit that TSMC imposes on us. We’re gonna take the entire wafer and have every die connect to every other die over scribe lines, and we’re just going to try to get as much SRAM as we can on the whole wafer.”

“You can move the entire parameter count of a large model like Kimi in and off the logic cores in about a millisecond. So there you go, you have a path to a thousand tokens per second.”

“Cerebras and Groq and maybe a couple others, you should think of them as accelerators. What they are really good at is being used in conjunction with a more traditional GPU-like device that critically has this off-chip memory built in.”

“Net new companies are great, such as Etched or SambaNova or dMatrix. All these companies are popping up. I think the main challenge for them is scale. Can they actually get enough wafer allocation from TSMC to pump out chips to make an impact on the market?”

“I think we’re probably one of the best teams in the world to use multiple silicon architectures and be quite aggressive in chasing down performance in unlikely places.”

“Any fab has a lot of spread in their worst chip that comes out of the production line and the best chip that comes out of the production line. TSMC, they work very, very hard to tighten what we call these process corners. But that means that they are adding a lot of controls in the process that maybe I don’t need. Maybe I’m actually willing to find a place for that worst chip.”

“If you look at Hopper to Blackwell to Rubin, and you compare like for like, what is the performance per watt of B float sixteen multiply? It hasn’t improved all that much.”

“People lose their minds over geopolitics, like what would happen if we lost access to TSMC for any reason. My contrarian take is that it wouldn’t be that bad. Supply would take a shock for sure, but the best processes that we have in the West, like Intel, not that far behind, at worst maybe two X worse performance per watt. The gap is just far smaller than you would make it out to be if you follow the chip war dialogue.”

“It’s all about the bottlenecks on supply chain. There’s TSMC wafer capacity, there’s HBM capacity, and there’s advanced packaging. Maybe a fourth one would be power. You should have a great answer to each of those four bottlenecks and how you’re gonna work around them, because it’s all arbitrage at the end of the day.”

“Nvidia makes a lot of choices that are difficult for them to change. They’re not perfect. They’re just really well-balanced. You wanna be spiky.”

“Cerebras, Groq, and a couple others that are coming out of stealth now, I think have made a very interesting bet on not just building another GPU, but actually building a different kind of accelerator that focuses on a different memory hierarchy. They want to maximize the amount of SRAM on the chip and use that as very, very fast memory.”

“Because it’s so physically close to the logic gates that actually do the computation, the arithmetic logic units are right next to the SRAM that they’re gonna pull from. The compute units that are doing the matrix multiplies can pull data from SRAM at mind-boggling speeds.”

“Blackwell has two hundred and eighty-eight gigabytes of HBM capacity around the logic die, and the logic die itself maybe only has like five hundred megabytes of SRAM. So it’s possibly three orders of magnitude difference in density for DRAM versus SRAM.”

“Cerebras quotes twenty-one petabytes per second for their Wafer Scale Engine 3. Compare that to HBM on an NVIDIA Blackwell, ten terabytes per second or so. More capacity, but proportionally less bandwidth.”

“As soon as you write that charge into that capacitor, the charge is dissipating. The dynamic part of DRAM is that you must, every fifty milliseconds or so, refresh every bit you’ve written. So you’re constantly juggling billions of balls in the air, essentially, billions of bits that have to be managed by a memory controller.”

“You have this crystallized knowledge in the model weights, and you have the dynamic knowledge of the exact conversation we’re having in the KB cache, is the way I like to think about it.”

“During training, the model did not get trained primarily on very long context conversations. It got trained primarily on, let’s say, eight thousand token conversations. So there’s always been a challenge for the frontier labs to figure out, how do we make the model exactly as intelligent at ten thousand tokens as we expect it to be at two hundred thousand tokens?”

“You want off-chip memory for capacity and on-chip memory for speed. We want to hybridize these two things.”

“I would say the original sin of transformers is that you’ve taken this extremely fundamentally memory-bound layer and juxtaposed it right next to a compute-bound layer. It is very difficult to have a single chip that is good at both compute operations and memory operations.”

“You would like to put the attention possibly on the GPU and the MLP on the Cerebras chip.”

“The KV cache is quite uncompressed right now. I think if you look at the entropy in a KV cache, it’s not earning its keep. Like, we’re storing many kilobytes of data in the KV cache per token, and that’s probably off by an order of magnitude or two.”

“It’s about sidestepping the HBM shortage and focusing on more extreme offload to other forms of memory, such as Flash. Everyone on my team knows that I keep banging the drum around, like, what would we have to change about the model architecture to make offloading KB cache to Flash work at a much greater level?”

“There’s no easy way to bring on a lot more fabs of memory.”

“The GPU is fundamentally a throughput machine. The GPU’s happiest when you give it a lot of work to do and let it chew through that work at peak utilization of its compute units.”

“We’ve pushed AI to be an interactive chatbot tool, is the most common form of AI usage today. It’s very difficult to put the GPU in its happy path of being fully compute utilized when you’re trying to spit out tokens quickly.”

“There’s a fundamental trade-off on the GPU between being throughput-oriented or latency-optimized, and everyone has chosen latency optimization because the shape of usage was chatbot-oriented. I believe that’s the most profound change we’re going to see in the next year. We’re gonna move away from chatbots to more proactive or background agents.”

“Narrow and fast or wide and slow is like a classic trade-off in all computer science.”

“We want to group many users’ work together into a batch that we can run all at once on the GPU. The thing is, though, you’re doing net more work when you run a large batch of compute together. Any individual user’s request in that batch is going to spend a longer time on the GPU being carried with other people’s traffic.”

“If you want to get downtown in SF, you can take the bus or you can take a private transit. A bus, it’s gonna have to serve many more people, and it has to fundamentally do something that works for everyone, so it takes a slower path, and it stops and waits for other people to get on and off.”

“If you have a large matrix multiply that you want to perform faster, you can actually cut that matrix multiply in half and shard it across two or more, up to eight, NVIDIA GPUs, and have them all work on pieces of that larger matrix multiply.”

“It’s sublinear scaling. You’ll use eight times more hardware, but you won’t get eight times the speed. You might get, like, four to five X the speed. This is because of communication overhead.”

“NVLink is mandatory for low-latency inference. So NVIDIA is excellent at low-latency inference, and I’m telling you that we don’t really care that much about low-latency inference, so where does that leave us?”

“Maybe there’s room for that other chip in my stack as a really, really good compute per dollar option. That’s what I actually optimize for in most cases, is how many flops does this chip have, and how much is it gonna cost me per hour to own and operate?”

“I like to say we write kernels on the whiteboard. We go to the whiteboard, we describe what we think the machine should be doing, then we succinctly describe that in natural language to a model, and then the model is able to do the execution.”

“The rising tide of something like Mythos or GPT 5.6 Sol, that lifts all boats. It really does. Broadly speaking, we’re all downstream of the frontier in terms of this capability.”

“That operation runs at seventy, eighty percent of peak utilization, and it’s limited not by software, but by power. The way NVIDIA quotes peak FLOPS is a little optimistic. You never hit that because of power throttling.”

“Our job is to basically build the engine around the chip such that we are feeding the GPU these large batches of work at all times.”

“You don’t program one GPU at a time anymore. You should think about the whole rack, and maybe you should think about the whole cluster, the entire data center at a time.”

“Their latest chip, the Grace Blackwell 300, ships as a rack of seventy-two units, and it’s an open race to figure out who can program the whole rack scale computer as efficiently as possible.”

“Transformers make no such human prior. Transformers just say, ‘Well, there’s gonna be a pattern in the sequence of data, and if there is a pattern, I’m gonna find it. I’m going to throw more and more parameters at this problem until it works.’”

“The biggest models were one hundred and fifty million parameters was a huge model for computer vision, and now we routinely talk about trillions of parameters. Transformers are the link to go from millions to trillions of parameters.”

“Transformers are such great sponges. You increase the compute available to a transformer by ten X, and you’ll get some log improvement somewhere, and so far, the scaling laws really work. They’re really quite beautiful.”

“Transformers represent any pairwise relationship that you want. Any token in the sequence can attend to any other token in the sequence. So if there’s any relationship that’s in the sequence at all, you’re gonna find it with a transformer.”

“One of the old Karpathy sayings is that if you have a new data set that you want to train a model for, your first goal should be to overparameterize the model and try to overfit the data that you have to prove that there is a relationship that you can model or memorize.”

“Once you can overfit, then you can compress, and the compression is how you get generalization.”

“I like the phrase that internet was a one-time subsidy on data. We got it for free.”

“There are about thirty trillion tokens of high-quality text, three hundred trillion tokens if you take a wider view on what qualifies as good text. And we’ve basically looked at it all already. Models have seen the entire internet many times over at this point.”

“The median model that we serve is so much more advanced than a random human giving feedback that the signal you get from unconditioned human preference is not actually worth anything anymore. You want expert human preference at this point.”

“More and more, we can just give the agent a computer, essentially, and have it act like it’s a human worker and give it feedback on whether it’s making progress towards the target outcome. That environment becomes the data.”

“If you want artificial general intelligence, the best way to get there is to just keep stacking specialized intelligences until you have no more gaps to fill. The only thing you need to make sure you do to make this work is you must make sure that your task is verifiable. You need to give the model a self-grading system. If you have that, you have the recipe for self-improvement on any task you like.”

“They used to spend that much on data. Now they spend a lot more on RL environments.”

“How consequential it is to decide to use something like sparse attention versus dense attention, or to use a different data type. We were training in B float sixteen, but now we can train in FP8 or FP4. That is just an arbitrary choice it feels like, but it has profound implications for what chips I can use.”

“The most conservative players in the entire AI stack have got to be the infra players, whether that’s data centers or even more conservative is TSMC, the chip infra people.”

“Training is a superset workload over inference. You can make any training cluster work for inference, but maybe not vice versa. The difference there is networking.”

“There’s actually a diseconomy of scale to data centers in some way. It’s way more expensive and difficult to build 100,000 GPUs in one data center than it is to build 10,000 than it is to build 1,000.”

“Basically, there’s no way to build a gigawatt data center in the United States easily anymore. Even 100 megawatts is increasingly hard. It’s basically impossible unless you’re a very special set of customers. 10 megawatts is probably on the edge of what’s possible today, and one megawatt I would argue is plentiful.”

“There’s this incredible lore on the market where you can find lots of aggregate power, but it will not be concentrated, and that was not interesting to anyone who’s building training data centers because you just assume it needs to all be in one spot.”

“Increasingly we’re seeing a few new thinkers realize that inference is gonna be suitable for these distributed one megawatt data centers. We’re very happy to buy small pools of compute across the United States and use that as our inference fleet.”

“Now you can actually pack that into around like eight racks worth of compute. Each rack is about the size of a refrigerator. You can just imagine eight of them lined up. That’s a megawatt.”

“You’re not gonna have power redundancy. Quite often, you’re not gonna have backup diesel generators on site. Those are all very expensive. We cut all that overhead. We’re not even gonna have redundant networking in a lot of cases.”

“We’re gonna trench one line of fiber to these data centers, but we’re not gonna have three lines of fiber with redundancy and failover and SLAs. It’s just gonna go down sometimes.”

“You’d have basically zero buyers for a data center that has 95% uptime. I’m that first buyer. I will buy 95% uptime.”

“The failures happen at some rate, and I am basically linearly happy with a data center that is 95% uptime versus 98% versus 99%. It’s just linearly good or bad for me.”

“We tell our customers, ‘Look, our average throughput is gonna be very competitive, but our P99, our ninety-nine percentile latency is not gonna be controlled. It cannot be. And in return, I’ll give you unbeatable economics.’”

“I’m a son of California. I love solar and wind. I think solar and wind power is way under-tapped in the United States, and the challenge has always been this intermittency.”

“I’ll just model the weather and figure out when my data centers are gonna be offline, move my workload somewhere else, and it’s fine. The trick is that it’s gonna give me better access to power that no one else is gonna touch because it is so annoying to deal with that kind of outage.”

“My whole goal is to so dramatically expand the supply of power across the United States that I have a home for a lot of chips that otherwise would not have earned their place in a data center.”

“We think that whenever you make something ten times cheaper, it’s a new product category. We aspire to do that for tokens.”

“Nvidia sees if they just let the most deep pockets buy all the chips, that maybe hurts them in the long term. They understand that compute is power today. They’re quite strategic about how they allocate compute.”

“Nobody wants to have a huge order of a chip rental come in from this new startup that says, ‘Oh, yeah, I’m gonna rent ten thousand Blackwells for three years.’ This startup has only been operating for months, typically. Who knows whether they’re good for the money?”

“I like to say there’s no bad chips, there’s only bad pricing. I will make any chip work at the right price.”

“There’s alpha and just other people have this perception that AMD is not as good as Nvidia. That’s music to my ears. I’m very happy for them to sleep on this chip and for me to buy as much as I can.”

“Every chip has a comparative advantage. We had to find that advantage and then squeeze it in that direction.”

“A lot of the investment in networking equipment historically was speculative. We anticipated this future demand for users that never came.”

“What’s interesting about token consumption or AI consumption broadly is that it’s no longer speculative. People buy tokens because they’re immediately valuable to them. You don’t hoard tokens. You use them immediately.”

“In that period it was all training-oriented spend, and training is inherently speculative. I do think inference spend monotonically increases. There’s no speculation on inference.”

“One of the ways that I describe what we do is we will buy any chip anywhere in the world for any duration of time. That is a level of flexibility and liquidity that no one else has right now.”

“First we scavenge chips, and then we scavenge power for those chips. The idea is, in both cases, I do not wanna be bidding against Anthropic or OpenAI for compute capacity. I’m not gonna win against them, and I don’t want to. I want to be more creative and use the supply that they don’t find legible today.”

“We are building a factory. We’re trying to build the best steel factory in the world, but it will come through mini mills, not through large monolithic steel plants.”

“I think of it as like I want to get to the scale where I earn the right to take this under our wing.”

“NVIDIA’s pumping out five million Blackwell chips this year. Where are they all going? Are they all being used at all all the time? I certainly doubt it.”

“A lot of the compute disappears into private pools of compute that will never see the light of day, and those GPUs sit very sadly idle. It pains me physically to see that.”

“In a line, I would say the labs pay an immense premium to be three to six months ahead of everything else. I think that’s probably still worth it.”

“An increasingly large percentage of the artifacts we put out on the internet are AI-generated. You just look at GitHub alone. What percentage of repos created in the last year do we think were created by Claude code? Do we consider that to be distillation? Because that’s probably all we need.”

“I don’t think it’s fundamentally possible to prevent the diffusion of information or model capabilities. It will happen. The question is just how fast.”

“Fundamentally, I don’t think open source ever goes away. If there’s a vacuum because one leader steps out, a new leader will step in. It gets easier every day to train a frontier-class model.”

“The boys in Boise don’t love huge CapEx for cyclical. They’ve been burned on that many times.”

“I think iPhones will cut their memory. iPhones are gonna go up in price, and we’re just gonna deal with it.”

“Nvidia is really smart about this. They don’t compete with their customers. Jensen is really good at making his friends billionaires. He’s made CoreWeave a many billion dollar company, and there’s no need for him to destroy that goodwill.”

“He wants to create a diverse community of Neo clouds and inference providers who are all jockeying to create demand for Nvidia, such that if any one of them decides to vertically integrate or go with AMD, he’s got three more people hungry to fill that position. It’s great to have competition amongst his buyers.”

“Sail Research is a token factory. We have an API where anyone can send us requests where they can use open source large language models for any task they want. We will serve those tokens to them at a price that is unbeatable in the market.”

“The theme of our company is abundance. We want to deliver this new commodity of intelligence to as many people as possible at a cost that is sustainable for almost every industry.”

“You think about agents that self-administer a token budget as opposed to a company setting a budget for how many tokens engineers can spend per month.”

“We are starting to see an increasing number of our customers and the broader market care about owning intelligence. They want to have control, sovereignty over the thing that they depend on.”

“Now it’s to me very obvious that the future of agentic inference is long-horizon tasks. You’re going to run the machine for hours or days at a time. It doesn’t matter if it spits out tokens at 100 tokens per second. Maybe 10 is just fine if that comes with corresponding advantages in efficiency.”

“When you’re waiting on it, you absolutely deserve the fastest answer possible. My trick is I don’t want you to be waiting on it. I want it to be proactive. I want it to be in the background. One way to say it is like the best latency is no latency at all. When you wake up in the morning, the work’s already been done overnight.”

“You don’t manage your colleagues every five minutes. You ask them to do a high-level task, and you come back and check in maybe every day, but more likely once a week. And that to me is the future of human-agent collaboration, more like human time scales.”

“I love this market because it’s unbounded. There’s no human in the loop, so you can consume as many tokens as you like in the background versus human attention span.”

“We’re gonna end this year at maybe fifty-fifty background and real-time workloads, but I see this going to ninety ten in favor of background.”

“Most questions where you want to have a definitive answer over not a hundred sources, not a thousand sources, but ten thousand sources or more. That is the kind of crazy exabyte scale task that you need a very different scale of intelligence to achieve.”

“At some point, people started to make this joke that security has become proof of work. When you want secure software, it’s really a question of how many dollars did you spend on Anthropic’s APIs trying to break into your software? That is the best indication for how secure it is because that’s the best tool in the world.”

“The frontier of intelligence here is quite jagged. You would find some bugs with a very small model that you don’t find with a large model.”

“You have to be willing to spend tokens without any promise of return. That is the unlock.”

“We have a form of intelligence that can tackle any verifiable problem. Any verifiable problem means most software, it means a lot of formal math proofs, and it could also mean scientific discovery.”

“It’s not millions, it’s thousands, and maybe it could be hundreds or even tens of dollars in the near future to have a definitive answer to any scientific question, to any research problem.”

“I put basically the entire category of human taste into that category. We have not solved human taste yet, and I don’t know that it fundamentally can be. We leave the quality of writing, we leave the beauty of art to people.”

“There is this larger sense around how do people consume a trillion tokens per day? That’s the world we want to create, the capability for them to do that.”

“I never will believe in that. There is always demand for intelligence in the world. The on-ramps to that intelligence are our challenge as a product community.”

“I want abundant tokens and diverse harnesses. I want everyone to build their own harness. Every company, every user even. Make the agent your own. I want people to own their intelligence.”

“My job is to make the tokens as cheap as humanly possible. I will achieve that, and I will do it through every layer in the stack available to me. I love the supply side levers. I will use every chip, I’ll use every source of power, and I will use every piece of land in the United States that’s suitable for this.”

“We still treat the agent as a person that is expensive to consult, and you should ask them when you have a hard question. That’s not the way to think about intelligence. It’s incredible that the machine can think, and we should try to get that into as many hands, as many people as possible.”

“It’s so rare that someone can actually understand the entire stack from the gate-level silicon all the way to building a great internet-scale service. You should aspire to be someone who, over the course of your lifetime, achieves that level of understanding. That level of expertise is so noble to chase.”