it seems the DOJ story was just a rumour 
1 Like
Ah interesting, so this would mean if Nvidia don’t stay constantly 2 steps ahead of an in house design team that mutli billion contact becomes harder to justify?
Is there a moores law senario going on here where development keeps going or are there cost / benefit limits?
1 Like
Yeah if Nvidia could make something so much better then it becomes worthwhile to pay the Nvidia tax. In the medium term I’m sure there will still be vast orders basically for that reason (e.g. Microsoft will buy huge numbers of GPUs, alongside Maia 100 and AMD) and outside of big tech plenty of companies can’t design their own chips.
However it’s pretty hard to make something better for everyone in the long run, because vertical integration is incredibly important here, when the question isn’t, ‘what’s the best chip?’ it’s ‘what’s the best 1GW data center?’ you can get a lot of different design considerations.
There are some fundamental limits of how good each individual chip is because the logic for ML is actually incredibly simple (compared to general purpose computing) - really it’s just multiplying vast matrices.
So lots of the innovation hasn’t been within the chip, it’s everything around it - interconnect (moving enough data fast enough to keep chips ‘fed’), cooling efficiency, power delivery, network latency etc…
Yesterday’s SemiAnalyis kind of indicates the scale of these other challenges and why the extra step of designing their own chips isn’t such a big deal for OA/Microsoft:
2 Likes
Really appreciate the reply and knowledge @Cameron thanks for this.
If were to draw an investing lesson from this then data centres would be the natural home for most AI compute to be done, which means either a price war or one of the big tech players will scoop up the contracts.
1 Like
retail investors pumping trillion dollar companies? hmmm…
nancy doesnt miss
Not saying it’s totally down to that, just a contributing factor.
The Dean of Valuation weighs in:
3 Likes
As a bizarre coincidence Hock Tan was basically asked the same question on the Broadcom earnings call yesterday:
Thank you very much. Hock, that was a perfect segue into my question. You’ve said in the past calls that you thought that AI compute would move away from ASICs and go to merchant market. But it looks like the trend is kind of heading the other way. Are you still the opinion that that’s going to be the long-term trend of this?
And secondly, as you just pointed out, power is becoming the defining factor for deployment with all the big guys at this point. And given the performance per watt of the ASICs over GPUs, which is superior to GPUs, why shouldn’t we see more of these guys moving to custom ASICs?
You’re right, and you’re correct in pointing out to me that, hey, I used to think that general purpose merchant silicon will win at the end of the day. Well, based on history of semiconductors mostly so far, general purpose merchant silicon tends to win. But like you, I flipped in my view. And I did that, by the way, last quarter, maybe even six months ago. But nonetheless, catching up is good.
And I actually think so, because I actually think there are two markets here on AI accelerators. There’s one market for enterprises of the world, and none of these enterprises are incapable nor have the financial resources or interest to create the silicon – the custom silicon, nor the large language models or the software and going maybe to be able to run those AI workloads on custom silicon. It’s too much, and there’s no reason for them to do it, because it’s just too expensive to do it.
But there are those few cloud guys, hyperscalers with the scale of the platform and the financial wherewithal for them to make it totally rational, economically rational, to create their own custom accelerators, because it’s all – right now, not going to – not trying to emphasize it, it’s all about compute engines. It’s all about especially training those large language models and enabling it on your platform. It’s all about constraint, to a large part, about GPUs.
Seriously, it came to a point where GPUs are more important than engineers – these hyperscalers in terms of how they think. Those GPUs are much more – or XPUs are much more important. And if that’s the case, what better thing to do than bring it under the control of their own destiny by creating your own custom silicon accelerators. And that’s what I’m seeing all of them do. It’s just doing it at different rates and do – and they’re starting at different times, but they all have started.
And obviously, it takes time to get there. But they’re all – a lot of them, there are a lot of learning in the process versus what the biggest guy of them who has done longest have been doing for seven years. Others are trying to catch up, and it takes time. I’m not saying they’ll take seven years. I think they’ll be accelerated, but it will still take some time step by the time to get there.
But those few hyperscalers platform guys will create their own if they haven’t already done it and start to train them on their large language models. And that is, yeah, you’re right; they will on go in that direction totally into ASIC or, as we call it, XPUs, custom silicon. Meanwhile, there’s still a market for in enterprise for merchant silicon.
3 Likes