The embedded world is an open source world. So, the world is used to Linux, it's used to computer vision platforms like OpenCL, OpenCV. And from an ML framework, they want to retain that same open source access.
The edge market is going to be the next AI gold rush. And if you really think about the edge market, it's 40,000, 50,000 customers globally. So it's diffused in its footprint.
But if you aggregate that, it's a much bigger market than the cloud.
By popular demand, Netsuite has extended its one-of-a-kind flexible financing program for a few more weeks. Head to netsuite.com/eye On AI. That's Eye On AI, E-Y-E-O-N-A-I, all run together.
Again, head to netsuite.com/eye On AI. netsuite.com/eye On AI, for its one-of-a-kind flexible financing program.
Craig, pleasure again. And so I took a long road to really come in to CMA, about 30 plus years in the industry. A background in software, silicon, and then I managed fairly large businesses.
In the last 10 years, I was at a company called Xilinx. I was there for 18 long years. I was their executive vice president, general manager when I stepped out.
And I was pretty passionate about watching what's happening in AI. And clearly, I think AI, even then, five years ago, took off in the cloud. And obviously in the consumer mobile experience as well.
And what was fascinating for me was that I think the physical world that we live in, the industrial world we live in, robotics, industrial automation, medical, automotive, we're still fairly using archaic old techniques and we're re-embracing AI. And I wanted to start a company that would really build a purpose-built platform to scale AI at what I call the embedded edge market. So that's the history and the backdrop on how we started, what led me to start SEMA, and now we're five and a half years into it.
And SEMA is designing chips for the edge, is that right?
Correct. We built our purpose-built chips for the edge. We also make the associated development software.
So our application developers or our customers can write their application and port that application through our software onto our chip.
Yeah. Just one question which I may or may not leave in. I had Dave Patterson on the podcast quite a while ago, and he was talking about instruction sets and creating an open-source instruction set for chip design.
Is when you talk about the programming environment, are you talking about the instruction set or how does the instruction set figure into that?
Okay. It's a simple question, but a very complicated answer, and I'll do my utmost. So you should think of our chip.
As a heterogeneous compute platform. So we have our own proprietary ML accelerator. We have an ARM processor complex, which is our control plane processor, and we have a DSP vector engine that we have licensed from Synopsys.
So those three make up the various compute elements we have on our chip. So that's number one. Number two, unlike the cloud where ML acceleration is the only problem.
In our world, it's application acceleration. So you need to accelerate not just the ML workload, but also the computer vision workload, the pre-processing, the post-processing, the control plane analytics, and the decision-making all needs to be on a single chip. So perhaps what Dave Patterson was referring to is that ARM has its own instruction set to program their chip.
He, I think, is driving a RISC-V platform.
That's correct.
Which is creating a open-source instruction set so that anybody could have... You're democratizing, if you will, the access for compute. And I think one day, no doubts, RISC-V will be a very, very popular, if not a parallel path or an opportunity for many designers.
In the space that we are in, the embedded market, the incumbent solution provider or lead solution providers are. And what's more important here is not just the instruction set alone, but the ecosystem of software vendors that surround that platform. Unlike other companies which may create a closed platform where they control the entire software stack, we create an open platform for anybody to be able to build applications on top of us and also leverage the software ecosystem.
So in that aspect, we found that finding Arm as a partner is a better way to go for who we are as a company. And I think they've been a fantastic partner. And so we use their instruction set for the control plane aspect of it.
But our secret sauce is really in the ML portion of it. But what we enable is what we call a product called a MLSOC, a machine learning system on a chip. And we allow people the opportunity to partition the problem in any of the heterogeneous compute elements.
Yeah. And when you say to open it up, I mean, when you were referring to the proprietary software development environment around a chip, I assume you're referring to Nvidia's CUDA.
Yeah.
Correct. And in what way is yours different? I mean, not different technically, but in terms of it being more flexible or more open.
Correct. So really good question. So let me try to stay at the high level and try, let's see if I can get my point across.
The embedded world is an open source world. So the world is used to Linux. It's used to computer vision platforms like OpenCL, OpenCV.
And from an ML framework, they want to retain that same open source access. So they want to be able to work with PyTorch or TensorFlow or Onyxx or any variant of an ML framework. So what we have done quite different than Nvidia from that perspective is we have built something made for the embedded market.
Everything we do touches open source. And so we kind of, I mean, if we're abstract, think of us as the Ellis Island of everything. So you could give us your poor, you're tired, you're hungry.
And once they take, once we ingest their application, then we convert them to a proprietary environment to deliver the performance and the power that they desire. So we are very open source and we are not limiting. CUDA was not built for the embedded market.
So to a certain degree, people are having to transform their environment to fit into an Nvidia world or CUDA world. So we do not force the transition. They can remain in the application and the environment they've been used to.
So that's a large difference that we go to. But obviously the combination of the hardware and the software, CUDA and their silicon, similarly, our version of our software is called Palette. Palette is a combination of MLSOC, Dolores, a world-class performance and power, and ease of use parallel.
I see. What you mean is people can use other platforms than Palette, or Palette works with Linux and other open-source systems?
The latter. So Palette is our entry. So anybody can bring in a thing, and they come to work with us through Palette.
Okay. And I'm sorry, I jumped into those questions that have been nagging me. But let's back up and talk about the evolution of chip architecture.
You know, from the Von Neumann architecture and CPUs and the rise of GPUs, and which was a kind of accelerator, and then the kind of split in, you know, as Moore's Law slowed, then people's, instead of trying to miniaturize transistors further, they started creating specific chips called accelerators, and then now we're moving to the edge. So can you just kind of walk us through that evolution?
Absolutely. And I actually think you did a great job already in walking through the progression of how, I think, compute architectures of a wall. I would still say at the underlying element of it, until we got into ML, everybody's architecture leveraged one alignment and some simple load store instruction.
You load and you store data and you compute, right? So the fundamental architecture hasn't really shifted. And combination of Moore's law slowing, but also the performance needs being exacerbated.
And power being a limiter has forced an hour, has created an opportunity for innovation. And a plethora of new approaches are still being brought together. GPUs have kind of taken pole position on solving AI ML.
But from our perspective, there's no one size that fits all. And no doubts, they've done a fantastic job. And in reality, Nvidia's strength is not just their GPU architecture, but more importantly, CUDA, right?
And they've done a fantastic job in building that as a mode to really navigate and protect their growth and opportunities. And I admire what they've really done. I think for the edge market, hard limiters power.
So things in the edge market cannot support a GPU like power consumption. So your world is really very simple and five watts, ten watts or 20 watts. And anything more than that is untenable from a system power performance or from a thermal heat dissipation perspective.
So you got to kind of think of it the other way in that how much compute can I get for a given power budget? That forces a radically different approach and an opportunity for people to think very differently, like we have. And what we really came up with is saying, hey, it's not ML acceleration alone or AI acceleration alone.
We need to think about application acceleration. And that's why we created a heterogeneous computer platform. And if you really dig into our ML architecture, it's a tile array of 10 by 10.
And we fundamentally stream applications to run through the tile array and efficiently utilize the compute that we have on our ML array. And so that's our approach that we have taken. And from our perspective, that's the best we could figure out to solve for the needs of the edge market.
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
no subject
Date: 2024-09-05 21:37 (UTC)The edge market is going to be the next AI gold rush. And if you really think about the edge market, it's 40,000, 50,000 customers globally. So it's diffused in its footprint.
But if you aggregate that, it's a much bigger market than the cloud.
By popular demand, Netsuite has extended its one-of-a-kind flexible financing program for a few more weeks. Head to netsuite.com/eye On AI. That's Eye On AI, E-Y-E-O-N-A-I, all run together.
Again, head to netsuite.com/eye On AI. netsuite.com/eye On AI, for its one-of-a-kind flexible financing program.
Craig, pleasure again. And so I took a long road to really come in to CMA, about 30 plus years in the industry. A background in software, silicon, and then I managed fairly large businesses.
In the last 10 years, I was at a company called Xilinx. I was there for 18 long years. I was their executive vice president, general manager when I stepped out.
And I was pretty passionate about watching what's happening in AI. And clearly, I think AI, even then, five years ago, took off in the cloud. And obviously in the consumer mobile experience as well.
And what was fascinating for me was that I think the physical world that we live in, the industrial world we live in, robotics, industrial automation, medical, automotive, we're still fairly using archaic old techniques and we're re-embracing AI. And I wanted to start a company that would really build a purpose-built platform to scale AI at what I call the embedded edge market. So that's the history and the backdrop on how we started, what led me to start SEMA, and now we're five and a half years into it.
And SEMA is designing chips for the edge, is that right?
Correct. We built our purpose-built chips for the edge. We also make the associated development software.
So our application developers or our customers can write their application and port that application through our software onto our chip.
Yeah. Just one question which I may or may not leave in. I had Dave Patterson on the podcast quite a while ago, and he was talking about instruction sets and creating an open-source instruction set for chip design.
Is when you talk about the programming environment, are you talking about the instruction set or how does the instruction set figure into that?
Okay. It's a simple question, but a very complicated answer, and I'll do my utmost. So you should think of our chip.
As a heterogeneous compute platform. So we have our own proprietary ML accelerator. We have an ARM processor complex, which is our control plane processor, and we have a DSP vector engine that we have licensed from Synopsys.
So those three make up the various compute elements we have on our chip. So that's number one. Number two, unlike the cloud where ML acceleration is the only problem.
In our world, it's application acceleration. So you need to accelerate not just the ML workload, but also the computer vision workload, the pre-processing, the post-processing, the control plane analytics, and the decision-making all needs to be on a single chip. So perhaps what Dave Patterson was referring to is that ARM has its own instruction set to program their chip.
He, I think, is driving a RISC-V platform.
That's correct.
Which is creating a open-source instruction set so that anybody could have... You're democratizing, if you will, the access for compute. And I think one day, no doubts, RISC-V will be a very, very popular, if not a parallel path or an opportunity for many designers.
In the space that we are in, the embedded market, the incumbent solution provider or lead solution providers are. And what's more important here is not just the instruction set alone, but the ecosystem of software vendors that surround that platform. Unlike other companies which may create a closed platform where they control the entire software stack, we create an open platform for anybody to be able to build applications on top of us and also leverage the software ecosystem.
So in that aspect, we found that finding Arm as a partner is a better way to go for who we are as a company. And I think they've been a fantastic partner. And so we use their instruction set for the control plane aspect of it.
But our secret sauce is really in the ML portion of it. But what we enable is what we call a product called a MLSOC, a machine learning system on a chip. And we allow people the opportunity to partition the problem in any of the heterogeneous compute elements.
Yeah. And when you say to open it up, I mean, when you were referring to the proprietary software development environment around a chip, I assume you're referring to Nvidia's CUDA.
Yeah.
Correct. And in what way is yours different? I mean, not different technically, but in terms of it being more flexible or more open.
Correct. So really good question. So let me try to stay at the high level and try, let's see if I can get my point across.
The embedded world is an open source world. So the world is used to Linux. It's used to computer vision platforms like OpenCL, OpenCV.
And from an ML framework, they want to retain that same open source access. So they want to be able to work with PyTorch or TensorFlow or Onyxx or any variant of an ML framework. So what we have done quite different than Nvidia from that perspective is we have built something made for the embedded market.
Everything we do touches open source. And so we kind of, I mean, if we're abstract, think of us as the Ellis Island of everything. So you could give us your poor, you're tired, you're hungry.
And once they take, once we ingest their application, then we convert them to a proprietary environment to deliver the performance and the power that they desire. So we are very open source and we are not limiting. CUDA was not built for the embedded market.
So to a certain degree, people are having to transform their environment to fit into an Nvidia world or CUDA world. So we do not force the transition. They can remain in the application and the environment they've been used to.
So that's a large difference that we go to. But obviously the combination of the hardware and the software, CUDA and their silicon, similarly, our version of our software is called Palette. Palette is a combination of MLSOC, Dolores, a world-class performance and power, and ease of use parallel.
I see. What you mean is people can use other platforms than Palette, or Palette works with Linux and other open-source systems?
The latter. So Palette is our entry. So anybody can bring in a thing, and they come to work with us through Palette.
Okay. And I'm sorry, I jumped into those questions that have been nagging me. But let's back up and talk about the evolution of chip architecture.
You know, from the Von Neumann architecture and CPUs and the rise of GPUs, and which was a kind of accelerator, and then the kind of split in, you know, as Moore's Law slowed, then people's, instead of trying to miniaturize transistors further, they started creating specific chips called accelerators, and then now we're moving to the edge. So can you just kind of walk us through that evolution?
Absolutely. And I actually think you did a great job already in walking through the progression of how, I think, compute architectures of a wall. I would still say at the underlying element of it, until we got into ML, everybody's architecture leveraged one alignment and some simple load store instruction.
You load and you store data and you compute, right? So the fundamental architecture hasn't really shifted. And combination of Moore's law slowing, but also the performance needs being exacerbated.
And power being a limiter has forced an hour, has created an opportunity for innovation. And a plethora of new approaches are still being brought together. GPUs have kind of taken pole position on solving AI ML.
But from our perspective, there's no one size that fits all. And no doubts, they've done a fantastic job. And in reality, Nvidia's strength is not just their GPU architecture, but more importantly, CUDA, right?
And they've done a fantastic job in building that as a mode to really navigate and protect their growth and opportunities. And I admire what they've really done. I think for the edge market, hard limiters power.
So things in the edge market cannot support a GPU like power consumption. So your world is really very simple and five watts, ten watts or 20 watts. And anything more than that is untenable from a system power performance or from a thermal heat dissipation perspective.
So you got to kind of think of it the other way in that how much compute can I get for a given power budget? That forces a radically different approach and an opportunity for people to think very differently, like we have. And what we really came up with is saying, hey, it's not ML acceleration alone or AI acceleration alone.
We need to think about application acceleration. And that's why we created a heterogeneous computer platform. And if you really dig into our ML architecture, it's a tile array of 10 by 10.
And we fundamentally stream applications to run through the tile array and efficiently utilize the compute that we have on our ML array. And so that's our approach that we have taken. And from our perspective, that's the best we could figure out to solve for the needs of the edge market.