Is there a future in light-powered AI chips?

The growing compute power necessary to train sophisticated AI models such as OpenAI’s ChatGPT might eventually run up against a wall with mainstream chip technologies.

In a 2019 analysis, OpenAI found that from 1959 to 2012, the amount of power used to train AI models doubled every two years, and that the power usage began rising seven times faster after 2012.

It’s already causing strain. Microsoft is reportedly facing an internal shortage of the server hardware needed to run its AI, and the scarcity is driving prices up. CNBC, speaking to analysts and technologists, estimates the current cost of training a ChatGPT-like model from scratch to be over $4 million.

One solution to the AI training dilemma that’s been proposed is photonic chips, which use light to send signals rather than the electricity that conventional processors use. Photonic chips could in theory lead to higher training performance because light produces less heat than electricity, can travel faster and is far less susceptible to changes in temperature and electromagnetic fields.

“One important insight for us has been that, for data center applications — and this is where most people think photonic AI compute would likely be applied — nobody is interested in inference or training alone,” said Christian Patze, the investment director at M Ventures, over email. M Ventures backed Celestial AI, one of the companies developing light-based AI accelerator chips. “A new technology, be it photonic or electronic, has to be able to work equally well for both applications, otherwise it’s not of interest for large customers in the space.”

Inference — that is, running trained models — is fairly expensive. According to one estimate, ChatGPT was costing OpenAI $100,000 per day at its peak, which works out to an eye-watering $3 million per month.

But on a more fundamental level, Patze isn’t convinced that “photonic compute” approaches are the best possible option for training. Photonic chips are physically larger than their electronic counterparts and difficult to mass-produce, for one, owing to the immaturity of photonic chip fabrication plants. Moreover, photonic architectures still largely rely on electronic control circuits, which can create bottlenecks.

“Where it concerns photonic chips, it very much depends on what the algorithm and workload is for the training and how those algorithms can be mapped onto photonic compute architectures,” Patze said. “People missed that AI compute is a lot more than doing [certain operations] efficiently and that mapping advanced AI models onto theoretically fast optical compute engines is not trivial — or rather very hard.”

The other issue with photonics chip tech, according to Patze, is the power required to convert data into a format that the chips can work with. With conventional, silicon-based hardware, data is transferred and stored in the digital domain, whereas in photonics, it’s analog. Converting between digital and analog necessitates dedicated converter components, which tend to be power hungry.

There’s yet another unresolved challenge in photonics: signal regeneration, or the process of regenerating optical signals degraded during transmission through photonic chips. Most chips perform signal regeneration through “optical to electrical to optical” conversion, in which a weak and distorted optical signal is detected and restored in a digital, electronic piece of hardware and then retransmitted to the optical domain. But this causes the signal to become distorted over time, leading to issues downstream.

For these reasons and others, Gartner analyst Anushree Verma, who studies emerging technologies and trends, believes that it’ll take more than eight years for photonic computing to become mainstream.

“If we think of an all-optical computing system, it needs a variety of components such as optical gates, optical switches, optical memories, optical interconnects and light sources,” Verma said. “Commercial availability of the critical components for photonic technology is still very far away for various reasons … The customer concentration as well as the technological challenges are not particularly hospitable to entrants.”

So is there a future in photonics for AI yet? Probably. It might just require a reframing.

Patze sees photonics helping to overcome one of the big bottlenecks in AI compute: delivering speedy access to large amounts of memory. For large AI models like OpenAI’s GPT-4, AI accelerator cards — think chips like Google’s TPUs or AWS’ Trainium — in server racks usually have tens or hundreds of cores and hundreds of gigabytes of memory that need to be interlinked. When strained with data, these classical electrical interconnects start to fail or become quite slow. But photonic interconnects don’t; they have far more capacity.

“Longer-distance connections at high bandwidth and low latency is exactly where photonic interconnects shine,” Patze said. “Hence, it’s not surprising that some companies are turning to those to overcome the data movement bottleneck in modern AI compute. The photonics interconnect approach also has much shorter development timelines, as photonic interconnect components and circuit design in principle are well known from the optical communication industry.”

Verma sees the photonics AI accelerator chip industry evolving in three stages over the next several years. First, she expects a wave of hybrid connectivity — silicon and photonics packaged together — followed by a flood of integrated platforms (the second phase), and then eventually full stack photonic computing as the third stage.

By 2027, Verma thinks that 10% of network switch deployments will have co-packaging optics, driven by the need for increasing bandwidth and reducing power consumption, up from a negligible percentage now.

“The primary challenge is the manufacturing cost reduction,” Verma said, “but the market will be driven by the need of increasing bandwidth and reducing power consumption, up from negligible now … Photonic computing is still in its very early stage.”

To her point, Mordor Intelligence predicts that the photonic integrated circuit market will be worth $26.42 billion by 2027.