The dictionary tells us that an inference is "a conclusion reached based on evidence and reasoning."
But that meaning takes on a whole new meaning in the world of artificial intelligence, where inference occurs when an AI model produces predictions or conclusions.
An AI model capable of making inferences can do so without examples of the desired result, according to cybersecurity company Cloudflare (NET) . "[In] other words, inference is an AI model in action," the company says.
An example of AI inference would be a self-driving car that can recognize a stop sign, even on a road it has never driven on before, the company said.
“AI inference is critical to the advancement of AI technologies and underpins its most exciting applications, such as generative AI, the capability that powers OpenAI’s ChatGPT application," IBM (IBM) said in a June 18 report. "AI models rely on AI inference to imitate the way people think, reason and respond to prompts."
While AI inference offers many benefits, the IBM study said that as a young, fast-growing technology, it was "not without its challenges, too.”
Hock Tan, CEO of Broadcom, is riding a wave of spending on artificial intelligence.
NurPhoto/Getty Images
OpenAI teams up with Broadcom on new AI chips
Among other issues, the tech giant noted compliance problems involving data sovereignty, the concept that data is subject to the laws of the country or region where it was generated, and a shortage of talent in the field.
IBM also noted a reliance on Taiwan, where 60% of the world’s semiconductors and 90% of its advanced chips, including the AI accelerators needed for AI inference, are made.
OpenAI, the company behind the popular AI chatbot ChatGPT, is reportedly working with semiconductor maker Broadcom (AVGO) to develop a new AI chip specifically focused on running AI models after they’ve been trained, Bloomberg reported on Oct. 29, citing people familiar with the matter.
The two companies are also consulting Taiwan Semiconductor (TSM) , the world’s largest contract chipmaker.
OpenAI has been planning a custom chip for AI technology for around a year, but the discussions are still at an early stage.
Nvidia (NVDA) has cornered the market on graphics processing units, or GPUs. GPUs are semiconductor chips used to train and build generative AI models. OpenAI is more focused on the inference process rather than competing with Nvidia.
Investors and analysts expect the need for chips to support inference to grow as more tech companies use AI models to perform more complex tasks.
OpenAI has assembled a team of roughly 20 chip engineers, including experts with experience designing Google’s Tensor processing units, which accelerate machine-learning workloads, Fortune reported.
Together, they hope to develop OpenAI’s first custom chip by 2026 — though that timeline is flexible.
Broadcom has certainly benefited from the AI explosion.
In September, the company, which is scheduled to report quarterly results next month, beat Wall Street's third-quarter-earnings expectations.
CEO Hock Tan said AI-chip sales were expected to reach $12 billion in fiscal 2024, up from a prior forecast of $11 billion.
"AI revenue continues to grow and grow strongly," Tan told analysts during the company's earnings call, adding that VMware bookings continue to accelerate and "non-AI semiconductor revenue has stabilized."
Analysts say Broadcom faces headwinds
Broadcom completed its $69 billion acquisition of the cloud computing firm VMware last year.
The company's shares are up 55% year-to-date and nearly 96% from a year ago.
Bank of America Securities analysts cited Broadcom's connection with OpenAI in a Nov. 5 research note.
The firm reiterated its buy rating and $215 price target but lowered its fiscal 2025 earnings expectations to reflect "seasonal headwinds" in the first half of the year, a product transition for Google’s TPUs, and the possibility of Wi-Fi socket changes at Apple (AAPL) .
Broadcom is an important supplier of chips that connect Apple devices to mobile data networks
However, partly offsetting these headwinds could be expanding custom-AI customers — such as OpenAI — and additional content opportunities at Apple, the investment firm said.
"Our recent company meeting and weak outlook from Apple and wireless peer Qorvo (QRVO) and Cirrus Logic (CRUS) reemphasized seasonal headwinds," the firm said.
Bank of America analysts said they also expected Broadcom's Google custom chip to take a one- to two-quarter pause ahead of the TPUv6 transition in the second half of fiscal 2025. Still, they also expect Broadcom's AI networking to grow solidly, complementary to ramping production of Nvidia's next-generation Blackwell GPU.
Alphabet's Google (GOOGL) unveiled the TPUv6 chip in May.
"Broadcom plans to move back to quarterly from annual guidance, starting next earnings call, which could put greater focus on noisy seasonal and lumpy AI shipment," BofA said.
"Unlike [fiscal 2024,] when AVGO provided very specific annual AI forecast (revised up several times), we are unsure if the move to quarterly outlook will remove the annual forecast," the firm said.
Bank of America analysts said they assume that Broadcom would still communicate its solid AI positioning across custom chip and networking by either giving AI as a percentage of sales or an outright range for FY25.
"We are not explicitly modeling a large OpenAI contribution (likely more in [fiscal 2026]) but do believe it could be an upside to our forecasts," BofA said.
