Multimodal Foundation Models Explained: What They Are and What Current Models Can Actually Do (November 2026)

Quick answer: A multimodal foundation model is a large AI model trained on broad data across more than one type of input, such as text and images, then adapted to many tasks through prompting or fine-tuning. The term builds on Stanford’s 2021 foundation-model idea. Inputs and outputs vary by model, so check each model’s documentation.

You paste a photo, a chart or a clip into an AI tool and expect it to “see” everything. Then it ignores the audio, or it can read the image but cannot answer in speech. The gap between what people assume and what a model accepts is the biggest source of confusion in this field.

Here is a verified example. According to OpenAI’s model page, read on November 9, 2026, its top model GPT-6 Astra accepts text and image input and returns text. The same page lists audio and video as not supported. Several third-party pages claim otherwise for OpenAI’s earlier flagship. This guide defines the term, shows what current models accept and produce, and gives you a way to check any model’s claims yourself.

What is a multimodal foundation model?

It is a model with two properties at once. It handles more than one modality (a type of data, such as text, images, audio or video), and it is a foundation model, meaning it is pre-trained on broad data and adapted to many downstream tasks.

A 2023 survey by Chunyuan Li and colleagues, “Multimodal Foundation Models: From Specialists to General-Purpose Assistants,” describes the idea this way. Foundation models, as defined in the Stanford foundation-model paper (Bommasani et al., 2021), are trained on broad data and can be adapted to a wide range of tasks. The survey narrows the focus to vision and vision-language versions of them.

Two related terms overlap with it:

• Multimodal LLM (or large vision-language model): a language model extended to read non-text input. These names describe the design. “Foundation model” describes the training philosophy. Neither is a formal standard, and many current models fit both.
• Unimodal model: handles one data type, such as a text-only model.

What most people miss: “multimodal” does not mean “every modality in and every modality out.” Most current models read several input types and answer in text. Generating video or speech is usually a separate model or endpoint.

Which multimodal models matter in November 2026, and what do they accept?

The answer depends on the model. The table below uses only what vendors state on their own pages, read on November 9, 2026, unless marked otherwise.

Read the table as a warning against assumptions. A model that reads video (Gemini 3.1 Pro) is different from one that makes video (Gemini Omni 1.2 Flash), and both differ from a model that turns media into searchable vectors (EmbeddingGemma 2). If you want the embedding case in detail, see this explainer on the open multimodal embedding model EmbeddingGemma 2.

Google also announced Gemini 4 Carbon in October 2026. According to Google’s October roundup, published November 2, it was in pre-release testing with the U.S. government and trusted cyber defenders, ahead of a broader launch. Treat it as not generally available as of November 2026.

Is GPT-4o still the example to use?

Not for ChatGPT. According to OpenAI’s help center, GPT-4o was retired from ChatGPT on February 13, 2026. The same page says the retired models remained available through the API, and that ChatGPT Voice was not changed by the retirement.

Many explainers still use GPT-4o as the standard omni example. That is out of date for ChatGPT users. OpenAI’s help page does not say how long API access will last, so check its deprecations page before building on it.

What changed recently

What does “omni” mean? A name-confusion table

“Omni” now appears in several unrelated places, which causes mix-ups.

How do multimodal foundation models work?

Most published designs combine three parts: an encoder for each non-text modality, an alignment step that maps those encodings into something the language model can read, and a language-model core.

A 2023 study of object hallucination, published at EMNLP, describes a typical vision-language model as a vision encoder, a language model and a cross-modal alignment network. Training generally runs in steps: pre-train the encoders on single-modality data, align them through image-text pre-training, then tune the model to follow visual instructions.

CLIP, from OpenAI researchers, is the best-documented example of the alignment step. According to the paper, it was trained on 400 million image-text pairs by predicting which caption goes with which image. It matched the accuracy of the original ResNet-50 on ImageNet without using ImageNet’s 1.28 million training examples, and it was tested on over 30 datasets.

What most people miss: that pipeline describes research systems. OpenAI and Google publish limited architecture detail on the pages I reviewed. Anyone who tells you exactly how a frontier model is built is describing a guess.

What kinds of multimodal foundation models exist?

Four practical categories cover the field. Examples come from sources checked above.

• Models that read several inputs and answer in text. GPT-6 Astra and Gemini 3.1 Pro fit here.
• Models that generate media. Gemini Omni 1.2 Flash makes video up to 15 seconds from references.
• Embedding models. EmbeddingGemma 2 puts text, code, images, video and audio in one vector space, so a text query can find a clip. According to Google, its modular design cuts local vector storage by up to 6x. That is a vendor claim.
• Vision-language-action (VLA) models. The RT-2 paper by Brohan and colleagues (2023) expressed robot actions as text tokens, so one model could handle images, language and actions. The authors report about 6,000 evaluation trials. That is a research result, not a product.

Where do multimodal models fail?

They invent details, they are hard to evaluate, and their stated abilities often exceed what is documented.

The EMNLP 2023 paper “Evaluating Object Hallucination in Large Vision-Language Models” found that the models it tested mostly suffered from severe object hallucination, meaning they described objects not in the image. It also found that objects frequently appearing in instructions, or alongside real objects, were especially likely to be invented. That study tested 2023-era models, so it does not measure today’s. It does show a failure type you should test for.

Myth vs fact

How much do multimodal models cost?

Pricing is per million tokens. Here is a worked example using published list prices.

Illustrative assumption: one call with 100,000 input tokens and 5,000 output tokens, text tokens only. In our check on November 9, 2026, we computed:

Three caveats:

• According to OpenAI, prompts above 272,000 input tokens are priced at 2x input and 1.5x output for the whole request.
• Image, audio and video inputs consume tokens at rates this table does not model.
• Google’s Gemini Omni 1.2 Flash is priced per second of video instead. Google lists $0.07 per second at 720p, so a 15-second clip is $1.05.

For more on managing token use, see this guide to context engineering. If you plan to self-host an open model, this GPU rental pricing guide covers hardware costs.

How do you choose a multimodal model? A Modality Match Check

This is an original checklist, built from the verified facts above. The mappings are illustrative.

Then ask two more questions: how long is your input, and what is your monthly volume? For agent-style use, compare how Gemini agents handle enterprise work against your own tasks.

How can you check a multimodal claim yourself?

1. Open the vendor’s model page, not a blog post about it.
2. Find the inputs and outputs fields and read them literally.
3. Note the date and model snapshot.
4. Test your own file type with a small sample.
5. Ask the model about a detail you know is absent, to see if it invents one.

Frequently Asked Questions

What is a multimodal foundation model?
It is a large model pre-trained on broad data across more than one modality and adapted to many tasks. The “foundation” part comes from Stanford’s 2021 framing of models trained on broad data. Each model’s inputs and outputs differ, so check its documentation.

Is a multimodal LLM the same thing?
They overlap but are not formally identical. “Multimodal LLM” usually describes a language model extended to read other inputs. “Foundation model” describes broad pre-training and wide adaptability. Many current models are both.

Is GPT-4o still available?
Not in ChatGPT. OpenAI retired it there on February 13, 2026. According to OpenAI’s help page, it stayed available through the API, so check OpenAI’s deprecations page for the current API status.

Can GPT-6 Astra process audio or video?
According to OpenAI’s model page as of November 2026, it accepts text and image input and returns text. Audio and video are listed as not supported on that page. Voice is covered in OpenAI’s separate Realtime and GPT-Live documentation.

What is a vision-language-action model?
It is a model that takes images and language and outputs robot actions. The RT-2 paper in 2023 introduced the term by expressing actions as text tokens. Today’s examples are mostly research systems.

Do multimodal models hallucinate?
Yes. A 2023 EMNLP study found object hallucination was common in the vision-language models it tested. Newer models may differ, so test your own inputs.

How much do they cost?
It depends on the model, token count and modality. On our illustrative 100,000-input, 5,000-output call, list prices ranged from $0.25 to $1.25 per call. Video generation is priced per second instead.

Conclusion

Key takeaways: a multimodal foundation model is defined by broad training across modalities, not by accepting every input. Current flagship pages from OpenAI list image input and text output. Google’s Gemini 3.1 Pro lists video and audio input with text output, and Google’s Omni model generates video. Vendors change their lineups quickly, as GPT-4o’s retirement from ChatGPT shows.

Your next step: pick one model, open its official page, and write down its exact input and output list before you build anything.

Related: How the open multimodal embedding model EmbeddingGemma 2 works

Disclaimer: This article is for informational purposes only and is not affiliated with, sponsored by or endorsed by any company mentioned. Prices, features, model availability and policies change often, and information was last verified on November 9, 2026. AI output can be inaccurate, so test any model on your own data before relying on it.