Multimodal model

What is a multimodal model?

A multimodal model is an AI model that can work with more than one type of input. Text is one modality. Images, audio, video, tables, screenshots, and documents are other modalities.

A text-only language model reads and writes text. A multimodal model can look at a chart, read a scanned invoice, listen to a short audio clip, or answer a question about a screenshot. Some models can also produce more than one output type, such as text, speech, images, or structured data.

The useful part is context. Many business questions are not pure text questions. A dashboard screenshot has layout and colours. An invoice has a visual structure. A phone call has tone and timing. A multimodal model can use those signals alongside the prompt.

Common modalities

Text. Prompts, documents, emails, chat messages, code, tables, and extracted fields.

Images. Photos, screenshots, scans, charts, diagrams, forms, product images, and labels.

Audio. Speech, meetings, calls, voice commands, and in some systems other sound patterns.

Video. A sequence of images, often combined with audio. Video is useful for meetings, inspections, training material, and movement over time, but model support and cost vary a lot.

Capabilities differ by provider and by model. One model may accept text and images but produce only text. Another may handle real-time audio. Another may read video as sampled frames. Always check the current model documentation before designing the workflow.

How it differs from OCR and speech-to-text

Older document and voice workflows often chain several specialised tools together. OCR extracts text from an image. Speech-to-text transcribes audio. A language model then reasons over the extracted text.

A multimodal model can sometimes handle more of that chain in one step. For example, it can read a photographed invoice and use the layout to understand which amount belongs to which field. It can inspect a screenshot and reason about the chart rather than only reading the visible labels.

That does not make specialised tools obsolete. For high-volume invoice capture, classic document intelligence may still be cheaper, faster, and easier to validate. For call transcription at scale, a dedicated speech model may be the right first step. Multimodal models are strongest when the context around the text matters.

Where multimodal models help

• Document understanding. Extract data from invoices, contracts, delivery notes, forms, and scans where layout carries meaning.

• Screenshot support. Let a user upload a Power BI screenshot, error message, or application screen and ask what looks wrong.

• Visual quality checks. Review product photos, packaging, shelves, or simple inspection images when the task is varied and low volume.

• Voice interfaces. Build assistants that respond to spoken input, summarise calls, or help people who cannot comfortably type.

• Meeting and training content. Summarise recordings, extract action points, or find moments in video where a specific topic appears.

The pattern is the same each time: use the model when the answer depends on a mix of text and another signal.

Multimodal model or specialised model?

Choose a multimodal model when you need flexibility, varied inputs, and natural-language reasoning. It is often the quickest route for prototypes, support workflows, document triage, and assistant experiences.

Choose a specialised model when the task is narrow, high volume, safety-sensitive, or needs strict measurement. A production line that rejects products at speed usually needs a trained computer vision model. A finance workflow that posts invoices automatically needs strong validation around every extracted field.

Many real systems combine both. A document pipeline may use OCR or document intelligence for extraction, then a large language model for explanation, exception handling, or classification. A support assistant may use a multimodal model for screenshots, but call a normal API once it knows which action to take.

What to watch out for with multimodal models

They can invent visual details. A model may describe something that is not in the image or miss a small but important detail. Check critical output against the original input.

Counting and measurement can be weak. General multimodal models are often poor at precise counts, small objects, and exact measurements unless the task is designed carefully.

Privacy rules still apply. Uploaded images, audio, and video may contain faces, personal data, medical details, documents, or confidential screens. Check retention, training-use, region, and access policies before sending data to an external API.

Costs are less predictable than text. Images, audio, and video can consume more tokens or processing units than plain text. Measure usage early, especially for video and real-time audio.

Test with real examples. A model that works on a clean sample invoice may fail on blurry scans, handwritten notes, screenshots with dark mode, or noisy calls. Use representative input before making it part of a workflow.