Multimodal
A multimodal model is one that accepts or produces more than one kind of content — combining text with images, audio, or video in a single request rather than handling each in a separate system.
Practically, this means a single model can read a screenshot and answer questions about it, transcribe speech and summarise it, or take a design and return the markup — work that previously required stitching several specialised services together.
The failure modes differ by mode. Models read fine detail in images unreliably — small text, precise counts, exact spatial relationships — so anything consequential extracted from an image should be verified against the source.
How it works
Each kind of input is turned into the same internal representation the model reasons over: an image is cut into patches, audio into short frames, and each is encoded into tokens alongside the text. The model then attends across all of them at once, which is why it can answer a question that depends on both a chart and the paragraph beneath it.
Output can be multimodal too: some models generate images or speech directly, while others hand off to a separate generator.
Example
A designer can give a model a screenshot of a page and ask for the semantic HTML that reproduces its structure, or a marketer can upload a photo of a printed brochure and ask for the text transcribed and translated, in one step rather than through separate OCR and translation tools.
Common mistakes
- Trusting exact figures, small print or counts read from an image without checking the original.
- Sending huge, uncompressed images when a cropped section would answer the question.
- Assuming every model handles every mode equally well; capabilities vary a lot by provider.
