Speech, vision, and multimodal AI

A multimodal model works with more than one kind of data, such as text, images, audio, or video. Some designs pair two encoders. OpenAI's CLIP trains an image encoder and a text encoder so that matching image and caption pairs score as similar, which lets it pick the most relevant text for an image without being trained for that specific task. Its model card describes CLIP as a research output: any deployed use, commercial or not, is currently out of scope; certain surveillance and facial recognition uses are always out of scope; and use should be limited to English.12

Vision-language models take images, and sometimes video, together with text and answer in text. Ai2's Molmo2-O-7B joins the SigLIP 2 vision encoder to Ai2's Olmo 3 7B Instruct language model and supports image, video, and multi-image understanding and grounding. Microsoft's Florence-2 uses simple text prompts to choose a task such as captioning, object detection, or segmentation. Google DeepMind's Gemma 4 12B Unified takes a different route: instead of separate encoders, it projects image patches and audio waveforms directly into the language model through lightweight linear layers, and it reads video as sequences of frames.345

Speech recognition models turn spoken audio into text. OpenAI's Whisper is a sequence-to-sequence model trained for multilingual speech recognition, translation of speech into English, and language identification. NVIDIA's parakeet-tdt-0.6b-v3 is a 600-million-parameter model that transcribes 25 European languages, detects the spoken language automatically, and adds punctuation and word-level timestamps. Whisper's model card warns that its output can include text that was not spoken, that accuracy varies across languages, accents, and dialects, and cautions against transcribing people recorded without their consent.678

Some vision models are building blocks for other software rather than chat systems. Meta's SAM 3 detects, segments, and tracks objects in images and video from short text phrases or from visual prompts such as points, boxes, and masks; SAM 3.1, released on March 27, 2026, added new checkpoints and a shared-memory approach to tracking several objects at once. Meta's DINOv3 is a family of vision foundation models that produce dense image features, and its repository includes code for semantic segmentation and depth estimation built on those features.911

Generative models create images, video, or audio. Hugging Face's Diffusers library for diffusion models is organized around three parts: pipelines that run a model in a few lines of code, interchangeable noise schedulers, and pretrained model components. Genmo's Mochi 1 preview is a 10-billion-parameter text-to-video diffusion model released under Apache 2.0. Its card says the initial release generates 480p video and that Genmo's reference code needs about 60 GB of GPU memory on a single GPU.1213

Terms are set for each release, so models of the same type can come with different conditions. parakeet-tdt-0.6b-v3 is under CC BY 4.0. Molmo2-O-7B is licensed under Apache 2.0, but its card says it is intended for research and educational use and was trained on third-party datasets subject to academic and non-commercial research use only. SAM 3 and DINOv3 weights are available only after an access request and come under Meta's own SAM License and DINOv3 License. Read the card and license for the specific checkpoint before you build on it.8391011