PaLIA Jointly-Scaled Multilingual Language-Image Model

Picture yourself in a foreign city. You point your phone at a street sign, a café menu, or a handwritten note on a door, and you ask, “What does this say, and what should I do?” Now imagine the phone not only reads the text in the image, but also explains the scene, answers follow-up questions in your language, and does this just as easily in French, Hindi, or Thai. That is the promise behind PaLI, a new model from Google Research. PaLI can see and talk, across many languages, and it uses one simple interaction: give it an image and a bit of text, and it replies with text. Before this work, we already had big leaps in language models and vision models. Language models like T5 and GPT learned to produce fluent text by getting bigger and training on more data. Vision models like the Vision Transformer learned to recognize objects by also getting bigger and using more images. Some systems combined images and text, but they were usually English-only, or relied on separate, task-specific parts, or treated answers as a multiple-choice list. That limits flexibility in the real world. Another issue: the language part was often huge while the vision part was small. If your eyes are weaker than your voice, you miss a lot. So the authors asked an ambitious question: Can one model handle language-only tasks, vision-only tasks, and image-plus-language tasks, in many languages, using the same input and output format? And if we scale both the eyes and the voice together rather than only growing the language side, do we get better returns for the added size? Finally, can we reuse strong existing models for language and vision so we do not need to train everything from scratch? Those questions drive PaLI: a single, simple interface that you can use for captioning, question answering, reading text in images, and even object detection, all expressed as text generation. The approach is deliberately straightforward. Think of PaLI as a pair of very good eyes feeding a very capable brain and voice. The eyes are a Vision Transformer, a model that chops an image into patches and turns them into a sequence of features, like visual words. These visual words are fed, along with any text prompt or question, into a large text encoder-decoder, which reads the input and generates an answer. There is no special module for each task. Everything is framed as image plus text in, text out. Want a caption? Prompt it to Describe this image in English. Want a list of objects? Ask, List the objects present. Data is the fuel. The team built a huge multilingual image-text resource called WebLI, collected from the public web. It spans around 10 billion images and tens of billions of associated texts across more than 100 languages. They also ran optical character recognition, or OCR, on those images to pull out text that appears inside pictures, like signs and labels, creating tens of billions more image-text pairs. To keep quality while staying big, they scored image-text matches and kept the top 10 percent, ending up with about a billion strong examples. On top of that, they designed a training mix of eight kinds of tasks to teach general skills. What is in that mix? First, text-only learning helps the model keep its language skills sharp. Second, captioning in many languages teaches it to describe images. Third, reading text in images with OCR teaches it to handle street signs, storefronts, and menus. Fourth, visual question answering and question generation teach it to follow instructions and ask sensible questions. Fifth, object-aware question answering and a text-based version of object detection give it a sense of what is present and where. Training passes once over a curated 1.6 billion-example mixture. For most of that phase, the vision backbone is frozen, which the authors found helpful. A brief high-resolution phase then updates everything. To make PaLI practical, the authors do not start from scratch. For language, they reuse mT5, a multilingual version of T5, in a very large size with 13 billion parameters. For vision, they use very large Vision Transformers: a previously trained 1.8 billion parameter model called ViT-G, and a new, even larger one they trained, called ViT-e for enormous, with 4 billion parameters. Combining these gives PaLI models from about 3 billion to 17 billion total parameters. The largest, PaLI-17B, splits capacity more evenly between vision and language than prior work, which turns out to matter a lot for performance. So what happened when they put it all together? On English image captioning with the popular COCO benchmark, PaLI-17B reaches 149.1 on the CIDEr score, a standard measure of caption quality. On NoCaps, which tests rare or out-of-domain objects, it scores 124.4 overall and is especially strong on images outside the COCO style. On TextCaps and on VizWiz-Cap, where reading text in messy photos is crucial, PaLI posts state-of-the-art results, particularly when given OCR strings as input, much like giving a person reading glasses. On visual question answering, PaLI-17B hits 84.3 percent accuracy on VQAv2 in a fully open-vocabulary setting. That means it generates answers in free-form text, not by choosing from a short list, yet it still surpasses models using the easier fixed-choice setup. On OKVQA, which requires outside knowledge beyond the picture, it reaches 64.5 percent, a large jump over earlier systems. On TextVQA, VizWiz-QA, and ST-VQA, which involve reading text in images and answering questions about it, PaLI is at or near the top, again strongest when allowed to use OCR strings. The multilingual story may be the most exciting. The team evaluates image captioning across 35 languages and sees large gains over previous baselines in every one. The average score rises to 53.6 on a common metric. Languages as diverse as French, Hindi, Hebrew, Thai, and Chinese all improve substantially. They also test cross-lingual question answering, for example asking in Korean and answering in English, and multilingual question answering where the answer matches the questions language. In both cases PaLI-17B significantly outperforms prior systems across all tested languages. In short, multilingual ability is baked in, not bolted on after the fact. Why does all of this matter? First, the unified interface simplifies system design and feels natural to users: image plus text in, text out. You can ask for a caption, a translation, a list of objects, or a specific answer with follow-ups, all with the same model. Second, multilingual ability is built in from the start, which means tools that work for more of the world, not just English speakers. Imagine education apps that explain diagrams in a students language, accessibility tools that help people who are blind get reliable answers about their surroundings, and global commerce where product photos and labels are truly understood. Of course, there are caveats. Web data can reflect biases, harmful content, and uneven coverage across languages, so quality may vary. The authors acknowledge these risks and stress careful use. They provide a data card and model card, and remove near duplicates of evaluation images from pretraining to reduce leakage. Training very large models requires significant compute, though they reduce cost by reusing pretrained parts and freezing the vision backbone for most of pretraining. Finally, evaluating open answers is tricky: sofa versus couch may sometimes be marked wrong. Here is the big takeaway. PaLI shows that when you scale both eyes and voice together, train on a rich mix of tasks, and include many languages from the start, you can build one simple model that captions, answers, reads, and reasons across languages. It does this while setting new records and keeping language understanding strong. The next step is to make these systems more responsible, more efficient, and even more helpful. The next time you point your camera at the world and ask a question, the answer may come from a model built very much like this one.