LLaMAOpen and Efficient Foundation Language Models

Imagine trying to build a language model that can compete with the biggest names in the field, but doing it with ingredients anyone can inspect and reuse. No hidden data, no locked weights. That’s the bet the LLaMA team makes. They add one more twist: judge models not just by how smart they are, but by how fast and affordable they are to run when you actually use them. In other words, optimize for the world where responses need to come back quickly on real hardware, not just for the leaderboard. LLaMA is their proof of concept: a family that spans roughly 7 to 65 billion parameters, trained on more than a trillion tokens of public text. The punchline is simple. The 13-billion parameter model beats GPT-3 on most tests, and the 65-billion parameter model plays in the same league as Chinchilla-70B and even PaLM-540B. To make that possible with open data, they obsess over the mix and the cleaning. About two-thirds of the corpus is English CommonCrawl, roughly fifteen percent is the C4 dataset, and then you get a healthy dose of GitHub, Wikipedia, and public books, plus arXiv papers and Stack Exchange discussions. Every slice gets deduplicated and scrubbed in its own way—CommonCrawl at the line level after language identification, GitHub by filtering for permissive licenses and removing boilerplate, books deduplicated when two texts overlap too much, and arXiv sanitized to keep the essential content of the paper. The tokenizer is a byte-pair model trained with SentencePiece. Numbers are split into digits to help arithmetic behavior, and any unknown character falls back to raw bytes so the model never struggles with odd symbols. All of it stays on the public side of the fence so other researchers can audit and extend it. Under the hood, this is a modern transformer with a few careful choices that add up. They normalize with RMSNorm before the attention and feedforward blocks. They swap in SwiGLU, a gated activation that tends to help training stability. They encode positions with rotary embeddings, or RoPE, so the model handles long-range order without fixed position vectors. They keep the optimizer practical: AdamW with a cosine schedule that reduces down to a tenth of the peak learning rate. The training loop pushes a 4-million-token batch through the system, but the key is throughput and memory. With memory-efficient attention from the xformers library—drawing on work by Rabe and Staats and a backward pass by Dao and colleagues—plus activation checkpointing and tight model and data parallelism, they squeeze performance out of the hardware. On 2,048 A100 eighty-gigabyte GPUs, they achieve about 380 tokens per second per GPU and finish the 1.4 trillion token run for the big models in roughly 21 days. That’s impressive, but it serves a bigger idea: smaller models, trained longer, can be cheaper to run later. So how do you test that idea? You line up a field of tasks and compare models using the same protocol. The team evaluates zero-shot and few-shot performance across 20 benchmarks that span open-domain questions, reading comprehension, reasoning and math, and code generation. They don’t just chase the biggest number; they ask what you can get at different inference budgets. A 13-billion-parameter model that you can run on a single NVIDIA V100 during inference is a very different proposition than a 500-billion-parameter behemoth that needs a whole rack. That’s the budget-aware perspective they focus on, and it anchors all the comparisons to models like GPT-3, Gopher, Chinchilla, PaLM, OPT, GPT-J, and GPT-Neo. On open-domain questions, the story holds. In NaturalQuestions, their 65 billion parameter model starts strong in zero-shot performance and climbs steadily with more examples. The exact match scores go from twenty-three point eight with no examples to thirty-nine point nine with sixty-four, and in those settings, it beats Chinchilla-70B everywhere except BoolQ and generally edges out PaLM-540B. That’s the headline, but the deeper point is consistency: smaller LLaMA variants scale predictably as you give them a few demonstrations, and the 13 billion parameter model—again, an order of magnitude smaller than GPT-3—holds its own. TriviaQA tells a similar tale. The 65 billion parameter model sits at sixty-eight point two with no examples and nudges past seventy-three with many, which essentially closes the gap to larger, closed systems. Reading comprehension adds a more structured twist—multiple-choice exams that lean on careful parsing. On the RACE benchmark, the 65 billion parameter model hits sixty-seven point nine on middle school questions and fifty-one point six on high school in the zero-shot setup. That puts it competitive with PaLM-540B under several conditions without relying on bespoke training tricks. The 33 billion parameter variant sits just behind with sixty-four point one and forty-eight point three, which is exactly the kind of curve you want from a model family you can scale to your budget. Reasoning, math, and code are where some smaller models stumble. LLaMA doesn’t shy away from those tests. On HumanEval, a code generation benchmark, the 65 billion parameter model reaches twenty-three point seven percent pass at one and seventy-nine point three percent pass at one hundred. On MBPP, which focuses on small Python problems, it lands at thirty-seven point seven percent pass at one and seventy-six point eight pass at one hundred. These are not eclipsing code-specialized systems, but they’re notable because LLaMA wasn’t tuned on code. The 13 billion parameter variant still tops GPT-3 on many code tasks, and the 65 billion parameter model outperforms PaLM-62B in several comparisons. On math, the team even reports that the 65 billion parameter model beats Minerva-62B on GSM8K without code-specific fine-tuning. This is a good sign that simply training more on diverse, public text helps the model internalize patterns it can leverage in problem-solving. If you want a single lens on general knowledge and reasoning, MMLU—Massive Multitask Language Understanding—has become a favorite. It’s a suite of 57 subject exams, and the LLaMA base models step up without any instruction tuning. In a five-shot setup, the 65 billion parameter model averages sixty-three point four across humanities, STEM, social science, and other categories. That slots in between Chinchilla-70B at sixty-seven point five and PaLM-540B at sixty-nine point three, while GPT-3 lags way behind. Scale down and you see the trend clearly: the 33 billion parameter model reaches the high fifties, the 13 billion parameter model lands in the mid-forties, and the 7 billion parameter model sits in the mid-thirties. Not miraculous, but solid footing. Then comes a tiny bit of fine-tuning that delivers a noticeable kick. Following the recipe from Chung and colleagues—minimal instruction-following fine-tuning—the team trains a single instruct model, LLaMA-I. On MMLU in the five-shot setup, the 65 billion Instruct version jumps to sixty-eight point nine percent. That bumps it past Flan-PaLM at sixty-two billion in one configuration and well ahead of OPT-IML-Max at thirty billion. It still trails the then state of the art from the GPT line, which hit seventy-seven point four with a code-augmented variant reported by Iyer and collaborators, but the point stands: a small, simple pass of instruction tuning can move a strong base model several points with very little extra compute. And because the base model was already trained on public data and released, others can iterate from there. Capability is only half the conversation; the other half is harm. The team audits toxicity, bias, and truthfulness across sizes. On RealToxicityPrompts—which uses PerspectiveAPI to score how toxic generations sound—scores for generic prompts are low and fairly flat across sizes, hovering around a tenth. However, on the "respectful prompt" subset, toxicity creeps up with size, and the 65 billion parameter model reaches zero point one four one. Even the generic subset shows a slight increase at the largest scale, with 65 billion around zero point one two eight. The caveat here, as the authors stress, is that PerspectiveAPI is a black box and different sampling pipelines make cross-paper comparisons tricky. Still, the pattern is a reminder: making models bigger doesn’t eliminate every risk. Bias shows up in more structured ways. On CrowS-Pairs, which probes stereotypes across nine categories, LLaMA’s average ends up around sixty-six point six, a touch better than GPT-3’s sixty-seven point two and better than OPT’s sixty-nine point five under the same scoring. But it’s not uniformly rosy. The religion category stands out, with LLaMA showing a roughly ten percent gap relative to OPT-175B, and age and gender also rank high in bias. WinoGender, a coreference test that challenges systems with gendered pronouns, reveals more "gotcha" failures on she or he cases in some sizes than on neutral pronouns like they or someone. On TruthfulQA, LLaMA models outperform GPT-3 both on plain truthfulness and on answers that are truthful and informative, yet the absolute levels remain modest—hallucinations haven’t been vanquished. Training at this scale draws serious power, so the team does the carbon math transparently. Using a Power Usage Effectiveness of 1.1 and the U.S. average grid intensity, they convert energy to emissions with a simple expression: emissions in tons equals the energy in megawatt-hours times zero point three eight five. According to this accounting, training the 65 billion parameter model consumed about one million twenty-two thousand three hundred sixty-two GPU-hours and corresponds to roughly 173 tons of CO2 equivalent when normalized to the same data center assumptions. It’s in the same ballpark as other recent 175 billion scale efforts. The broader argument is pragmatic: if you release strong open models that can be deployed on a single GPU or a small cluster, you reduce the need for others to retrain similarly large systems and shift usage to lower-cost inference, which is where most energy gets spent over time. That logic ties back to openness. The LLaMA team trained exclusively on public sources and released the models to the research community, with the goal of democratizing the study of robustness, toxicity, and bias. There were earlier open models—OPT, GPT-NeoX, BLOOM, GLM—but none were really matching the closed leaders like PaLM-62B or Chinchilla on broad benchmarks. By closing that gap at multiple inference budgets, LLaMA lowers the barrier to entry. It also invites scrutiny: if bias worsens in some slices as models scale, or if toxicity spikes with certain prompts, an open model makes it easier for others to investigate and mitigate. There are limits here, and the authors are candid about them. Some of the safety comparisons aren’t apples to apples because prior teams used different sampling or scoring pipelines. The evaluation suite, while extensive, is still an abstraction of the messiness of real deployment. And instruction tuning, which delivered a clean gain on MMLU for LLaMA-I, was only tested in a single, minimal setup—so it’s a proof of potential, not a full exploration. Step back and the throughline is clear. If you hold training compute roughly fixed and feed a smaller model more tokens—carefully cleaned, publicly sourced, and broadly representative—you can achieve performance that rivals much larger, closed alternatives. You also get something practical: models that you can run on modest hardware and adapt quickly with a small dose of instruction tuning. The details matter—the RMSNorm and SwiGLU choices, the RoPE positions, the memory-efficient attention from xformers, and the tokenizer that splits digits and falls back to bytes—but they serve that bigger strategy. Where does it go next? The team hints at training on even larger public corpora, expanding instruction tuning, and deepening the audit of biases and toxicity. That’s the right balance. Push capability, keep the systems open, and measure the externalities as you go. If the field keeps optimizing for the world where someone is actually waiting for an answer—on a phone, on a workstation, in a clinic—then "budget-aware scaling" won’t just be a slogan. It’ll be how we build models that are not only effective, but also good to use.