MizAR 60 for Mizar 50

Jakubův, Jan, Chvalovský, Karel, Goertzel, Zarathustra, Kaliszyk, Cezary, Olšák, Mirek et al. (+4)•View original

HighlightsAccessibleballad voice

Imagine opening a math textbook, picking any theorem, and asking a computer, "Can you re-prove this from scratch, just by knowing the definitions and theorems in the library?" That's the hammer idea. It lives inside Mizar, a proof assistant where mathematics is written in a precise, checkable language, and it treats the giant Mizar Mathematical Library as a toolbox. The job is to pick the few right tools and finish the task quickly. Here's how that works in practice. Mizar statements are translated into first-order logic—the kind of symbolic sentences automated provers understand—using a pipeline called MPTP. An automated theorem prover, like E Prover or Vampire, then searches for a proof. Crucially, it doesn't peek at the human-written script. It has to rediscover the argument by selecting relevant premises—supporting facts—from the library and stitching them together under a fixed time limit. Think of it like speed chess with a massive opening book: you have to choose the right lines, quickly, from millions of possibilities. Jakubův, Kaliszyk, Urban and colleagues make this concrete in their MizAR 60 for Mizar 50 study. They work with a large library snapshot, MML 1147, which contains fifty-seven thousand eight hundred ninety-seven theorems. Each target theorem is translated and attacked by a portfolio—a coordinated set—of strategies across E and Vampire. The search is guided by ENIGMA, a machine learning system that learns which proof steps and premises tend to pay off. They use both graph neural networks, which see structure, for fast runs, and gradient-boosted decision trees, which are strong on tabular signals, for longer ones. The whole thing runs under a strict four hundred twenty-second cap per problem, with models preloaded on a GPU server and, at evaluation time, up to one hundred sixty E instances sprinting in parallel. It's industrial-scale math. Building a good portfolio isn't guesswork here. The team repeatedly splits the library at random, trains on one half, tests on the other, and does this a thousand times. They assemble a ninety-five slice portfolio—a menu of base strategies paired with ENIGMA variants—that covers as many problems as possible without overfitting, then hand-pick the strongest, most general sets on fresh splits. They also minimize clutter by learning from subproblems: once an automated proof appears, they prune premises and add only what helps—definitions, background facts, or highly ranked extras—so later searches stay focused. So what does this buy you? In the hammering or large-theory mode—the tough setting where the prover must hunt across the whole library—they automatically prove fifty-eight point four percent of Mizar's top-level lemmas within four hundred twenty seconds. That's a big jump from about forty point six percent on the earlier Mizar forty benchmark. A single strong method gets forty percent in just thirty seconds, which shows speed matters. If you make life easier and feed only the premises humans actually used—the bushy or used-premises mode—the system proves about sixty percent in one hundred twenty seconds in their standardized test. In another evaluation of the same idea, success reached about seventy-five percent, underscoring a simple lesson: targeted premises are valuable. On held-out data, the hammer portfolio sits around fifty-eight point thirty-six percent, and on the development side about sixty point four percent, showing the gains aren't a fluke. Behind those numbers is a loop: grow more automated proofs, retrain ENIGMA and its Deepire variant, and repeat. As they report, that cycle scales to millions of proofs and pushes the headline hammering rate higher, with milestones like seventy-five percent in sight when the training engine really operates smoothly. The takeaway is clear. Translation makes Mizar math speak the language of automated provers. Machine learning tells the search where to look. A carefully engineered portfolio and hard time caps turn it into a practical tool. From scratch, the system now cracks about fifty-eight percent of the library; with the right ingredients handed to it, that rises toward sixty to seventy-five percent. And every extra point isn't just a statistic—it's another theorem the computer can now find on its own.

Here's how that works in practice. Mizar statements are translated into first-order logic—the kind of symbolic sentences automated provers understand—using a pipeline called MPTP. An automated theorem prover, like E Prover or Vampire, then searches for a proof.

Crucially, it doesn't peek at the human-written script. It has to rediscover the argument by selecting relevant premises—supporting facts—from the library and stitching them together under a fixed time limit. Think of it like speed chess with a massive opening book: you have to choose the right lines, quickly, from millions of possibilities.

Jakubův, Kaliszyk, Urban and colleagues make this concrete in their MizAR 60 for Mizar 50 study. They work with a large library snapshot, MML 1147, which contains fifty-seven thousand eight hundred ninety-seven theorems. Each target theorem is translated and attacked by a portfolio—a coordinated set—of strategies across E and Vampire.

The search is guided by ENIGMA, a machine learning system that learns which proof steps and premises tend to pay off. They use both graph neural networks, which see structure, for fast runs, and gradient-boosted decision trees, which are strong on tabular signals, for longer ones. The whole thing runs under a strict four hundred twenty-second cap per problem, with models preloaded on a GPU server and, at evaluation time, up to one hundred sixty E instances sprinting in parallel. It's industrial-scale math.

Building a good portfolio isn't guesswork here. The team repeatedly splits the library at random, trains on one half, tests on the other, and does this a thousand times. They assemble a ninety-five slice portfolio—a menu of base strategies paired with ENIGMA variants—that covers as many problems as possible without overfitting, then hand-pick the strongest, most general sets on fresh splits.

They also minimize clutter by learning from subproblems: once an automated proof appears, they prune premises and add only what helps—definitions, background facts, or highly ranked extras—so later searches stay focused.

So what does this buy you? In the hammering or large-theory mode—the tough setting where the prover must hunt across the whole library—they automatically prove fifty-eight point four percent of Mizar's top-level lemmas within four hundred twenty seconds. That's a big jump from about forty point six percent on the earlier Mizar forty benchmark.

A single strong method gets forty percent in just thirty seconds, which shows speed matters. If you make life easier and feed only the premises humans actually used—the bushy or used-premises mode—the system proves about sixty percent in one hundred twenty seconds in their standardized test. In another evaluation of the same idea, success reached about seventy-five percent, underscoring a simple lesson: targeted premises are valuable.

On held-out data, the hammer portfolio sits around fifty-eight point thirty-six percent, and on the development side about sixty point four percent, showing the gains aren't a fluke.

Behind those numbers is a loop: grow more automated proofs, retrain ENIGMA and its Deepire variant, and repeat. As they report, that cycle scales to millions of proofs and pushes the headline hammering rate higher, with milestones like seventy-five percent in sight when the training engine really operates smoothly.

The takeaway is clear. Translation makes Mizar math speak the language of automated provers. Machine learning tells the search where to look.

A carefully engineered portfolio and hard time caps turn it into a practical tool. From scratch, the system now cracks about fifty-eight percent of the library; with the right ingredients handed to it, that rises toward sixty to seventy-five percent. And every extra point isn't just a statistic—it's another theorem the computer can now find on its own.

Related lectures