Gemini's Language Abilities
CMU's impartial, reproducible evaluation of Gemini Pro against GPT and Mixtral across standard LLM benchmarks.
Ask this paper
Reproducible methodology: Provides an open, reproducible evaluation pipeline - a response to concerns about Google's own Gemini launch benchmarks being hard to independently verify.
Gemini Pro vs. GPT 3.5 Turbo: Gemini Pro achieves comparable but slightly lower accuracy than GPT 3.5 Turbo, countering marketing claims of broad parity on language tasks.
Gemini & GPT beat Mixtral: Both Gemini and GPT outperform Mixtral on these benchmarks, suggesting open mixture-of-experts has not yet closed the gap to frontier proprietary models.
Evaluation norms: Positioned as evidence that independent replications remain essential, and that first-party model reports shouldn't be the final word on comparative capability.