Bluesky Thread

gemma3n is now open source, available everywhere — huggingface, transformers,...

June 26, 2025 View original thread

gemma3n is now open source, available everywhere — huggingface, transformers, gguf, ollama, mlx, etc.

huggingface.co/blog/gemma3n

Gemma 3n fully available in the open-source ecosystem!

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Tim Kellogg @timkellogg.me

Gemma 3n: the 4b LLM that’s up with sonnet-3.7 in chatbot arena

the new innovation is Per-Layer Embeddings, which let it consume dramatically less memory

it was created for phones, and is being rolled out to Android phones soon

developers.googleblog.com/en/introduci...

Bar chart titled “Chatbot Arena Elo Score” compares five chatbot models ranked by performance. Each vertical bar represents a model and its Elo score:
• Claude 3.7 Sonnet scores highest at 1287 (Proprietary).
• Gemma 3n is second with 1283, shown in a glowing blue gradient bar. A note clarifies it is a 4B model, with 1.4B active parameters using PLE caching and a total of 7B parameters.
• GPT-4.1-nano-2025-04-14 follows with 1268 (Proprietary).
• Llama-4-Maverick-17B-128E-Instruct has a score of 1266, with 17B parameters and a MoE (Mixture-of-Experts) configuration.
• Phi 4 ranks last at 1202 and is listed with 14B parameters.

Footnote at bottom notes scores are as of May 19, 2025, with Gemma 3n’s confidence interval listed as ±11.

22 3

17 hours later

i had o3 line up the comparison of gemma 3n with other models of varying sizes

Alt text:
Grouped bar chart titled “Benchmark Comparison Across Models.” The x-axis lists seven language models (left → right): Gemma 3-n E4B IT, GPT-4o, Claude 3 Opus, Llama 3.3 70B, Mixtral 8×22B, Llama 3 8B, Phi-3 mini 4B.
For each model there are three adjacent vertical bars:

Model MMLU Accuracy MBPP pass@1 HumanEval pass@1
Gemma 3-n E4B IT ~65 % ~64 % ~75 %
GPT-4o ~86 % ~71 % ~86 %
Claude 3 Opus ~89 % ~70 % ~86 %
Llama 3.3 70B ~86 % ~88 % ~88 %
Mixtral 8×22B ~78 % ~64 % ~74 %
Llama 3 8B ~73 % ~73 % ~73 %
Phi-3 mini 4B ~69 % ~70 % ~59 %

The y-axis spans 0 % to 100 %. Taller bars indicate stronger performance; Llama 3.3 70B, GPT-4o, and Claude 3 Opus lead across all three benchmarks, while Gemma 3-n E4B IT and Phi-3 mini trail, especially on HumanEval.

More like this

it’s here! a real Qwen3 model

Gemma 3n: the 4b LLM that’s up with sonnet-3.7 in chatbot arena

Gemma 27B variant discovered a new cancer pathway treatment that has been val...