~/beer-and-code
▪ Clã Beer and Code the largest AI Engineering community in Brazil · live, every week join the Clã ›
~ / tag / #benchmark $ grep

#Benchmark

31 posts
25 #ia · #google
Gemini 3.5 Pro: Release Tracker, Leaks, and What's Actually Confirmed

Gemini 3.5 Pro still isn't out, and every week there's a new date going around. I separated what Google has confirmed in writing from what's just a leak with no reproducible data, with the timeline of delays since I/O in May. And I built a 20-line watcher on the Gemini API models endpoint that pings you the minute gemini-3.5-pro shows up, plus a model fallback in Laravel so the switch becomes one line of .env.

08 Aug · 6 min ›
26 #llm · #custos
DeepSeek V4 Flash 0731: New Weights, 82.7 on Terminal Bench and the Bill That Hurts OpenAI

DeepSeek republished the V4-Flash weights on July 31 without changing the model name in the API: anyone calling deepseek-v4-flash woke up running a different model, with no changelog. The real 0731 numbers (82.7 on Terminal Bench, but 79% in the independent measurement), where it beats GPT-5.6 Luna and where it loses, the 169 GB to run it locally and what to do if your agent points at a model name that became a moving target.

31 Jul · 12 min ›
27 #ia · #llm
Qwen 3.8 Max (2.4T): Benchmark vs Kimi K3, Pricing and Open Weights

Alibaba dropped Qwen 3.8 Max, a 2.4-trillion-parameter MoE it claims is the second-best model in the world, behind only Fable 5. With no public benchmark at launch, the only independent test scored it 80/100 against Kimi K3's 83. Here: what it is, what it costs (Token Plan from $6 to $68), how to access it via API, and where each model in the Qwen 3.8 family fits.

20 Jul · 9 min ›
28 #ia · #llm
Kimi K3 Pricing: $3/$15 per Million Tokens, Is It Free? Benchmarks vs Claude

Kimi K3 costs $3 input and $15 output per million tokens on the API (the chat at kimi.com is free), 1.7x cheaper than Claude Opus 4.8, not the "5x" your timeline is claiming. We did the honest pricing math, separated the verifiable benchmarks from the hype, and show where it beats Claude (and where it doesn't).

17 Jul · 10 min ›
29 #produto-ia · #llm
Claude Sonnet 5: How to Read the Benchmarks in Practice

Claude Sonnet 5 is out. Instead of screenshotting the table, this post shows how to read the benchmarks in practice (cost per task, effort, with and without tools) for anyone using Claude Code or building AI into a product.

30 Jun · 9 min ›
30 #openai · #ia
Claude Code vs Codex: Codex Wins the Terminal by 13 Points, Claude Wins the Hard Repo by 10

Claude Code or Codex? The answer comes with numbers: Codex opens a 13-point lead on Terminal-Bench (82.7% vs 69.4%) and Claude Code opens a 10-point lead on SWE-bench Pro (69.2% vs 58.6%), which is the benchmark for real multi-file problems. On SWE-bench Verified they tie. Here is the verdict by scenario, the real cost per dev, and the criterion that matters more than quality: how much control you want during the task.

19 Jun · 10 min ›
31 #ia · #ai-agents
Fable 5 vs Opus 4.8: which one to use (and the 10 tasks where the difference shows)

Fable 5 or Opus 4.8, which one should you use? A straight verdict by task type, with the 10 concrete situations where Fable gets it done and Opus 4.8 did it badly or not at all: migration at scale, code from a screenshot, long-running agents and reasoning over documents. Plus what each one costs and the cases where Opus 4.8 is still the right call.

09 Jun · 11 min ›
Meet the Clã Beer and Code
playing