Models18d ago
JetBrains ships Mellum2.1: 12B MoE (2.5B active) open coding-agent model, SWE-bench Verified jumps 2.0→47.0
On Oct 8 JetBrains released Mellum2.1 Thinking: same 12B MoE / 2.5B active / 131K / Apache 2.0 architecture, with gains almost entirely from large-scale RL in real sandboxes. Self-reported same-pipeline scores: SWE-bench Verified 47.0 (vs Mellum2 2.0), LiveCodeBench v6 82.0. On Hugging Face; vLLM serve now; GGUF/MTP coming soon.
Tooling15d ago
JetBrains Unveils SWE-TaskFlow: Aligning Coding Agent Benchmarks with Real Developer IDE Workflows
Mainstream coding agent benchmarks like SWE-bench frame evaluation around issue-derived tasks: feeding an agent a bug description and expecting an end-to-end patch. But does this reflect genuine developer practices? JetBrains Research analyzes 4,782 production sessions of professional software engineers inside JetBrains IDEs (arXiv:2610.09633). For long interactive sessions (>=3 turns, comprising 33% of production traffic), user requests span a diverse mix of intent types—project codebase Q&A, strategic planning, code review, refactoring, and test execution—with users frequently transitioning between types mid-session. To align benchmarks with real developer task flows, JetBrains introduces SWE-TaskFlow, an approach that transforms issue-based benchmarks via prompt-splitting and verifiable repository QA into calibrated multi-stage workflows guided by a TaskFlow Alignment Score (TFAS). A pilot study across 700 SWE-Bench Pro tasks demonstrates that sequential multi-step interaction protocols nearly double agent inference expenditure without stable improvements in resolve rates, proving that the interaction protocol itself is a critical, previously unmeasured dimension of agent evaluation.
Tooling34d ago
GitHub Copilot for JetBrains 1.18: assisted approvals, org skills, Codex plan mode
GitHub Changelog (@GHchangelog) announced Copilot for JetBrains 1.18.0: public-preview AI-assisted tool approvals auto-approve low-risk calls; organization and enterprise shared skills/instructions; and plan mode for the Codex agent so you can review a plan before implementation. You can also re-edit earlier messages (rewinding chat and file edits) and chat alongside sessions.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.