Microsoft Open-Sources Mage-VL-Cowork, a 4B Computer-Use Agent That Chooses Between Clicking the GUI and Running Bash, Scoring 66.0% on OSWorld-Verified

Background

Computer-use agents that work only from screenshots pay a high price: one screenshot costs thousands of tokens, and editing a spreadsheet can take dozens of clicks. Simply giving a frontier model both GUI and terminal tools does not teach it when to use which. On October 30, Microsoft's Mage team released Mage-VL-Cowork on Hugging Face and GitHub, aiming for a 4B model that decides for itself at every step whether to click or to code.

How it works

The model starts from an intermediate 4B Mage-VL checkpoint (Mage-ViT vision encoder plus Qwen3-4B-Instruct-2507) and goes through five training stages: visual grounding, GUI execution, and coding and terminal skills are trained separately first, then unified through hybrid SFT and asynchronous online RL. It has two action channels: computer_use (11 GUI primitives) and a persistent bash and Python environment. Each turn keeps only the 3 latest screenshots and 4 full responses, with earlier steps folded into action summaries. Microsoft says it first trimmed the teacher's harness (dropping xdotool and verbose accessibility trees, pre-installing libraries such as openpyxl and python-pptx), which lifted the GPT-5.4 teacher to 92.0% and gave the student an extra 12.5 points.

Benchmarks

All numbers are self-reported by Microsoft. On OSWorld-Verified (361 tasks excluding Google Drive, 50-step budget), two runs scored 65.33% and 66.75%, averaging 66.0%. In its comparison table, Qwen3.6-27B scores 63.9% and Muse Glimmer-30B 65.9%, but the 9B SCALECUA-Qwen3.5-9B still leads at 68.7%. Visual grounding rose from 30.6% to 56.9% on OSWorld-G and from 78.8% to 87.2% on ScreenSpot-v2. In Microsoft's demo, a Calc chart task finished in 39 seconds, versus 9:56 for GUI-only Qwen3.6-27B.

What developers can do

Weights are at microsoft/Mage-VL-Cowork on Hugging Face. Loading with transformers (4.57.3 or later) or SGLang requires trust_remote_code=True because it is not a standard Qwen3-VL architecture. Microsoft provides a pinned SGLang Docker image and a full OSWorld evaluation suite. Because the agent runs arbitrary shell commands, Microsoft says to run it only in isolated containers or VMs, never mount SSH keys, cloud tokens or browser cookies, and require human confirmation for irreversible actions.