LiveMACE Benchmark Unmasks Agent Decision Blackboxes in Live Evolving Markets: Decoupling Compliance from Volatility

Background and the Problem

Most existing benchmarks for autonomous LLM agents rely on static, short-horizon, or synthetic environments, evaluating agents purely on downstream outcomes. However, in persistent, open-ended environments (such as live quantitative finance, enterprise DevOps, and autonomous supply chains), final outcomes are confounded by external environmental drift and stochasticity. An agent might achieve positive returns simply due to macro tailwinds despite hallucinated logic and reckless safety violations. The community urgently requires an evaluation framework that deconstructs the entire reasoning process. In arXiv:2610.09872, researchers presented LiveMACEBench, a process-aware benchmark using live evolving markets.

Architecture and How It Works

1. Naturally Evolving Real-World Testbed:

• Integrates live streaming market feeds and multi-asset price ticks, requiring agents to maintain persistent internal states, balance account equity, and execute multi-round closed-loop actions across weeks of live evaluation;

1. Three-Dimensional Process-Aware Evaluation Framework:

• Tool Use: Evaluates how agents decompose information gaps, route scans across the asset universe, and validate signals with news feeds;
• Persistent Memory: Measures memory retrieval coverage (MCQ), memory utility (MUQ), and empirical contributions to maximum drawdown (MDD) mitigation;
• Rule Execution vs Auditability: Decouples mechanical compliance (Hard Rule Compliance Rate, HPR) from verifiable reasoning (Rule Auditability Score, RAS), isolating instances where agents comply purely by accident without verifiable safety audits.

Benchmarks and Measured Results

Conducted comprehensive end-to-end evaluations on GPT-5.4, DeepSeek-V3.2, Gemini-3.1-Pro, Qwen3-Max, and Grok-4.20 (Table 1, Table 2, Figure 14):

• Overall Financial Performance & Safety Telemetry (Table 2):
• DeepSeek-V3.2: Achieves the highest final return at +5.765% with the lowest Maximum Drawdown (4.270%) and a solid HPR of 0.877, but shows a lower Rule Auditability Score (RAS) of 0.623;
• GPT-5.4: Reaches +5.738% return, but exhibits a higher MDD of 5.169% and lower HPR of 0.682;
• Gemini-3.1-Pro: Delivers frontier risk reasoning, scoring the highest Rule Auditability Score (0.882) and compliance rate (0.888 HPR), yielding +3.503% return and 4.927% MDD;
• Qwen3-Max: Yields +4.418% return with 5.674% MDD and 0.511 RAS;
• Grok-4.20: Generates +3.085% return with 5.165% MDD and 0.834 HPR;
• Critical Agent Failure Modes Uncovered:
• Silent Compliance: In 14 of DeepSeek-V3.2's 163 evaluated rounds, the agent executed compliant trades without citing a single safety rule or audit check, achieving mechanical compliance without verifiable evidence;
• Boundary Targeting & Increment Blindness: Both GPT-5.4 and Qwen3-Max treated maximum concentration caps (e.g., a 15% single-asset ceiling) as targets rather than upper limits. When an asset occupied 13.66%, the models placed incremental buy orders treating existing exposure as zero, driving post-trade positions up to 19.44%~28.04% in direct violation of risk parameters.

Getting Started for Developers

• Technical Report: Full trace analysis and benchmark protocols are detailed in arXiv:2610.09872;
• Architectural Recommendations for Production Agents: LiveMACE provides a vital warning for autonomous systems engineers: never delegate critical compliance and safety boundaries exclusively to LLM self-regulation. Production architectures must enforce deterministic external rule checkers and mandate structured Chains-of-Compliance in agent system prompts to prevent catastrophic boundary breaches caused by increment blindness.