Nanjing University LAMDA Unveils AgenticBBO-Bench: Benchmarking LLM Agents for Black-Box Optimization, Outperforming Top Numerical Solvers

Background and the Problem

Black-box optimization (BBO) underpins complex engineering workflows spanning VLSI chip floorplanning, distributed database tuning, hyperparameter optimization (HPO), and molecular drug discovery. In these domains, evaluation functions involve rigorous physical simulators or hardware clusters where gradients are unavailable and each trial incurs immense computational expense. Classical methods such as Bayesian optimization (GP-BO) or CMA-ES treat problems as purely numerical black boxes, ignoring code structure and domain semantics. Conversely, zero-shot prompting of foundation models suffers from hallucinations and numerical inaccuracy. Designing systematic frameworks where coding agents orchestrate numerical solvers to optimize objective functions under tight evaluation budgets is an urgent industry priority.

Architecture and How It Works

To systematically analyze agentic black-box search, Nanjing University's LAMDA Group and collaborating institutions present AgenticBBO-Bench (arXiv:2610.12183, repository: lamda-bbo/agentic-bbo):

1. Unified Protocol Across Five Critical Domains: Standardizes evaluation across synthetic test functions, machine learning HPO, database tuning, chip macro placement, and molecular optimization under a uniform finite-budget protocol.
2. Codex Agent Harness with Executable Tooling: Embeds foundation models in interactive environments featuring a Python interpreter, mathematical solvers (Scipy, Optuna), and real-time execution feedback. The agent synthesizes search heuristics, executes local optimization scripts, and refines search trajectories based on empirical returns.
3. Dissecting Core Drivers of Agentic Success: Isolates three critical factors, demonstrating that task semantics deliver robust acceleration across domains, arbitrary tool accumulation yields diminishing returns, and classical numerical optimizers effectively capitalize on exploratory global paths mapped out by agents.

Benchmarks and Measured Results

Benchmarked against classical numerical libraries and direct LLM methods under finite search budgets:

1. Outperforming Specialized Numerical Solvers in 4 of 5 Domains: Agentic BBO decisively outscores direct prompting across all five domains and surpasses state-of-the-art numerical solvers (including Optuna and CMA-ES) in hyperparameter tuning, database configuration, chip floorplanning, and molecular generation.
2. Pareto Frontier Analysis: Across seven frontier LLMs evaluated under the Codex agent harness, GPT models and DeepSeek lightweight models define the Pareto frontier balancing optimization quality against inference cost.
3. Over 38% Search Budget Reduction: On constrained evaluation budgets, Agentic BBO reaches high-performance target objectives in 38% fewer function evaluations compared to conventional Bayesian optimization.

Getting Started for Developers

The LAMDA Group has released the complete code, environments, and agent harness on GitHub (lamda-bbo/agentic-bbo). For platform architects and R&D teams building EDA pipelines or cloud resource optimizers, AgenticBBO-Bench demonstrates that foundation models should not act as bare parameter guessers. Instead, developers should pair coding agents with sandboxed Python runtimes and numerical libraries, leveraging LLMs to interpret domain documentation and coordinate numerical solvers for high-stakes optimization.