Sarthak Gupta

Applied ML

What limits coding agents on open-ended ML experimentation?

October 11, 2026

Sarthak Gupta

Sarthak Gupta

Software & ML Engineer

Intro

As a Research Engineer intern @ Experian DataLabs, I wanted to know whether the agents topping benchmarks could realistically improve an existing tabular model, so I built benchmarks and tested systems against it.

Google's MLE-STAR paper reported a 63.6% medal rate on MLE-bench Lite with an agent that searches the web for models, ablates its own pipeline, refines targeted code blocks, and builds ensembles.

I now think the more useful question is a different one: when the benchmark, the model baseline, and the evaluation criteria are fixed, what limits a general-purpose coding agent's ability to run productive, open-ended ML experiments? That question breaks into variables that can be tested separately, and I examined some of them much better than others.

  • Benchmark: Early mortgage delinquency (60+ days past due within 12 months of origination), with a 0.884% positive rate, a chronological train/validation/test split, and an XGBoost baseline.
  • Evaluation contract: The agents could edit one file. They could not touch the benchmark, the leakage checks, or the held-out test set, which was only evaluated after selection.
  • MLE-STAR: I rebuilt it to run outside Google Cloud and spent most of the effort making its failures visible and fixable. The runs did not produce a substantial gain over the baseline.
  • Codex with beam search: The best candidate increased held-out PR-AUC from 0.045093 to 0.045265, a 0.38% relative lift against a 0.0500 target. The search reached that candidate on its fourth try and then stopped producing patches. The observed proposals grew increasingly similar, and four candidates are too few to say anything about the size of the underlying hypothesis space.
  • Variables: I examined execution reliability in depth, and hypothesis breadth, throughput, statistical confidence, and agent context only partly. CI/CD integration and a compute-to-lift frontier were not tested.
  • The challenges I faced overlapped with Meta's A-MLE effort, view here.

The question and the variables

A colleague at Experian DataLabs was working with a dataset that had client data so I couldn't immediately access it. I looked for a public dataset with a similar rare-event risk-ranking shape and used Fannie Mae's single-family loan performance data to model early delinquency. Mortgage delinquency differs from telecom fraud in its causes, costs, and available signals. It works as a proxy because it has the same class imbalance and the same need to rank a small set of high-risk observations.

I started with this question:

Given a human-directed XGBoost baseline, can an autonomous coding agent generate valid, material improvements under a fixed modeling objective, or does it mainly accelerate trial-and-error around the existing solution?

A lift number from one benchmark says little about why an agent succeeded or failed. Holding the benchmark, baseline, and evaluation fixed turns the question into variables that can be tested separately, and the rest of this post is organized around them:

  • Execution reliability: does the agent produce code that runs to completion? Examined in depth (truncated output, timeouts, repair loops, gateway retries).
  • Breadth of hypotheses: does the agent propose different ideas?
  • Usable experimental throughput: how many valid experiments come out per unit of time and effort? Examined partly. I have run durations and no count of valid experiments per hour or of human time.
  • Statistical confidence: how much of a measured lift is signal? Examined partly. A five-seed sweep covers validation only, and the test lift is a single measurement.
  • Available context: does what the agent knows about the task change what it proposes? Examined partly. I fixed missing task context in MLE-STAR. Giving Codex a modeling brief and a history of earlier experiments was not tested.

Benchmark design and its trade-offs

Each choice below had alternatives with real costs. I draw them from the facts of this benchmark and label standard reasoning as such.

The horizon

I first tried 90+ days past due (DPD) within 24 months, then a 36-month horizon and broader serious-credit-event definitions. Changing the horizon changes the number of positive events, so a higher precision number after such a change says nothing about whether the algorithm improved. I settled on 60+ DPD within 12 months of origination, which gave a fixed positive rate I could hold constant across every system.

Labels need a known observation window

The raw data was about 361.9 million monthly performance rows for 8.74 million loans. I consolidated them into one row per loan, with the outcome derived from later payment history.

A loan is positive if it reached 60+ DPD during its first year. It is negative only if it never reached that threshold and was observed for the full year. A loan observed for eight months without delinquency has an unknown outcome, so it isn't a confirmed negative. This is a right-censoring problem.

There were three options. Treating unresolved loans as negatives keeps every row but mislabels loans that might still have become delinquent. Survival modeling uses partial follow-up correctly but changes the task into something the baseline and both agent systems weren't built for. Excluding unresolved loans costs data and could shift the population, which I did not measure. I excluded them because that kept every label correct and left a plain classification task.

Fannie Mae361.9M monthly rows8.74M loansOne row per loanevent timing,12-month outcomeLabel filterPositivereached 60+ DPD within 12 months73,710 loansNegativeobserved 12 months, never 60+ DPD8,262,290 loansExcludedwindow not fully observed400,260 loans25 featuresorigination-time onlyleakage fields droppedTrain2021 to 20226,514,654Validation2023921,658Test2024, held out899,688
Figure 1: From monthly performance records to the benchmark dataset. A loan is a positive if it reached 60+ DPD in its first year and a negative only if it was observed for the full year without doing so. Loans with an unresolved window are excluded, which leaves 8,336,000 eligible loans and a 0.884% positive rate.

That left 8,336,000 eligible loans: 73,710 positives and 8,262,290 negatives with complete follow-up. 400,260 loans were censored or otherwise excluded. The positive rate was 0.884%.

Choosing the selection metric

With a 0.884% positive rate, accuracy is a poor objective, since predicting no delinquency for every loan scores above 99% without finding a single positive. The candidates were PR-AUC (average precision), ROC-AUC, KS, and precision among the highest-risk 1% of loans.

  • ROC-AUC is insensitive to class prevalence, so on a rare-event problem it can look good while precision among flagged loans stays low (standard reasoning).
  • KS measures the maximum separation between the score distributions, which is one point on the curve.
  • Precision@top-1% maps to a segment a business could act on, but it depends on a single cutoff and gives an optimizer a coarse signal.
  • PR-AUC summarizes the precision-recall tradeoff across all thresholds and reflects the low prevalence directly.

I chose PR-AUC for selection and kept the other three as diagnostics. My first search run optimized KS, and I switched to PR-AUC once the container path worked. The cost of the choice shows up in the results: the best candidate improved PR-AUC and KS and slightly lowered precision@top-1%.

Features, split, and baseline

The feature matrix was limited to information available at origination, which gave 25 modeling features (14 categorical, 11 numeric) after I dropped one origination FICO field that was entirely missing. Fields describing future delinquency, label status, loan identity, or observation completeness were excluded.

I split chronologically:

  • Train: 2021 to 2022, 6,514,654 loans
  • Validation: 2023, 921,658 loans
  • Held-out test: 2024, 899,688 loans

The alternative was a random split across origination years. A random split gives every partition the same distribution but lets the model learn from loans that originated after the ones it is scored on, which tends to give optimistic estimates (standard reasoning). The chronological split costs some comparability between partitions: baseline PR-AUC was 0.039528 on validation and 0.045093 on test, so the two years differ in difficulty and a validation number cannot be read as a test number. I accepted that because the question was whether patterns learned from earlier cohorts transfer to later loans.

Before involving an agent, I compared conventional models on held-out test PR-AUC. XGBoost scored 0.0451, logistic regression 0.0261, and LightGBM 0.0157. XGBoost became the baseline. It was never deployed. I call it production-style because I applied production-relevant discipline to target construction, point-in-time features, temporal evaluation, and reproducible candidate comparison.

The evaluation contract

Both agent systems ran under the same rules, and these rules are what make the results believable.

  • Editable surface: Most successful candidates changed only solution.py.
  • Selection metric: benchmark.py wrote validation PR-AUC to artifacts/metric.json, and that file was the only signal the optimizer saw.
  • Leakage checks: contract_test.py verified that loan_id and target_60dpd_12m stayed out of the model's feature matrix.
  • Test set isolation: Nothing in the search loop was able to read the held-out test set. final_evaluate.py was a separate script, run only after selection.
  • Fitting discipline: Missing-value handling, encoding dictionaries, frequency maps, and interaction transforms were fitted on training data only.
Coding agentMLE-STAR orCodexEditablesolution.pymodel codefitted on train onlyRun by the harness, lockedbenchmark.pyfits on trainscores validation PR-AUCcontract_test.pyloan_id and target stayout of the feature matrixfinal_evaluate.pyscores the held-out test setrun once, after selectionmetric.jsonvalidation PR-AUConly signal the agent seesfeedback to the next candidate
Figure 2: The evaluation contract. The agent can edit one file. The benchmark, the leakage checks, and the held-out test script sit outside its reach, and the only number it sees is the validation PR-AUC written to metric.json. The test set is scored once, after selection.

Two of these choices carry costs worth stating.

One editable file. The alternative was letting the agent edit the whole repository. That widens the space of possible improvements but makes each candidate harder to audit and gives the agent a path to the benchmark itself. One file keeps the diff small enough to read after every run. The cost is a narrower search space, which I list below as one explanation for the plateau.

Reserved test set. The alternative was to score every candidate on the test set too. That gives the optimizer more feedback, but once the test set influences selection it no longer gives an unbiased estimate of the chosen model. Reserving it means the test lift is one measurement taken after selection, so I have no spread for it.

As of Oct 2026, advanced agents can break past these as OpenAI's agents did to HuggingFace, however, in my work during June 2026 the guardrails above made it harder for a candidate to improve the metric by leaking the target or changing the evaluation and results were reviewed after each run to ensure no leakage.

Running MLE-STAR

I chose Google MLE-STAR because its authors made concrete choices about the weaknesses of earlier agents. It retrieves model ideas through web search, uses ablation to decide which pipeline component deserves attention, runs nested refinement loops over that component, and builds ensembles. It also includes modules to debug failed code, inspect potential data leakage, and check that the solution uses the supplied data. The paper's results come from 22 MLE-bench Lite Kaggle competitions under a 24-hour limit, so they don't guarantee improvement on an existing repository with a fixed temporal credit-risk benchmark.

Model retrievalweb search2 candidate modelsInitial solutions2 solutionsone per modelAblationrank pipeline blocksby impactRefinement1 outer loop1 inner loopEnsemble1 loop, planner picksstrategy, best by valApplied to every generated scriptDebuggerup to 5 repair roundsper failed scriptLeakage checkflags target informationin the featuresData-use checkconfirms the solution usesthe supplied data
Figure 3: MLE-STAR as I configured it for this benchmark, with the open version that runs outside Google Cloud. Every generated script passes through the debugger and the leakage and data-use checks, and all LLM calls go through the internal AI gateway with retries and backoff.

Making it run in our environment

MLE-STAR assumes access to Google Cloud Gemini. I had to build an open version that preserved its workflow and ran independently, and route its LLM calls through our internal AI gateway so that prompts, model selection, retries, and responses followed the same path as other internal tools.

Agent workflows produce many LLM calls in a short period. The gateway allowed 200 requests per minute, which was enough for these runs, and I still added retries with backoff so that one failed call wouldn't end a run that lasted 30 to 45 minutes. I used GPT-5.5, Gemini Pro 3.1, Gemini 3.5 Flash, and Claude Opus 4.7 (SOTA models in early June 2026) for full runs of roughly 50k to 100k tokens each.

Execution reliability

Early runs didn't reliably produce complete, executable training code, and the final output didn't say whether the cause was the generated code, a timeout, or the model interaction. I had Codex add logging around the API exchanges and read the JSON traces. They showed an output-token limit truncating the generated code, which left later iterations with a broken starting point. I raised the output budget, then found the agent was also missing context it needed for parts of the task, and fixed that input too.

Many failures appeared downstream of their original cause, such as a weak prompt, a malformed intermediate file, or a failed call. To catch them earlier I added validation for generated code (does it parse, does it have the expected structure, does it run) and wrote a script that streamed the agent's reasoning, generated code, and token consumption live while a run was in progress. Before integrating MLE-STAR or any other agent, I also ran it repeatedly against representative workloads and failure cases, and defined a common runner contract with clear completion signals, because some agents could return a response before their artifact was usable.

Capping tokens or watching live. A token cap bounds cost and stops a runaway loop early, but it can also cut off a run that was making progress and hides what the agent was doing at the limit. I chose not to cap tokens so the agent could use all the room it wanted, and I watched the stream instead. That is how I found a failure a cap would have hidden. Twice, an agent wrote error-prone code and kept trying to repair it instead of moving on. The failed output then hurt the agents downstream, and the loop used up the entire token budget. Those two budgets were the price of the choice.

Even after these fixes, candidate scripts such as ablation_0.py and train0_improve0.py hit 900-second execution limits and went to repair attempts.

What the runs showed

My configuration was narrower than the paper's setup, as mine focused on avoiding context bloat by having two model candidates, two solutions, one outer refinement loop, one inner loop, one ensemble loop, and up to five debugging rounds. It should not be read as a reproduction of the paper's main setup, which allows 24 hours per competition. I kept full runs to 30 to 45 minutes because later rounds stopped producing useful ideas. What the agent proposed late in a run was either a marginal gain or a more tangled train.py that I wouldn't expect to survive review for production. I never ran a 24-hour configuration, so I can't rule out that it would behave differently, and I can't compare my numbers with the paper's.

I checked this judgment in two ways. I had Cursor and Codex launch sub-agents to look for flaws in the approach, explain why it plateaued, and propose changes, and I read their proposals before they started implementing them. They applied many different changes and reached the same ceiling. I also asked two data scientists on my team, who said that marginal gains were what they would expect, since improving a model on a given metric usually takes more data with relevant features. In light of being open to further exploration, I treat the plateau as an observation about this benchmark instead of generalizing it as a one-all outcome.

MLE-STAR's last stage generates multiple solutions in parallel, has a planner propose ensemble strategies such as stacking or weighted averaging, and keeps the best one by validation score. Google's paper reports that this stage lifts the any-medal rate from 37.9% to 43.9% with Gemini-2.0-Flash. The ensembles I got were complicated, which is what the method is built to produce. The paper's prompts frame the agent as a Kaggle grandmaster trying to win a competition, and the only objective is the score, so a complicated ensemble is a reasonable trade for a medal. My reading is that production models also need maintainability, latency, and interpretability, so adapting MLE-STAR would mean changing what its search selects on, and removing the Kaggle wording from the prompts would not be enough. I did not modify it, so this is a judgment and not a result. On my benchmark a stacked ensemble for a tiny PR-AUC gain is hard to justify.

One run report recorded a selected ensemble candidate (ensemble_0) with a slightly better test KS than the baseline, about 0.474997 versus 0.474591. My notes also flagged inconsistencies in the run records and artifact provenance, so I don't consider that result verified enough to claim a robust improvement.

GPT-5.5 and Opus 4.7 both reached executable code, proposed similar kinds of ideas, and neither beat the baseline by a meaningful margin. I also tried Gemini 3.1 Pro / 3.5 Flash, however its code generation was too poor to use. The bounded runs didn't establish a reproducible gain, which says nothing against Google's published results since my setup had different constraints.

Next I tried a different structure: a frontier coding agent with beam-search-style branching over candidate code changes. The search infrastructure is the open-source codex-optimize project. I adapted its beam search, Git worktree handling, and container isolation to this ML experiment.

Existing XGBoostcode and fixedbenchmarkCandidate round, in parallelBranch Agit worktree + containerrunning one Codex agentBranch Bgit worktree + containerrunning one Codex agentBranch Cgit worktree + containerrunning one Codex agentEvaluationBenchmark and testsfresh containernetwork disabledReset in gitbenchmark and testsrestored after each editKeep best validbranches, thenrepeat or stopnext round
Figure 4: One round of the Codex beam search. Each branch gets its own git worktree and its own container with one Codex agent, and branches run concurrently. Benchmarks and correctness tests run in separate short-lived containers with networking disabled, and the benchmark and tests are reset in git after the agent edits.

How candidates are sandboxed

codex-optimize gives every candidate its own git worktree and its own container running one Codex agent, and it launches all candidates in a round concurrently. Each benchmark and correctness-test run then happens in a separate short-lived container started with networking disabled. The tool also keeps the source code, the benchmark, and the correctness tests as separate parts in git, so the benchmark and tests can be reset after an agent edits the code. Its README gives the reason: agents tend to cheat benchmarks, even unintentionally. This matched what my contract was trying to enforce.

Running agents directly on the host is simpler, but codopt starts Codex with approvals set to never and full filesystem access (details in the next subsection), so the container is the only isolation. Worktrees alone would isolate files but not the process. The cost of containers is setup, memory use, and a container runtime.

The tool runs on Docker, and we didn't have a commercial Docker license, so I had Codex port its container calls to Podman. The alternatives were obtaining a license or dropping containers, and dropping them would have removed the isolation the design relies on.

Running that many containers at once was a memory problem on my MacBook, which had 48 GB of RAM with about 20 GB taken by the system and other apps. codopt runs the containers concurrently with no per-container memory cap. I limited the number of branches to fit. The options were fewer branches, sequential candidates (slower), or more memory. I also ran the search on an m5.8xlarge EC2 instance with 128 GB of RAM. The improvement was marginally the same as on the laptop, which suggests memory and compute weren't what limited the search. That was one comparison on one benchmark.

What Codex's system prompt tells the agent

Every candidate ran with Codex's built-in instructions underneath codopt's task prompt, so those instructions shape what the search can propose. The Codex CLI base instructions are open source, and codopt reaches Codex through its SDK. Model-specific prompts and the desktop app's prompt differ in places, so the points below come from the open-source base instructions. Several parts matter for this task:

  • Persistence. The agent is told to keep working until the problem is fully resolved, and to verify instead of guessing. That is helpful for ordinary coding. In an optimization loop it could keep an agent on a failing approach, similar to the repair loops I saw in MLE-STAR, although MLE-STAR ran on different models and prompts.
  • Minimal edits in an existing codebase. The prompt separates new projects, where the agent should be ambitious, from existing codebases, where it should do exactly what was asked with surgical precision, keep changes small and consistent with the existing style, and leave unrelated problems alone. Our task was an existing codebase with an existing model. This is a plausible reason the proposals stayed close to the current solution, such as preprocessing tweaks and XGBoost tuning, although I didn't test it by changing the prompt.
  • Validation behavior depends on the approval mode. The base instructions tell the agent to run tests and lint proactively when approvals are set to never. codopt's own prompt pulls the other way, telling the agent not to spend its turn budget running benchmarks or tests unless essential, because the harness runs them. The task prompt is the more specific instruction, and that was the one I wanted.
  • Git rules. The base instructions say not to commit or create branches unless asked, and codopt repeats the rule; the harness owns the branches.
  • Sandbox and approvals. codopt starts Codex with approvals set to never and filesystem sandboxing set to full access, so the agent's own safeguards are off. The container is the only isolation. That is why I cared about the container setup: which paths were writable, whether the evaluation ran without network access, and keeping the benchmark and tests resettable in git.
  • Repository instructions. Codex reads AGENTS.md files and follows them for any file it edits. It is a natural place to put a modeling brief and a history of earlier experiments, which is the follow-up I describe at the end.

The search and its results

An initial run optimized KS and confirmed that the container execution path worked, which wasn't yet a useful model result. I then switched selection to validation PR-AUC.

The best result was small:

MetricSplitBaselineBest candidateChange
PR-AUCValidation0.0395280.040108+0.000580
PR-AUCHeld-out test0.0450930.045265+0.000172
KSValidation0.4466870.449529+0.002842
Precision@top-1%Validation7.540%7.486%−0.054 pp
Table 1: Baseline against the best candidate. Validation metrics guided selection, and the held-out test PR-AUC was evaluated after selection. The test lift is 0.38% relative.

The 0.38% relative lift in test PR-AUC fell well short of the 0.0500 target, and it didn't improve every ranking diagnostic. The candidates, in order:

CandidateChangeValidation PR-AUCVs baseline
FANNIE-XGB-001Baseline0.039528—
FANNIE-XGB-002Missingness handling and feature cleanup0.039753+0.000225
FANNIE-XGB-003PR-AUC-oriented XGBoost fit0.040090+0.000562
FANNIE-XGB-004Preprocessing and encoding changes (best)0.040108+0.000580
FANNIE-XGB-005Next branch, no code changes——
Table 2: Candidates in the order the search produced them. Candidate 004 was the best observed, and candidate 005 made no code changes.
Validation PR-AUC0.03950.03970.03990.04010.04030.039528001Baseline0.039753002Missingness0.040090003PR-AUC fit0.040108004Encoding005no patch1 SD0.000067five seeds
Figure 5: Validation PR-AUC of each candidate in the order the search produced it. The bracket at right is drawn to the same scale and shows one standard deviation (0.000067) from the five-seed sweep. The step from 003 to 004 is +0.000018, well inside it.

A later round returned node produced no code changes, and the final run record had winner_node_id = null despite the earlier improved candidates. The training code had worked. The search stopped producing viable patches.

Hypothesis breadth and the plateau

About 1 in every 4 of an agent's proposals were near-duplicates of earlier ones. Follow-up experiments covered native-categorical CatBoost, bounded interaction features, alternative positive-class weighting, and monotonic score transformations, and none beat the best validation PR-AUC. The observed search produced increasingly similar proposals. The key idea I observed is to look at when the underlying hypothesis space became exhausted, and how a different prompt, a different edit surface, or a deeper search could have gone elsewhere.

I see three plausible explanations:

  1. The agent kept proposing related preprocessing and XGBoost-tuning ideas instead of substantially different hypotheses. The duplicate rate and the similar behavior of GPT-5.5 and Opus 4.7 point this way.
  2. The edit scope and benchmark limited how far it could depart from the existing solution, since nearly all changes stayed inside solution.py.
  3. The existing features may not have contained enough extra signal for incremental tuning to produce a large gain.

The larger EC2 instance with more RAM gave the same result, so we can rule out that more compute wasn't the constraint.

Conceptual limits of searching over code

The next points are reasoning about how beam or tournament search over code edits behaves in general, including codex-optimize and, as I understand it, the nested refinement loops in MLE-STAR. They are conceptual considerations that tie into what I saw where I could.

Locality. Every candidate is a child of a retained parent, so the search explores the neighborhood of the current program. Reaching a different model family takes either one large edit or a chain of edits that each score well. Candidates 002 to 004 were all local edits (missingness handling, a different fit objective, encoding changes), which fits this picture and does not show that a distant region held anything better.

A noisy scalar can prune stepping stones. The selection signal is one validation number per candidate. An edit that only pays off in combination with a later edit looks neutral on its own and may be dropped before the second one arrives. The five-seed sweep gives a standard deviation of about 0.000067, and the step from candidate 003 to 004 was +0.000018, well inside that spread if the spread applies to those candidates. A beam that ranks on differences that small is partly ranking noise. This is the one point where the post's numbers give direct support, and it rests on two numbers without a significance test.

Most edits are neutral or harmful. Among syntactically valid edits, most change nothing that matters or make the model worse (standard reasoning), so branching factor is spent on near-duplicates. About 1 in 4 proposals were near-duplicates.

Edit surface and the agent's prior. A single-file surface and a base prompt that favors small, style-consistent changes both push toward incremental edits. I varied neither, so I cannot say how much each contributed.

Data and features are fixed. The search edits code against a fixed dataset and feature set. The two data scientists I asked said that improving a model on a given metric usually takes more data with relevant features, and the edit surface could add neither. That would bound what any code-edit search can reach on this benchmark, and it supports the third explanation above.

Selection on validation score. Choosing the best of many candidates by validation PR-AUC is a multiple-comparisons problem, because the winner is partly the candidate with the luckiest validation result. That is why I reserved the test set. The validation lift was +0.000580 and the test lift +0.000172, which is consistent with some selection optimism, but validation and test are different years with different baseline levels, so the two are not separable.

Depth. The search had a few candidates and a few rounds. A longer search with a larger beam would test every point above.

Statistical confidence

A five-seed diagnostic sweep gave a mean validation PR-AUC of about 0.040052 with a standard deviation of about 0.000067. The validation lift of 0.000580 is about nine times that standard deviation, so it is unlikely to be seed noise. I measured seed variability on validation only, so the +0.000172 test lift is a single measurement, and none of this tests generalization across independent datasets. Repeating the final evaluation across seeds and across additional temporal cohorts would be the first thing to add.

A separate smaller experiment compared bare Codex with a Codex-plus-Research-OS configuration on a different benchmark, and bare Codex scored higher on validation PR-AUC (0.4862 versus 0.4698) and precision@top-1% (81.93% versus 78.31%). That result cannot be compared with the mortgage result, but it suggests that adding orchestration doesn't guarantee improvement.

How I would evaluate these systems in a team setting

Developer velocity. The question is whether the agent shortens the time from idea to a verified result. The quantities to measure are valid experiments per hour and human interventions per run. I have run durations of 30 to 45 minutes for full MLE-STAR runs. The time I spent on logging, validation, the runner contract, and the Podman port is velocity cost that I did not track in hours.

Compute-to-lift. The axes are compute spent (tokens, wall-clock, machine size) and held-out lift. The few points I have come from different systems: full MLE-STAR runs of roughly 50k to 100k tokens with no meaningful lift, and a Codex beam search with +0.38% relative test PR-AUC that gave about the same result on a 128 GB EC2 instance as on a 48 GB laptop. They do not trace a frontier. Drawing one would need the same benchmark, time budget, and evaluation rules for every system, repeated across seeds.

Integration barriers. The ones I met were licensing (no commercial Docker license, hence Podman), gateway limits (200 requests per minute, plus retries), sandboxing (approvals off and full filesystem access, so a container runtime with network-disabled evaluation is a prerequisite), the evaluation contract (benchmark, leakage checks, and test set locked away from the agent and reset after each edit), verification of results (seed variance, and run records whose provenance I could not fully verify, as with ensemble_0), and code that a reviewer would accept (the more tangled train.py files that late rounds produced are a barrier on their own). How these behave inside an enterprise CI/CD pipeline is untested.

That entire month, I wondered whether I was doing something wrong. The following month in September, Meta launched a paper on Agentic ML Exploration (A-MLE) for ads ranking (Gao et al., 2026). Their paper describes a single agent that works through five stages (hypothesis generation, exploration strategy, experiment execution, result analysis, and a shared knowledge substrate), runs against a sandboxed execution layer, and has human review checkpoints between stages. That is a much larger system than my single-benchmark experiment, but its reported failure modes overlap with the ones I ran into:

  • Over-confident results. Their agent sometimes promoted a candidate based on a single seed, and they added an automatic re-run when within-run variance is high. I ran a five-seed sweep for the same reason, although my test-set lift is still a single measurement.
  • Infrastructure errors read as model failures. Transient incidents looked to their agent like training divergence, so viable candidates were abandoned, and they extended the retry loop to tell the two apart. My equivalent was truncated output and timeouts that looked like poor model quality.
  • Invented code. Their agent sometimes called plausible but non-existent functions, which pre-flight checks catch before a run launches. I added validation of generated code for a similar reason.
  • Differences between LLMs. With the agent loop held fixed, they found that Claude, Gemini, and GPT models differed in execution reliability and in how aggressively they explored. In my runs GPT-5.5 and Opus 4.7 behaved similarly, and Gemini 3.1 Pro's code generation was too weak to use.
  • A different result, with different levers. They report gains where my search plateaued: a +2.557% relative improvement on one production ranking model with their multi-source variant, and improvements on a majority of the models they evaluated. Their system had levers mine did not, including technique transfer across a portfolio of models, architecture-level edits, domain-specific skills, and human review. My search could only edit one file against fixed data and features. The comparison is not like for like, but the contrast fits the question of what limits these agents.

Their central conclusion is that an agent's reliability depends more on the surrounding harness (skill coverage, statistical rigor in evaluation, and resilience of the execution layer) than on the base model's reasoning. That is the closest published statement of what I took from my own runs.

What I took away

At Intuit Credit Karma my engineering work had a clear production outcome, which was improving a running inference system's performance and cost. The Experian DataLabs work tested claims about an emerging research capability, so I had to decide how to test them and what the results justified.

Three lessons came out of it, and they apply to anyone evaluating an optimization agent:

  1. Check the harness before judging the agent. Truncated outputs, missing context, and timeouts all looked like poor model quality until I read the traces.
  2. Separate selection data from final evaluation, and enforce it in code. The contract tests and the post-selection evaluation path are what make a 0.38% lift believable.
  3. Measure how varied the agent's proposals are. The observed search produced increasingly similar proposals after four candidates, and a final PR-AUC alone would not have shown that.

If I repeated the investigation, I would run a controlled comparison with the same dataset, edit surface, baseline, metric, time budget, and evaluation rules for every system, and measure valid experiments per hour, distinct hypothesis families, repeated proposals, human interventions, and execution failures. The variable I most want to test is context: bare Codex against Codex given a modeling brief and the history of previous experiments. Does it propose substantively different tests, avoid repeated branches, and improve held-out performance? I didn't build or validate that during my internship. A second test would vary the edit surface and the minimal-edit prompt separately, to separate the explanations for the plateau.

There are three levels of success here: running an experiment correctly, measuring an improvement honestly, and finding a useful new modeling direction. The tools I tested could do parts of the first, my benchmark and checks supported the second, and on this problem I found no convincing evidence of the third. A breakthrough model would have made a more dramatic post. The open question I would keep working on is how to get agents to propose genuinely different hypotheses, and which of the variables above limits them most.

Run an experimentagent writes, runs, anddebugs the training codeMeasure honestlylocked benchmark, test setscored once after selectionFind a new directiona different modeling ideathat beats the baselinepartly: after I fixed truncatedoutput and missing contextyes: +0.38% relative test PR-AUC(+0.000172)not on this benchmark: searchstalled after candidate 004
Figure 6: The three levels of success, left to right. The tools reached the first after I fixed the harness, the evaluation setup supported the second, and I did not see the third on this benchmark.

References and scope notes

  • Google MLE-STAR: Nam et al., MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement, arXiv:2506.15692v3 (2025). https://arxiv.org/abs/2506.15692
  • Meta A-MLE: Gao et al., Agentic ML Exploration (A-MLE) for Ads Ranking, arXiv:2609.08248v1 (2026). https://arxiv.org/abs/2609.08248