Not saving experiments but saving decisions

The work I am reviewing here is Decision-Focused Active Learning for Scale-Aware Critical-Materials Recovery, a study that analyzes records from PNNL's CICERO workflow and is listed under arXiv:2609.09413. Its claim in one sentence is that batches of experiments should be chosen by how much they reduce downstream decision loss at scale-up rather than by how much they improve prediction accuracy. The original paper is linked under the title of this post.

Hello, this is Patrick.

I picked up this paper for a fairly practical reason. When people hear about rare earths or cobalt, they usually think of mines or supply chains, yet the moment that decides money comes later. There is a wide gap between a small beaker that worked in the lab and a decision to run the same route at tons of scale. What fills that gap is the question of this paper. So can we run fewer experiments and still reach plant decisions sooner?

What sits between a lab score and a plant decision

A good lab result usually holds under one condition. It says how well a given feed was separated into what we wanted. But once the conversation moves to a scale-up meeting, the questions change. People ask whether the purity meets product specs, whether the process makes money, and whether the result survives when the scale grows. That is why one sentence near the front of the paper stayed with me. Choosing a recovery process for scale-up requires connecting laboratory results with product requirements, process costs, and scale effects.

People make similar choices all the time. Getting one practice question right does not prove that the study method works for the whole exam. One correct answer is a score, while the whole exam strategy is a decision. This paper tries to move active learning into the language of decisions. Active learning uses prior results to choose experiments, and it usually probes where model uncertainty is largest. Here the rule is different. The next batch is chosen by how much it reduces decision loss downstream.

What interests me more than any single number is that structure. A model that predicts well and an experiment plan that supports decisions well are not the same thing. If a slightly better prediction leaves the decision unchanged, that experiment is worth little, while if a small improvement flips the decision, that experiment is worth a lot. Starting from that distinction feels right. So how does this view behave on real records?

How to say separation performance in one number

To follow the story we need to settle the word enrichment. In this paper, enrichment is the selected rare-earth-to-iron ratio relative to that in the feed. In plainer terms, it measures how much the selected side tilts toward rare earths compared with the starting mixture. When iron-rich magnet waste has to be sorted, that tilt has to be strong.

The definition is useful because the reference point is clear. Since it reports a ratio relative to the feed rather than an absolute concentration, we can compare conditions even when the incoming material varies a little. Of course one number never describes a whole process. Even high purity cannot support a plant if the recovered amount is tiny. So the paper treats the purity yield tradeoff separately later on. Numbers are reported one by one, and interpretation ties them together, and I liked that restraint.

Here an interesting question comes up. Is finding the condition with the highest enrichment quickly the same as making a good plant decision. At first glance it looks that way, but in practice it may not be. Hitting one record value and knowing which range of conditions earns money reliably are different tasks. That is why the paper reaches for Bayes risk. It looks at expected decision loss rather than the maximum score.

What the CICERO precipitation line actually recorded

CICERO is the name of the Computer Intelligence for Critical Element Recovery and Optimization workflow at PNNL, a line that runs autonomous selective precipitation. It applies different conditions across wells, reads the precipitation outcomes, and picks the next conditions in an automated loop. This paper reanalyzes records left by that line. It is closer to a retrospective contest among selection strategies on accumulated records than to a fresh large campaign.

This is where I would be careful not to over-read the experiment. Retrospective analysis can look smarter than it is because the answers are already on file. Pretending to choose after seeing the records is different from choosing the next well with genuine uncertainty. The paper seems aware of this, so it frames the test as a conditional retrospective benchmark. Instead of picking freely from all records, it conditions on recycled NdFeB magnet records and compares adaptive policies against nonadaptive space filling inside that boundary. Space filling simply covers the experimental space evenly, and it never changes course based on earlier results.

That contrast is honest because the baseline is diligent rather than foolish. Uniform coverage is the standard sincere method when there is little prior information. The question is how much sooner an adaptive method passes that bar. The records themselves also have weight. They are well level individual experiments from a real line, not synthetic data. Yet the retrospective limit remains. Choosing from history and choosing the next well in a live lab are different jobs, and I will return to this point.

What the gap between 16 wells and 48 wells tells us

On the conditional benchmark with recycled NdFeB magnet records, the result was fairly crisp. Adaptive policies reached the recorded enrichment maximum by 16 to 24 wells, while nonadaptive space filling needed 48 wells. In other words, the adaptive side found the same best recorded result with less than half the experiments. Placed side by side, the numbers speak directly. One side reads sixteen to twenty four, the other reads forty eight.

On the surface it looks simple. The adaptive side is faster. But the gap says more once you translate wells into operations. Wells mean individual experiments, so the difference turns into reagents, hours, and robot time. For someone running the line overnight, whether the answer arrives before all 48 wells are filled changes the next morning schedule. So I rate this result above a lab score. It protects the morning of operations.

Still, the conditions matter before anyone generalizes. The benchmark shows how quickly the best value already present in the records was found, not that a new unknown maximum was discovered. That is exactly what conditional retrospective means. Even so, the comparison itself is fair. Same records, same start, same goal. Then do adaptive methods differ among themselves. The next part answers that question.

What changes when the search is rebuilt in two stages

The authors' own reconstruction is a two-stage approach, and it tied two adaptive alternatives at 16 wells. That means their method belonged to the fastest group, and all three reached the recorded maximum by that point.

I looked at the structure before the numbers. Two stages usually mean a broad sweep followed by focused digging. Instead of probing precisely from the start, the plan narrows the promising region first and then firms up the decision inside it. It resembles fishing by casting wide to find where fish gather and then fishing hard in that spot. Such a structure fits tight experiment budgets because the early stage buys information that makes the later stage cheaper.

The next question follows at once. A tie at 16 wells shows early arrival, but it does not show later stability or decision quality. Three runners arriving together says little about whether they stay together afterward. The paper is fairly candid here. It does not inflate the early arrival, and in later sections it brings out a stricter ruler called decision loss. So how did it handle metrics that resist improving together, like purity and yield?

Why yield wobbles when SmCo purity rises

The conditional analysis on SmCo magnets shows a Round 2 tradeoff between purity and nominal yield, where nominal yield means the recovery fraction from an assumed starting amount. When conditions are tightened to select cleanly, some material is lost to the discarded side, and when conditions are loosened to recover more, impurities come along. That tug of war is the tradeoff.

Anyone familiar with plants will recognize the scene. The quality team pushes purity up while the production team defends yield, and both sides point at opposite ends of the same graph in a meeting room. The fact that the paper names Round 2 matters for this reason. This is not a one shot story. After seeing earlier results and adjusting conditions, the tradeoff became sharper in the later round.

What caught my attention here is the word nominal. Since the starting amount is assumed, the yield number moves when that assumption moves. The tradeoff itself is real, but the exact position of the curve stands on assumed ground. Writing that qualifier next to the number is a good habit. It shows the figure without hiding the floor it stands on. Then is there a case where the floor shakes even more. It comes next.

Why produced water rankings flip on one assumption

For produced water from oil and gas extraction, rankings depend on phase and dilution assumptions requiring confirmation. Which phase is assumed and how much dilution is assumed can change which source looks more promising. So the ranking should be read only after checking those conditions.

From a systems perspective, the next question is obvious. Water arriving at a real site comes in several streams, so which stream anchors the ranking. Dilution is also not a pipette issue in the lab but a plumbing and storage and mixing issue in the field. Rain or process changes can shift the dilution rate. That is why I read this passage as an honest part of the paper. It could have published the ranking and stopped, yet it wrote down the assumptions beneath the ranking and said they need confirmation.

The seemingly soft result actually connects to the main argument. When scale-up rankings rest only on lab scores, they shake whenever the assumptions shake. So what we need is not more scores but an experiment order that finds which assumption moves the decision first. That idea leads naturally into the decision focused criterion. Experiments that collect scores and experiments that settle decisions are different.

What happens when we choose experiments by downstream risk

Now we reach the proposal. The decision focused criterion chooses batches by their expected reduction in downstream Bayes risk, which is the minimum expected loss among available process decisions under current beliefs. The phrasing sounds heavy, but the meaning unpacks cleanly. It measures the remaining regret if we make the best decision we can with what we know now, and then it picks the experiment that shrinks that regret the most.

I like to compare it to an error notebook. There are two ways to raise exam scores. One is to memorize more of what you already know, and the other is to reduce the types of questions you miss. The second usually pays better. The Bayes risk rule works the same way. Instead of spending budget on average model error, it spends budget where decisions might be wrong. If a better prediction would leave the choice unchanged, that experiment waits, and experiments that split the decision move forward.

In exploratory simulations, a hybrid filtering policy shows lower estimated loss than joint search. Joint search covers a broad space together, while hybrid filtering mixes broad pruning with focused digging. Of course the word exploratory counsels restraint. Lower loss in exploratory simulations shows promise, but it does not guarantee a win in the field. Even with that caveat, the direction is clear. Moving the objective for experiment selection from accuracy to decision loss can buy better decisions with the same budget. So what else is needed to use this direction in a plant?

How I would use this work next

When my team looks at a new AI architecture, we do not look only at benchmark numbers. We ask what actually changed and what cost structure that change creates in a system. The same ruler fits this paper. What changed is the objective for choosing experiments, and the cost structure shows up in wells, rounds, and analysis effort. Reaching the recorded maximum around 16 wells reads directly as budget saved.

But applying it in the field leaves blanks to fill. We still need to see whether early arrival around 16 wells repeats in forward selection rather than retrospective replay, whether the SmCo tradeoff curve keeps its shape across feed compositions, and how quickly phase and dilution assumptions for tricky sources like produced water can be settled by field measurement. The last point is a data pipeline issue. If field phase fractions and dilution rates arrive slower than lab data, even a smart criterion will choose experiments on stale assumptions. A smarter model and timely data are different problems.

What I would watch next is threefold. First, I want to see forward well selection repeat the arrival around sixteen wells. Second, I want the purity versus nominal yield tradeoff redrawn in the language of decision loss, because one point of purity and one point of yield rarely carry the same value in a plant decision. Third, for assumption sensitive sources like produced water, I want to see how much loss is actually reduced by experiment batches ordered by sensitivity. Once those three are filled, a good retrospective record can become a good field habit.

So the question that remains is this

Looking back, this paper is not a showcase of scores. It reports that adaptive methods beat uniform search, yet its weight keeps leaning toward decisions. Beyond finding the best record sooner, it asks how cheaply that record settles the downstream choice. That is why the enrichment definition, the SmCo tradeoff, and the produced water assumptions read as one thread rather than separate stories. They answer what is measured, what shakes, and what should guide the next pick.

So what is the human role. Even when an automated precipitation line picks wells and a robot moves the pipette, people still decide which purity meets product requirements, which yield balances plant profit, and which assumption to settle first. Inside the phrase decision loss there is ultimately a human profit calculation. That is why I read this work less as progress in lab automation and more as an early sign of decision automation. It resembles less a machine that runs experiments and more a machine that measures regret about decisions.

The next time I see similar records, I expect to ask the same question. Not how much one batch raised the score, but how much cheaper it made the decision. When records that answer that question accumulate, the morning of the lab and the morning of the plant may finally move at the same pace.