Putting outside evaluators inside artificial intelligence companies offers an appealing response to concerns about safety: bring scrutiny closer to the technology being developed. Yet proximity is not the same as reliable oversight. Irene Solaiman, Hugging Face’s chief policy officer, has challenged the assumption that embedding assessors will provide a straightforward solution. Her argument is not simply about who performs an evaluation. It concerns whether the tests themselves are sufficiently mature, comparable and reproducible to support the confidence placed in them.

The central tension is between an oversight arrangement that companies are beginning to adopt and an evaluation discipline that, in Solaiman’s account, still lacks consistent standards. Anthropic has selected Accenture as its first embedded evaluator, giving the approach a concrete institutional form. Solaiman, meanwhile, says evaluator qualifications need greater standardisation and that assessing agents is difficult and costly. These developments can coexist: a company can introduce outside assessment while the methods required to make that assessment convincing remain under development.

The interest in embedded evaluators has grown over recent months as AI laboratories confront safety incidents and concerns about the potential harm their technology can cause. Hugging Face itself has been the target of such an incident. The proposed arrangement places third-party observers within AI organisations, rather than treating evaluation solely as an external exercise. That brings observers inside the organisations whose systems they assess. But the arrangement alone does not establish what observers can inspect, which questions they can answer or how others should judge their findings.

Read more: France’s security hedge exposes Europe’s defense dilemma

For Solaiman, reproducibility is central to that judgment. An evaluation result becomes more useful as evidence if it can be reproduced, rather than accepted solely on the authority of the assessor. She connects that requirement to adequate access to the model and highlights the importance of open weights. This shifts the discussion from the evaluator’s institutional location to the conditions under which testing occurs. Model access and the reproducibility of a result are therefore separate questions from where an evaluator is based.

That distinction separates the organisational and methodological questions. Anthropic’s appointment of Accenture represents a practical move towards putting an outside organisation into the evaluation process. Solaiman’s caution concerns the reliability of what such a process produces. Naming an assessor identifies who will perform the evaluation; that is different from showing that its methods are appropriate. Conversely, incomplete testing standards do not establish that embedded assessment has no value. The unresolved question is how much assurance any particular evaluation can reasonably provide while the science continues to evolve.

Accenture’s selection drew criticism from people who questioned whether the consulting firm was qualified for the assignment. Those objections cannot, on their own, establish that it lacks the necessary capability. They do, however, illustrate the problem Solaiman identifies: qualifications need to be assessed against particular tasks, with more consistent standards. The relevant question is therefore narrower than whether a familiar organisation inspires confidence. It is whether the evaluator has the expertise and methods needed for the specific behaviour or system it is examining.

Read more: Ueda says further rise in underlying inflation above 2% is unacceptable

Comparability presents another obstacle. Solaiman says the field does not yet know how to compare evaluations meaningfully, and that tests can lose effectiveness relatively quickly. She also sees an evaluator ecosystem that is less developed than it needs to be. Taken together, those concerns complicate the interpretation of results. An assessment may be informative within its own limits without providing a dependable basis for comparison with another model or system. Embedding more evaluators would not automatically resolve those differences in method or durability.

The object being tested is also expanding. Evaluations are moving beyond the model to include its harness and the agent as a whole. The harness is the tool through which a user accesses the model, and its contribution can materially change performance. Solaiman says a weaker model supported by a better harness could outperform a frontier model. That observation challenges any assumption that evaluating the underlying model is sufficient to understand the capability of the system a user actually encounters.

This broader scope strengthens the case for detailed assessment while making it harder to deliver. Solaiman says evaluations of models are more plentiful than those of harnesses, while better assessments of complete agents are still being developed. She also describes agent evaluation as expensive and technically demanding. The resulting challenge is to assess not just the model but also the system built around it. Those complexities reinforce Solaiman’s caution: adding an evaluator cannot be treated as equivalent to having a comprehensive testing framework.

What comes next therefore matters more than the label attached to the assessor. The issues Solaiman identifies are concrete: adequate model access, reproducible findings, qualifications matched to tasks and methods that examine the surrounding system as well as the model. Anthropic’s appointment provides an example of the organisational approach; it does not settle those methodological questions. The balanced conclusion is that embedded evaluators and stronger evaluation science address different parts of the oversight problem. Progress on the former should not be mistaken for proof that the latter has been solved.