Two broad uses of 'AI-first' now recur in management writing.

The first describes tool adoption: models procured, assistants embedded in existing software, and weekly usage tracked. Even the firms advising on AI transformation increasingly acknowledge that adoption alone changes little structurally. The second makes an operating-model claim: agents become operators of some workflows, while people move toward setting intent, handling exceptions, supervising outcomes, and defining decision rights explicitly rather than leaving them to informal practice.[1]

That second definition is more consequential. In physical operations, however, it remains incomplete.

Much of the current operating-model literature is written around knowledge work: finance, sales, service, and HR. In those settings, authority is largely an organizational variable, and an exception can often be routed to another qualified person. In a refinery, transmission grid, or chilled-water plant, authority is also constrained by the physical process and its safety case. What a system may write at the supervisory level differs from what it may write at the controller. Actions governed by interlocks or safety instrumented functions cannot be delegated to an AI application merely through an operating-model decision; any change to those protections belongs within the applicable hazard analysis, functional-safety lifecycle, and approval process.[2]

The useful question, then, is narrower than whether the operating model must change: Which decisions move, how far do they move, and what must be true of the person who retains the ones that stay?

There is also little that is greenfield about this work. Every industrial deployment I have worked on has been brownfield. The control system, alarm philosophy, approval limits, shift structure, and decades of operating practice were in place before the first AI model arrived. Each was designed around an existing distribution of human authority. In this setting, becoming AI-first is not the invention of a new organization around models. It is a renegotiation of authority that already has owners - and safeguards.

‍

Three layers, not one

I find it useful to separate the problem into three layers because organizations invest in them unevenly.

Perception establishes what appears to be true: telemetry, historian data, documents, plant state, asset hierarchy, and the current plan. Industry has invested in this layer for decades, although data quality, sensor validation, contextualization, and drift remain persistent constraints. Many digital-transformation programs have concentrated here, producing systems of record and monitoring. What operations increasingly need is a system that can support decisions.

Decision determines what should happen next within defined constraints. Plants already automate part of this territory. Advanced process control, model predictive control, and real-time optimization manage variables where process behavior, objectives, and constraints can be modeled with sufficient reliability. Their applicability becomes harder to justify as uncertainty, cross-unit dependencies, conflicting constraints, and changing objectives dominate the decision.

Occupying that space can require several forms of reasoning. Fault isolation often involves abductive inference: reasoning toward the most plausible explanation from incomplete evidence. Experienced operators also recognize similarities to prior incidents and adapt what worked then, a form of case-based reasoning. Both can exploit explicit relationships and constraints rather than relying only on statistical correlation.

The decision layer must also emit structure. Recommendations can be constrained to defined schemas; actions can be exposed as callable tools with typed arguments, authorization rules, and preconditions. That structure makes automated checking possible. It does not make an action safe by itself: the constraints, permissions, runtime enforcement, and fallback behavior must also be complete enough for the use case and independently validated.

Judgment determines where people remain in the causal chain: which classes of action require a person, at what depth, with what authority, and under what operating conditions. This is where product design, organizational design, control engineering, and process safety meet. No single discipline can define the answer alone.

The phrase “human in the loop” often stands in for this entire design problem. Too frequently, the implementation is only a confirmation dialog.

An organization becomes meaningfully AI-first when all three layers are deliberately designed. Stop at perception and the result is monitoring. Build the decision layer but leave judgment undefined, and the system is likely to remain advisory because the organization has not established the conditions under which greater autonomy would be acceptable.

‍

Two failure modes that look like progress

‍

Agents without agency

Agents act, but action alone does not establish meaningful agency. A rules engine acts. A scheduled job acts. An assistant that responds when addressed acts while holding almost no discretion over whether to act at all.

For operational design, agency is better treated as a question of control over what happens next than as a binary product label. A system may have wide latitude over how it executes a task while its objective remains fixed by a person. That is a legitimate design and often the right one where operating envelopes are hard.

The risk lies in mislabeling the design. Call a constrained assistant autonomous, and the first time it declines to act—or acts inside a boundary the user did not know existed—the user may conclude that the product is broken. I have watched a demonstration lose a room in about fifteen seconds this way. Operational trust can disappear at the first consequential surprise and is expensive to rebuild.

‍

Oversight that exists on paper

In this operational context, human-in-the-loop oversight means deliberately placing a person at a decision point to observe, validate, correct, approve, or intervene. A gate changes the risk profile in both directions. Placed well, it can catch a case the system cannot see. Placed badly, it adds latency, diffuses responsibility, and creates a record of approval without ensuring that meaningful judgment occurred.

An operator who approves outputs they no longer have the capacity to evaluate has effectively handed judgment to the machine while retaining the liability. What survives is a timestamp and a name.

Process industries have seen a related failure mode in alarm management. Alarm floods can overwhelm operators and obscure the signals that require action. The remedy includes rationalization: identifying which conditions warrant alarms, documenting causes and consequences, and reducing or reprioritizing alarm load so each signal remains meaningful.[3] Approval gates can decay in a similar way. Excessive volume turns deliberate review into routine acknowledgment. Alarm rationalization therefore offers a useful design analogy, even if approvals and alarms are not the same mechanism.

Four arrangements are commonly collapsed into the phrase “human in the loop”:

1. The system acts; the record is reviewable.
A person audits afterward and owns nothing in the moment.

2. The system proposes; a person approves.
A person evaluates before execution - the mode most products ship and the easiest to make performative.

3. The system acts within bounds; a person monitors.
A person can interrupt or override in real time.

4. The system advises; a person decides.
A person carries the decision with support.

These are not levels of maturity, and moving upward on the list is not automatically progress. Some recent agentic operating-model proposals shift oversight from per-action approval toward monitoring and exception handling, partly to avoid turning human review into a throughput bottleneck.[1] That solves one problem but can create another.

Decades of human-factors research show that passive supervision of automation can reduce situation awareness and impair takeover performance. The size of the effect depends on the task, interface, warning time, operating tempo, and operator experience.[4] Mode 3 may reduce approval fatigue while introducing a different risk: degraded readiness to re-enter the control loop. Which trade-off is acceptable is an evidence question about the process - not a general preference for more or less human involvement.

Aviation illustrates the separation between responsibility and direct operation. During carrier launches, F/A-18 procedures call for pilots to avoid restraining or immediately commanding the control stick while the flight-control system executes its programmed catapult-launch response.[5] The pilot remains responsible for the aircraft while deliberately refraining from manual input during that phase. Responsibility and operation can come apart, and designs that automatically fuse them may be less safe.

Four requirements for a judgment layer

The following are proposed design requirements, not a substitute for plant-specific hazard analysis, engineering standards, or operating procedures.

Map authority. For every class of action, state whether the system may commit it, propose it, or only advise on it—and what causes the action to be blocked, modified to a safer alternative, or escalated.

In industrial settings, this map has a physical dimension. The architectural layer at which an action lands helps reveal its interfaces, blast radius, and required controls. But assurance does not follow from an ISA-95 or Purdue level alone. It must be derived from the applicable hazards, consequences, safety and cybersecurity functions, required risk reduction, and lifecycle requirements.[2]

Reversibility is one useful sorting criterion, alongside consequence severity, time to harm, detectability, propagation, and the independence of existing safeguards. An action that can be withdrawn within a shift, with observable effects and no downstream commitment, belongs in a different class from one that locks in a blend, shipment, or shutdown sequence.

Write-capable actions should earn progressive authority. Where meaningful and supported by a validated model, that progression can include offline evaluation, shadow operation, dry runs, predicted-effect review, bounded execution, and only then broader autonomy. A recommendation that changes process technology, equipment, procedures, or approved operating conditions may also trigger the applicable management-of-change process, regardless of whether a person or AI system proposed it.[6]

Enforce competence. An approval carries weight only when the person has the training required for that decision, awareness of the current process state, enough time to evaluate the evidence, and authority to deviate from the recommendation. By hour ten of a shift, with dozens of approvals already processed, each condition may be degraded. The presence of an approval button does not restore them.

Part of the remedy is interface design. Part is organizational design: staffing, training, role definition, and authority to deviate. The second set of responsibilities rarely belongs entirely to an AI program, which is precisely why cross-functional ownership must be explicit.

Preserve attribution. Weeks later, an incident review should be able to reconstruct the recommendation, the evidence available at the time, the constraints evaluated, the system and rule versions involved, the tool calls made, the approvals recorded, and the resulting process state.

This is better understood as an evidence-and-action trace than as a transcript of “reasoning.” A generated explanation is not necessarily a complete or faithful record of internal inference. Reliable attribution comes from preserving the inputs and decisions that materially shaped the action, together with their provenance.[7]

Report coverage. When operational knowledge enters a system, the useful readout is not merely whether import succeeded. It is what the system can now support, where the evidence is incomplete, and what remains outside its competence.

Greenfield projects design their data models. Brownfield operations inherit them. Knowledge sits in spreadsheets, alarm databases, P&IDs, operating procedures, historian tags, and change records—in forms that require interpretation before a system can use them reliably. The quality, completeness, and interpretability of those sources constrain achievable coverage unless the program invests in reconstruction and validation.

I have gotten this wrong in the obvious direction. We scoped charting in chat and live plan editing while new users were still asking what a tag state meant and which fields an objective required. Day one hundred is easier to design for than day one, and it is the wrong place to start.

The same event, three products

Consider a unit that begins drifting at two in the morning.

The monitoring product raises an alarm: a value has crossed a threshold. Diagnosis under time pressure still belongs to the operator, which may be the problem the system was purchased to reduce.

The fluent product returns a well-written paragraph recommending corrective action. It sounds convincing but leaves out the governing constraint, the consequence of declining, its uncertainty, and whether this operator may take the action at this hour. The operator either trusts it on faith or discards it. Both are failures of design.

The decision-grade product states what changed, which rule or operating envelope the condition implicates, which prior cases are relevant, what action it recommends, why the action appears to remain within limits, what uncertainty remains, the consequence of doing nothing before the next shift, and who is authorized to commit it. The operator’s task becomes judgment: accept, modify, decline, or escalate—with grounds.

What separates the third product from the second is not fluency. It is the deliberate design of the decision and judgment layers.

The measure that matters

The useful question for any AI-first program is where judgment has moved, how far it has moved, and whether that movement has been made real.

Constraint-aware architecture and reliable infrastructure are necessary foundations for bounded autonomy. They help determine what a system may propose, what it must refuse, what it may execute, and what evidence it preserves about an action. They are not sufficient on their own. Data quality, model validity, uncertainty, cybersecurity, runtime monitoring, fallback behavior, and lifecycle management remain part of the machine side.

The organizational side is a design problem of its own: decision rights with physical boundaries, demonstrable competence, meaningful oversight, attributable action, and honest reporting of coverage. That work determines whether autonomy is ever permitted to operate beyond a demonstration.

Decision-grade AI without a decision-grade operating model is still a demonstration.

Notes and further reading

1. Boston Consulting Group describes becoming AI-first as an organizational rather than merely technical challenge and emphasizes operating-model redesign. See “How Companies Can Prepare for an AI-First Future” and its linked 2026 article, “Design Your Company for AI, Not AI for Your Company.” See also the World Economic Forum contributor article “How AI-first operating models unlock scalable value”. The latter reflects its author’s views and is used here as evidence of the management discourse, not institutional consensus.

2. See the International Society of Automation’s overview of the ISA-84/IEC 61511 functional-safety standards. Plant-specific decisions must follow the controlling standards, hazard analyses, and approved procedures for the facility and jurisdiction.

3. See the ISA overview of the ISA-18 alarm-management standards and “From Managing to Optimizing Alarms”.

4. Mica R. Endsley and Esin O. Kiris, “The Out-of-the-Loop Performance Problem and Level of Control in Automation”, Human Factors 37, no. 2 (1995): 381–394. For a later review emphasizing the context dependence of takeover performance, see the UK Department for Transport/TRL report Safe Performance of Other Activities in Conditionally Automated Vehicles (2020). These sources establish the human-factors mechanism; they do not directly measure refinery operators over a full shift.

5. See Michael M. Wallace, F/A-18E/F Catapult Minimum End Airspeed Testing (University of Tennessee, 2002), which describes a longitudinal flight-control algorithm designed for hands-free catapult launch. The precise procedure depends on aircraft variant and the controlling flight manual.

6. As a U.S. regulatory example, OSHA’s Process Safety Management standard requires management of changes to covered process chemicals, technology, equipment, procedures, facilities, and processing conditions, other than replacement in kind. See 29 CFR 1910.119(l) and Appendix C. Other jurisdictions and operators apply their own controlling requirements.

7. The NIST AI Risk Management Framework 1.0 treats AI as a socio-technical system and calls for clearly defined human roles and responsibilities. The AI RMF Playbook recommends documenting system capabilities that require human oversight, along with operational context, limitations, and training needs.

Author’s note: The three-layer model, four oversight arrangements, and four judgment-layer requirements in this article are proposed design frameworks derived from product work in industrial AI. They are not substitutes for process-safety analysis, functional-safety engineering, cybersecurity assessment, or facility-specific operating authority.

‍