Building Physical AI Systems

Overview

AI systems have achieved superhuman capabilities in constrained tasks with easily measurable environments such as playing Go, coding, and mathematics. Recently, persistent AI agents, such as Meta Muse, Grok Bot and OpenClaw, have made AI useful for more general tasks. Much less has been achieved in using these approaches to create an AI agent for physical industries, despite physical industries representing a large part of the economy.

There are several difficulties with creating such a system, including how to interface with the physical world to give the agent all the context it needs, since that context is rarely contained in existing digital databases¹.

Prax Industries has designed and deployed such systems — we call them Physical AI Systems. They perform critical tasks in physical industries such as controlling operations, optimising decisions, and doing analysis. They are systems of AI agents integrated with sensors, edge computers, and industrial equipment; along with specialised models and tools developed for problems in physical industries. The users are engineers and operators that work in environments like factories and mines. These users communicate with the AI system via SMS, email, radio, and graphical interfaces. The systems are composable: different modules — sensors, tools, and integrations — are applied for different use cases.

We used steelmaking as an initial venue to develop this technology, because we believe it is the quintessential physical industry, representing many of the difficulties and benefits of successfully applying AI to physical industries. In steelmaking, our systems (branded ScrapChef) have already been used to make billions of dollars of steel while being used daily in European and North American steelmaking companies. Not a single customer has churned and we have positive gross margins. The technology is generalisable by design and we are now engaging a wide range of industries, from semiconductor fabrication, to oil refining, to precious metal mining.

To the best of our knowledge, Prax Physical AI Agent Systems are among the first systems of their kind to be used reliably in production, at scale, delivering significant improvements over prior technology.

Introduction

Much of the economy is physical: manufacturing, mining, process industries. In these industries, people like process engineers and operations managers make physical products as efficiently, productively, and safely as possible. This boils down to continuously making complex technical and operational decisions. Which raw materials shall we use to make the steel? Which part of the mine should we use to maximise lifetime? What are the right process parameters in my chip fab to maximise yield?

These are not solved problems in general. People make the best decisions they can, using a mixture of heuristics derived from experience, textbook engineering, and simple models in Excel spreadsheets. The rate of growth of the physical economy is bottlenecked here.

AI is very good at making complex decisions in environments like games, software development, and mathematics. Clearly, it should also be possible to create and deploy AI agents that work alongside operators and engineers in physical industries, helping them to make critical decisions, and delivering superhuman performance. This is what we set out to achieve with Physical AI Systems. Below, we describe the key features of these systems, and explain why these features make them successful.

Read and write layer

The main reason that AI agents haven’t had impact in physical industries yet is that they can’t walk the production line.

All great engineers and operators in physical industries make decisions by being on the production line. This is essential to get the context they need. They need to look at the process. They need to listen to the equipment. They need to speak to their colleague to get the ephemeral information that is not stored in databases. Toyota and Elon Musk are infamous for knowing that you must walk the production line in order to make good decisions.

When designing a Physical AI System for a use case, we start with the interface between the AI and the physical world. The types of components we use in this layer are:

Sight and sound

We build and deploy sensor suites to the site, containing sensors like cameras, microphones and edge computers. This gives the AI some of the sensory input a human gets from walking the line: the AI can look to see whether a material is in stock, and where it is; the AI can listen to the equipment running to see if it sounds normal.

Natural language

We build integrations so that our AI systems can talk to engineers and operators in natural language, using tools like email, mobile chat interfaces, and radio. So much context in a factory or mine is contained in the recent natural language exchanged between engineers and operators. Our AI system will text their colleague in the logistics department to see if a shipment has arrived. It will read a daily email report to see what observations the operators made overnight.

Databases and equipment

Just as humans also look at data in databases and equipment, so do our AI systems. We build traditional integrations with MES systems, ERP systems, and similar sources of data.

All of these sources travel through pipelines that transform the raw data into structured databases and/or context for an LLM. This is often done with specialised models, such as vision models that recognise and classify events, that have been trained in house for the particular use case we are solving.

Below we show an example of such a specialised model applied to understanding raw material operations in steelmaking (note: this and all examples in the article use synthetic data. This is to protect the privacy and confidentiality of our customers, and to make the examples easier to interpret for people that don’t work in the industry). A vision model recognises the handling of raw materials used for making steel in an electric arc furnace. Without this system, the handling of raw materials is reported manually by operators, which is a dangerous distraction and leads to large errors in data collection, which in turn is problematic for correlating the use of different materials with process outcomes.

The Intelligence layer

The data described above is collected continuously and is available to the rest of the system. The rest of the system is typically a combination of traditional process control tools and agents that use those tools. These agents perform tasks for engineers and operators, tailored to each customer, and can include analysis, reporting, or entire process control decisions.

One of the problems that we have built a specialised tool for is feedstock blend optimisation. Feedstock blend optimisation is a common problem in many industries. In steelmaking, the challenge is to calculate the optimal blend of feedstock (mostly scrap), in order to make the steel according to an objective (often, to make the correct quality for the lowest possible cost). This is a difficult problem that normally a process engineer or operations manager will solve. It is difficult because: it is dynamic (there is often a new optimal solution every day); it is high dimensional (many materials, grades of steel, etc); the data are not readily available in a structured format (inventory, quality, and other data scattered between emails and databases); and you have to plan over long time horizons with high uncertainty (you must plan over dozens of production cycles during which time many assumptions can change). Not only is it difficult, it is important: raw materials are over half of the costs of making steel, and it’s a commodity business, where cost efficiency has a large impact on company success. This problem is described by steelmakers in the video below.

In theory, feedstock blend optimisation is a simple mathematical problem that can be solved with techniques like Linear Programming. In practice, this is insufficient. It is much more effective to create a Linear Programming tool, then make it available to a suitably skilled agent, and in turn embed that agent into a Physical AI System. This is a great example of the general point we are making in this post, and the general point of Physical AI Systems. The agent is able to tirelessly use the tool, explore parameters, and monitor conditions to check for changes, therefore reaching far more optimal solutions than a human, and it does so far more responsively. At the same time, the agent, along with the AI-physical world interface layer, can collect, structure, and unify far more context than a human can do manually; or indeed than can be achieved by building traditional integrations.

Meeting people where they are

A critical aspect of our Physical AI Systems is that they meet people where they are. Many engineers or operators spend their time on the shop floor, not at a desk. Their most commonly used tool is their phone, using email and chat apps; not a desktop computer. Where possible, we therefore make natural language the primary mode of communication with the Physical AI System. These natural language interfaces can be used to ask the agent to do a task, or to approve the agent to take an action, among other things.

When the AI system touches workflows that were previously not digitised at all, we will install tablets, HMIs, etc, and develop the required interfaces.

Generally speaking, we also implement graphical user interfaces so that users can directly interact with the underlying traditional tools and workflows that any agent is using. Just like how one can readily read and edit code manually when using a coding agent, this ensures maximum control and observability.

Understand, Decide, Act

Typically, our aim is for our Physical AI Systems to perform end-to-end UDA workflows:

• Understand. Dynamically maintain an understanding of the process: what inventory do we have? Are we running on schedule? Is our process under control? Do this with all means available, both structured data, natural language, and vision and sound.
• Decide. Make a decision. What is the optimal feedstock blend? What is the optimal production schedule? Make these decisions with traditional, interpretable models that the agent uses, such that decisions are understandable.
• Act. Write a schedule back to an MES, send a plan to someone via email.

For example, in the feedstock optimisation task in steelmaking, the workflow is shown in the diagram and caption on the left.

The agent is essentially running this workflow 24/7 on behalf of the engineer: checking for changes in circumstances that should trigger a new plan; checking the plan is running as intended; or reporting on other related information. The engineer is managing this process, modifying the underlying tooling parameters, or setting up new reporting, all from their mobile phone in seconds. The agent has memories such that it learns the preferences of the engineer and plant over time.

Previously, at best, the engineer would make a single decision each day, requiring 10 minutes to 2 hours of manual work, needing to return to their desk, and producing highly suboptimal results with limited tools like spreadsheets. The agent not only improves the quality of life for the engineer, but it makes the decision more frequently and more optimally, significantly increasing cost efficiency and other objectives.