AI-driven optimization in the energy sector

The challenge

Lack of automation and optimization challenges

The modern oil and gas industry heavily benefits from automation and the digitalization of their workflow. In this particular case, the Crude-oil Distillation Units (CDU) were of concern or rather the tanks material required to fill them up during the production process. Currently, the full process of filling the tanks from the tanker, feeding the CDU with raw materials and producing fuel from it is done manually. 

The client and Solvd team were in charge of designing software to support the automation of this process, thus enhancing the profitability and reducing the pressure put on the operators. 

The CDU itself is an industrial installation with a high level of complexity which is irrelevant in this particular use case. Yet it is crucial to understand two facts regarding the challenge:

• The CDU takes crude oil and other ingredients of the produced fuel from eight charging tanks. The tanks enable the operator to ensure that the CDU is constantly fed with the proper ingredients to avoid any interruptions in the production process. Tanks are filled from tankers at the berth, so there are multiple (more or less random) events that need to be taken into consideration when managing the oil flow from tanks to CDU.
• Also, the oil from the tankers needs to be discharged as fast as possible as to not hold up the queue and ensure a smooth transition to the next ship.
• A situation when the CDU gets no oil or production is stopped is to be avoided – an interruption is not only a tremendous cost but can also result in damaging the installation.
• Depending on the type of oil and ingredients, the refinery produces various types of petrol, either more or less valuable, thus the management of oil tanks has a direct impact on the whole installation’s business performance.

Currently, all the management work is done manually by the operator. The role is stressful and the specialist works under high pressure, being responsible for millions-of-dollars worth of workflow and complex machinery. 

Aiding this process with machine learning was the point where the client and Solvd decided to join forces to deliver a practical artificial intelligence application for the oil and gas industry.

From Solvd’s point of view, the workflow described above was a complex optimization task. The main issue of this process was to make optimal scheduling decisions according to the given objective functions. Moreover, the following challenges presented us with such optimization problems:

• Large space of decision variants.
• Large space of constraints.
• Uncertainty of some events (like ship delays).

The approach

Delivering two different policies: static and dynamic

Considering multiple aspects of the challenge, the client with Solvd support decided to deliver two policies. A static one, which is based on fixed rules, and a dynamic one, which leverages the power of reinforcement learning. From the operational point of view, they considered two agents:

• The Berth agent, which oversees the flow of the oil from the tankers to the oil tanks. 
• The CDU agent, which is responsible for managing the flow of the oil from the tanks to the CDU

The responsibility of these two agents is shown on the image below:

 After dividing the workflow between the agents, the team started to build a testing environment.

The refinery simulator

The first step during the delivery stage was designing a simulated environment of the refinery. Considering the cost of malfunction and the overall scale of the operations, one would be insane to test the technology on-site. The cost would be tremendous even assuming they only use one in the testing process, without accounting for the eventual damage done by suboptimal policies. Though, on the other hand, there was a need for the test environment to validate and train the agents that would be in charge of the facilities management. 

That’s why the Solvd and client’s engineering team delivered a refinery simulator – a sandbox environment enabling the team to test multiple policies and ideas.

The core of the simulated environment is the representation of the “current stage” of the facility. It included the pipes topology, all the tanks, CDUs, valves to operate and information about the oil flow. 

The simulator responded in real-time to the actions of the agent, making it a fully functional testing environment for any policy given, be that a manual one, an automated one or a fully ML-based policy. Also, the environment enabled the agents to be trained in an accelerated time frame while using the parallelization technique, so after a few hours of the simulated environment running, the ML-based agent could gather several hundred years of training.

Last but not least, the simulated environment included information about price differences between multiple types of products and the demand for them, so the agents were able to work on income maximization. 

The simulator was the first step to test and deliver agents to control the facility. The team has prepared two of them.

Static policy agent

In the first approach, they considered only static (fixed) policies for both Berth and CDU agents. They analyzed the following human-developed policies:

• BerthAgent – chooses the most empty tank to discharge the material.
• CDUAgent – chooses the tanks (both Low Sulphur and High Sulphur) with the most material for feeding and works until the demand is satisfied.

Such a policy is not able to satisfy the maximal target demand (working with maximum flow rate) for CDU units. It is worth mentioning that such policies may not give a feasible solution at all.

Dynamic policy agent

Contrary to the static one, the dynamic policy is based on the machine learning agents trained in a simulated environment with the reinforcement learning paradigm, one of the most innovative approaches in machine learning.

What is reinforcement learning

In the most common approach of supervised learning, the machine learning model is fed with data in order for it to extract patterns and learn how to recognize them. So, the model can achieve superhuman accuracy in recognizing road signs or cancer tumors on ICT scans. 

Although powerful and versatile, this approach suffers from several limitations. First of all, it is nearly helpless when the model encounters something before unseen, confusing the solution. 

In reinforcement learning, the model is trained by its interactions with the environment and being rewarded for certain outcomes. A great example of this comes from training autonomous cars, where the model can gather points by sticking to the rules, avoiding speeding, and driving safely. The model, at first, acts randomly and then memorizes the actions it is rewarded for, thus performing them more often to gather bigger rewards. Thus, the reinforcement learning paradigm is comparable to the natural model of learning, often seen in the animal and human world.

In this particular case, the model would get points for optimizing the refinery workflow for higher income and constant work. The key challenge was in delivering the environment from which the model could gather the data – it would be insane to run training sessions in an industrial facility – as the random actions of the model would have quickly resulted in disaster. 

The outcome

Generating and applying new behaviors for optimization

The client considered some fixed scenarios of vessels approaching the Berth with a given amount materials, Low Sulfur (LS) and High Sulfur (HS). They analyzed three objective functions, i.e. with minimal, average and maximal target demand for production processes for all CDU units.

The main achievement is that the models were able to find dynamic policies that provided feasible solutions for the minimum, the average and the nearly maximum target demand, while the static policies were not able to solve even the average demand.

The reinforcement learning-based training

The main results concern the dynamic policies for CDU agents that are developed during the training process. The team ran the simulation many times in order to observe more and more (state, action) pairs and the policies are being improved iteratively. In this approach, the client generates new behaviors for the agent to apply to fulfill the optimization requirements.