MIT Researchers Develop an Effective Way to Train more Reliable AI Agents

Fields varying from robotics to medicine to political science are attempting to train AI systems to make meaningful decisions of all kinds. For example, utilizing an AI system to smartly manage traffic in a busy city could assist motorists reach their destinations quicker, while enhancing security or sustainability.

Unfortunately, teaching an AI system to make great decisions is no simple job.

Reinforcement learning models, which underlie these AI decision-making systems, still frequently fail when confronted with even small variations in the jobs they are trained to carry out. When it comes to traffic, a design may struggle to manage a set of intersections with different speed limits, numbers of lanes, or traffic patterns.

To enhance the reliability of reinforcement knowing models for complex jobs with variability, MIT researchers have introduced a more efficient algorithm for training them.

The algorithm strategically chooses the best jobs for training an AI agent so it can efficiently perform all jobs in a collection of associated jobs. In the case of traffic signal control, each job could be one crossway in a task space that consists of all intersections in the city.

By concentrating on a smaller sized number of crossways that contribute the most to the algorithm’s general efficiency, this approach maximizes efficiency while keeping the training expense low.

The scientists found that their method was in between 5 and 50 times more effective than basic techniques on an array of simulated tasks. This gain in effectiveness helps the algorithm discover a better solution in a quicker manner, ultimately improving the efficiency of the AI agent.

“We were able to see incredible efficiency improvements, with a really basic algorithm, by believing outside the box. An algorithm that is not really complex stands a better opportunity of being adopted by the community because it is simpler to execute and easier for others to comprehend,” states senior author Cathy Wu, the Thomas D. and Virginia W. Cabot Career Development Associate Professor in Civil and Environmental Engineering (CEE) and the Institute for Data, Systems, and Society (IDSS), and a member of the Laboratory for Information and Decision Systems (LIDS).

She is joined on the paper by lead author Jung-Hoon Cho, a CEE college student; Vindula Jayawardana, a college student in the Department of Electrical Engineering and Computer Science (EECS); and Sirui Li, an IDSS graduate student. The research study will exist at the Conference on Neural Information Processing Systems.

Finding a middle ground

To train an algorithm to control traffic control at lots of crossways in a city, an engineer would usually pick in between 2 main techniques. She can train one algorithm for each crossway independently, using just that crossway’s data, or train a larger algorithm utilizing data from all intersections and after that apply it to each one.

But each method comes with its share of disadvantages. Training a different for each job (such as an offered intersection) is a lengthy process that requires a massive quantity of information and computation, while training one algorithm for all jobs often results in subpar efficiency.

Wu and her collaborators looked for a sweet spot between these two methods.

For their method, they pick a subset of jobs and train one algorithm for each task independently. Importantly, they tactically select specific jobs which are probably to improve the algorithm’s overall efficiency on all jobs.

They leverage a common technique from the reinforcement learning field called zero-shot transfer knowing, in which an already trained design is used to a brand-new job without being additional trained. With transfer knowing, the model typically performs extremely well on the brand-new neighbor task.

“We understand it would be perfect to train on all the jobs, but we wondered if we could get away with training on a subset of those tasks, apply the outcome to all the tasks, and still see a performance boost,” Wu states.

To determine which jobs they should select to make the most of anticipated performance, the researchers established an algorithm called Model-Based Transfer Learning (MBTL).

The MBTL algorithm has 2 pieces. For one, it designs how well each algorithm would perform if it were trained independently on one task. Then it models just how much each algorithm’s efficiency would degrade if it were moved to each other task, a concept referred to as generalization performance.

Explicitly modeling generalization efficiency permits MBTL to approximate the value of training on a brand-new job.

MBTL does this sequentially, picking the job which leads to the greatest performance gain initially, then choosing additional jobs that provide the greatest subsequent minimal enhancements to general efficiency.

Since MBTL just focuses on the most appealing jobs, it can considerably enhance the effectiveness of the training procedure.

Reducing training expenses

When the scientists checked this technique on simulated jobs, including managing traffic signals, handling real-time speed advisories, and carrying out numerous traditional control jobs, it was 5 to 50 times more effective than other approaches.

This indicates they could come to the same option by training on far less information. For circumstances, with a 50x performance increase, the MBTL algorithm might train on simply two tasks and achieve the same performance as a standard approach which uses data from 100 jobs.

“From the viewpoint of the 2 primary techniques, that means information from the other 98 jobs was not necessary or that training on all 100 tasks is puzzling to the algorithm, so the efficiency winds up even worse than ours,” Wu says.

With MBTL, adding even a percentage of additional training time could cause far better performance.

In the future, the scientists plan to design MBTL algorithms that can encompass more complex issues, such as high-dimensional task spaces. They are likewise interested in using their method to real-world problems, especially in next-generation movement systems.