Training neural networks more efficiently
New method significantly reduces AI energy consumption
AI applications such as large language models (LLMs) have become an integral part of our everyday lives. The required computing, storage and transmission capacities are provided by data centers that consume vast amounts of energy. In Germany alone this amounted to around 16 billion kWh in 2020, or around 1% of the country’s total energy consumption. For 2025, this figure is expected to increase to 22 billion kWh.
New method is 100 times faster with comparable accuracy
The arrival of more complex AI applications in the coming years will substantially increase the demands on data center capacity. These applications will use up huge amounts of energy for the training of neural networks. To counteract this trend, researchers have developed a training method that is 100 times faster while achieving accuracy comparable to existing procedures. This will significantly reduce the energy consumption for training.
The functioning of neural networks, which are used in AI for such tasks as image recognition or language processing, is inspired by way the human brain works. These networks consist of interconnected nodes called artificial neurons. The input signals are weighted with certain parameters and then summed up. If a defined threshold is exceeded, the signal is passed on to the next node. To train the network, the initial selection of parameter values is usually randomized, for example using a normal distribution. The values are then incrementally adjusted to gradually improve the network predictions. Because of the many iterations required, this training is extremely demanding and consumes a lot of electricity.
Parameters selected according to probabilities
Felix Dietrich, a professor of Physics-enhanced Machine Learning, and his team have developed a new method. Instead of iteratively determining the parameters between the nodes, their approach uses probabilities. Their probabilistic method is based on the targeted use of values at critical locations in the training data where large and rapid changes in values are taking place. The objective of the current study is to use this approach to acquire energy-conserving dynamic systems from the data. Such systems change over the course of time in accordance with certain rules and are found in climate models and in financial markets, for example.
“Our method makes it possible to determine the required parameters with minimal computing power. This can make the training of neural networks much faster and, as a result, more energy efficient,” says Felix Dietrich. “In addition, we have seen that the accuracy of the new method is comparable to that of iteratively trained networks.”
Rahma, Atamert, Chinmay Datar, and Felix Dietrich, “Training Hamiltonian Neural Networks without Backpropagation”, 2024. Machine Learning and the Physical Sciences Workshop at the 38th conference on Neural Information Processing Systems (NeurIPS) https://neurips.cc/virtual/2024/99994
Bolager, Erik L, Iryna Burak, Chinmay Datar, Qing Sun, and Felix Dietrich. 2023. “Sampling Weights of Deep Neural Networks.” In Advances in Neural Information Processing Systems, 36:63075–116. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2023/hash/c7201deff8d507a8fe2e86d34094e154-Abstract-Conference.html
- The Physics-enhanced Machine Learning professorship is part of the TUM School of Computation, Information and Technology
- Prof. Felix Dietrich is a core member of the Munich Data Science Institute (MDSI) and an associate member of the Munich Center for Machine Learning (MCML)
- AI research at TUM
Technical University of Munich
Corporate Communications Center
- Julia Rinner
- presse@tum.de
- Teamwebsite
Contacts to this article:
Prof. Dr. Felix Dietrich
Physics-enhanced Machine Learning
Technical University of Munich
felix.dietrich@tum.de
www.tum.de
