Researchers from several physics and geology laboratories have developed Deep Emulator Network SEarch (DENSE), a technique for using deep-learning to perform scientific simulations from various fields, from high-energy physics to climate science. Compared to previous simulators, the results from DENSE achieved speedups ranging from 10 million to 2 billion times.
The scientists described their technique and several experiments in a paper published on arXiv. Motivated by a need to efficiently generate neural network emulators to replace slower simulations, the team developed a neural search method and a novel super-architecture that generates convolutional neural networks (CNNs); CNNs were chosen because they perform well on a large set of "natural" signals that are the domain of many scientific models. Standard simulator programs were used to generate training and test data for the CNNs, and according to the team,
The algorithm can find and train fast emulators for a wide range of applications while offering major improvements in terms of accuracy compared with other techniques, even when the training data is limited.
Scientists often use simulations---computer implementations of mathematical models---to test out new ideas and explore potential behaviors of natural systems. Because these simulations are implementations of theoretical models, they may run very slowly, even on modern computer hardware; for example, global climate models may take thousands of CPU-hours. Speeding up simulators could allow researchers to try out many ideas and has shown promise as a way to improve drug discovery or to improve and optimize expensive plasma and fusion experiments.
For cases where scientists are only interested in the model output and not the internal model state, an emulator may provide the answer. An emulator is a program that can mimic the simulator's output for a given input, but need not accurately model the internal state of the system. By using the slower simulator to generate many input/output examples, scientists can then train a neural network emulator that runs much faster than the simulator. The challenge is finding the proper neural network architecture for the emulator that can best fit the training data.
To solve this problem, the research team developed a neural architecture search (NAS) called Deep Emulator Network SEarch (DENSE). First, the team defined a network super-architecture which constrains the algorithm's search space. The super-architecture is a graph of nodes; the first node represents the inputs, and the last node represents the outputs; between are more than two dozen nodes representing the activation outputs of the hidden network layers. Nodes are connected by groups of operations, including CNN layers of various kernel sizes as well as identity layers (or "skip connections") as well as zero-layer that multiplies its input by zero. Candidate networks are sampled from this super-architecture space and trained. The candidates are then ranked by running them on a held-out validation dataset to find the best-performing network.
The research team used DENSE to build emulators for 10 different simulation cases from fields such as:
- high-energy-density physics
- astrophysics
- fusion energy science
- climate science
- earth science
Training data was gathered by running each simulator 14,000 times with random input data, except for two simulators that take hundreds or thousands of CPU-hours to run; for these two, less than 1,000 data points were gathered. The team found that the emulator output "generally matches closely" to the simulator output, even for the two cases with limited training data. The emulators ran much faster as well; for simulations that run in "minutes to days," the emulators can run in "milliseconds to a few seconds." DENSE also outperformed other non-neural network emulators, such as random forests, or manually-designed neural networks.
Paper co-author Duncan Watson-Parris, responding to a tweet about the work, noted
[The] emulator only emulates one aspect of the model, but it’s the bit we care about in this case. It makes things like optimal parameter estimation and UQ possible - which they wouldn’t be with the full model. Emulation has been around a long time and large speed-ups can be expected, the cool thing for me is how little training data was needed for such a general (ML) model.
Watson-Parris also noted that a "full write-up of the climate aspect" will be coming soon.