Most existing theoretical analysis of reinforcement learning (RL) is limited to the tabular setting or linear models due to the difficulty in dealing with function approximation in high dimensional space with an uncertain environment. This work offers a fresh perspective into this challenge by analyzing RL in a general reproducing kernel Hilbert space (RKHS). We consider a family of Markov decision processes $\mathcal{M}$ of which the reward functions lie in the unit ball of an RKHS and transition probabilities lie in a given arbitrary set. We define a quantity called perturbational complexity by distribution mismatch $∆_{\mathcal{M}}(\epsilon)$ to characterize the complexity of the admissible state-action distribution space in response to a perturbation in the RKHS with scale $\epsilon$. We show that $∆_{\mathcal{M}}(\epsilon)$ gives both the lower bound of the error of all possible algorithms and the upper bound of two specific algorithms (fitted reward and fitted $Q$-iteration) for the RL problem. Hence, the decay of $∆_{\mathcal{M}}(\epsilon)$ with respect to $\epsilon$ measures the difficulty of the RL problem on $\mathcal{M}.$ We further provide some concrete examples and discuss whether $∆_{\mathcal{M}}(\epsilon)$ decays fast or not in these examples. As a byproduct, we show that when the reward functions lie in a high dimensional RKHS, even if the transition probability is known and the action space is finite, it is still possible for RL problems to suffer from the curse of dimensionality.

Reinforcement learning is a field of machine learning concerned with how agents interact with an unknown environment sequentially to maximize the expected cumulative reward. In practical problems with enormous states, function approximation is necessary to represent value or policy functions. However, when estimating the value or policy functions, one can not access the probability distribution under which the functions should be approximated. That is the state distribution induced by the estimated value or policy functions, which are unknown a priori. This phenomenon is termed distribution mismatch in this paper: a mismatch between the distribution used for the estimation and the distribution induced by the current approximation of the value or policy functions. This phenomenon is ubiquitous in the analysis of reinforcement learning algorithms and poses significant challenges for analysis.

This paper provides a new perspective to deal with this difficulty. An obvious sufficient condition is to show that the errors are uniformly small under the probability distributions induced by all possible policies. This work shows that this condition is also necessary, in the setting when the problem is defined in the reproducing kernel Hilbert space. It also introduces a key new concept, perturbational complexity by distribution mismatch, which characterizes the complexity of the corresponding reinforcement learning problem.

We present a variational density matrix approach to the thermal properties of interacting fermions in the continuum. The variational density matrix is parametrized by a permutation equivariant many-body unitary transformation together with a discrete probabilistic model. The unitary transformation is implemented as a quantum counterpart of neural canonical transformation, which incorporates correlation effects via a flow of fermion coordinates. As the first application, we study electrons in a two-dimensional quantum dot with an interaction-induced crossover from Fermi liquid to Wigner molecule. The present approach provides accurate results in the low-temperature regime, where conventional quantum Monte Carlo methods face severe difficulties due to the fermion sign problem. The approach is general and flexible for further extensions, thus holds the promise to deliver new physical results on strongly correlated fermions in the context of ultracold quantum gases, condensed matter, and warm dense matter physics.

Accurate prediction of thermal properties of quantum matter has been a challenging task in physics. At finite temperatures, Nature tries to balance the energy and entropy to bring physical systems into a minimal free energy state. Ironically, it was rather difficult to turn such a free energy minimization principle into a practical algorithm to simulate quantum matters, especially those consisting of fermionic particles. Such a difficulty is mainly due to the prohibitive cost associated with entropy calculation in the free energy, also known as the "intractable" partition functions problem. The present work makes such a variational calculation possible by capitalizing on the latest advances in generative machine learning.

Generative machine learning aims at modeling, learning, and sampling from high-dimensional probability distributions. To achieve these goals, the machine learning community has developed a variety of expressive neural network models with tractable partition functions. By jointly optimizing two generative models, one for the classical Boltzmann distribution and one for the quantum wavefunction, the authors successfully solved a system of a few electrons in a quantum dot. Further applications of the approach to prototypical quantum matters such as the uniform electron gas and dense hydrogen have offered new insights into these problems. In addition, this paper is also an example of how physical principles should be integrated with machine learning techniques.

We prove a general Embedding Principle of loss landscape of deep neural networks (NNs) that unravels a hierarchical structure of the loss landscape of NNs, i.e., loss landscape of an NN contains all critical points of all the narrower NNs. This result is obtained by constructing a class of critical embeddings which map any critical point of a narrower NN to a critical point of the target NN with the same output function. By discovering a wide class of general compatible critical embeddings, we provide a gross estimate of the dimension of critical submanifolds embedded from critical points of narrower NNs. We further prove an irreversibility property of any critical embedding that the number of negative/zero/positive eigenvalues of the Hessian matrix of a critical point may increase but never decrease as an NN becomes wider through the embedding. Using a special realization of general compatible critical embedding, we prove a stringent necessary condition for being a “truly-bad” critical point that never becomes a strict-saddle point through any critical embedding. This result implies the commonplace of strict-saddle points in wide NNs, which may be an important reason underlying the easy optimization of wide NNs widely observed in practice.

Understanding the loss landscape of a deep neural network is obviously important in analyzing the training trajectory and the generalization performance. This work proposes a new approaching for studying this problem by examining the (embedding) relation between the loss landscapes of neural networks of different widths. The paper proves a general Embedding Principle, namely the loss landscape of a neural network "contains" all critical points of the landscape for narrower neural networks. The paper also demonstrates that (i) critical points embedded from narrower neural networks form submanifolds; (ii) a local minimum is more likely to become a strict saddle point in the landscape of wider neural networks but not vice versa.

A long standing problem in the modeling of non-Newtonian hydrodynamics of polymeric flows is the availability of reliable and interpretable hydrodynamic models that faithfully encode the underlying microscale polymer dynamics. The main complication arises from the long polymer relaxation time, the complex molecular structure and heterogeneous interaction. DeePN$^2,$ a deep learning-based non-Newtonian hydrodynamic model, has been proposed and has shown some success in systematically passing the micro-scale structural mechanics information to the macro-scale hydrodynamics for suspensions with simple polymer conformation and bond potential. The model retains a multi-scaled nature by mapping the polymer configurations into a set of symmetry-preserving macro-scale features. The extended constitutive laws for these macro-scale features can be directly learned from the kinetics of their micro-scale counterparts. In this paper, we develop DeePN$^2$ using more complex micro-structural models. We show that DeePN$^2$ can faithfully capture the broadly overlooked viscoelastic differences arising from the specific molecular structural mechanics without human intervention.

Developing reliable physical models for the macro-scale hydrodynamics of non-Newtonian fluids has remained a challenging task for many decades. This study presents a deep learning-based non-Newtonian hydrodynamic model, DeePN$^2$, for constructing accurate and interpretable models directly from micro-scale descriptions. Molecular-level fidelity is retained by mapping the polymer configurations into a set of symmetry-preserving macro-scale features. The extended constitutive laws for these macro-scale features, including a new form of the objective tensor derivative, can be directly learned from the kinetics of their micro-scale counterparts. The construction is end-to-end and strictly preserves physical symmetries. It is demonstrated that DeePN$^2$ can faithfully capture the broadly overlooked viscoelastic differences arising from the specific molecular structural mechanics without human intervention.