Manara - Qatar Research Repository
Browse

Entropy, Free Energy, and Work of Restricted Boltzmann Machines

Download (1.67 MB)
journal contribution
submitted on 2024-07-08, 06:15 and posted on 2024-07-08, 13:24 authored by Sangchul Oh, Abdelkader Baggag, Hyunchul Nha

A restricted Boltzmann machine is a generative probabilistic graphic network. A probability of finding the network in a certain configuration is given by the Boltzmann distribution. Given training data, its learning is done by optimizing the parameters of the energy function of the network. In this paper, we analyze the training process of the restricted Boltzmann machine in the context of statistical physics. As an illustration, for small size bar-and-stripe patterns, we calculate thermodynamic quantities such as entropy, free energy, and internal energy as a function of the training epoch. We demonstrate the growth of the correlation between the visible and hidden layers via the subadditivity of entropies as the training proceeds. Using the Monte-Carlo simulation of trajectories of the visible and hidden vectors in the configuration space, we also calculate the distribution of the work done on the restricted Boltzmann machine by switching the parameters of the energy function. We discuss the Jarzynski equality which connects the path average of the exponential function of the work and the difference in free energies before and after training.

Other Information

Published in: Entropy
License: https://creativecommons.org/licenses/by/4.0/
See article on publisher's website: https://dx.doi.org/10.3390/e22050538

Additional institutions affiliated with: Arts and Sciences Program - TAMUQ

History

Language

  • English

Publisher

MDPI

Publication Year

  • 2020

License statement

This Item is licensed under the Creative Commons Attribution 4.0 International License.

Institution affiliated with

  • Hamad Bin Khalifa University
  • Qatar Environment and Energy Research Institute - HBKU
  • Qatar Computing Research Institute - HBKU
  • Texas A&M University at Qatar