Markov chains and kernels

Markov chains, and their generalizations to Markov kernels and Markov processes, are fundamental objects in probability and statistics. In particular, Markov chain Monte Carlo (MCMC) is the workhorse of Bayesian statistics.

This page is about discrete-time Markov chains (DTMCs), i.e., Markov processes having both discrete time and discrete state spaces. General Markov processes have their own page.

Literature

General

Several experts have written nice perspective pieces on MCMC:

  • Häggström, 2007: Problem solving is often a matter of cooking up an appropriate Markov chain (doi)
  • Diaconis, 2009: The Markov chain Monte Carlo revolution (doi, pdf)
  • Diaconis, 2013: Some things we’ve learned (about Markov chain Monte Carlo) (doi, arxiv, pdf)

Mixing times

  • Levin & Peres, 2017: Markov Chains and mixing times, 2nd ed. (doi, pdf)
  • Vilnis, 2013: Markov chain Monte Carlo, mixing, and the spectral gap (pdf)
  • Montenegro & Tetali, 2006: Mathematical aspects of mixing times in Markov chains (doi, pdf)

Intertwinings

Intertwinings, a kind of Markov kernel, are morphisms between Markov chains.

  • Diaconis & Fill, 1990: Strong stationary times via a new form of duality (doi)
    • First systematic study of intertwinings
    • Fill, 1992: Strong stationary duality for continuous-time Markov chains (doi)
  • Jansen & Kurt, 2012: On the notion(s) of duality for Markov processes (doi, arxiv)
  • Swart, 2013, lecture notes: Duality and intertwining of Markov chains (pdf)
  • Sturm, Swart, Völlering, 2018, lecture notes: The algebraic approach to duality: An introduction (arxiv)

Category theory

For literature on Markov kernels from a category-theoretic perspective, see categorical probability theory.