Ebook: An Introduction to Markov Processes

This book provides a rigorous but elementary introduction to the theory of Markov Processes on a countable state space. It should be accessible to students with a solid undergraduate background in mathematics, including students from engineering, economics, physics, and biology. Topics covered are: Doeblin's theory, general ergodic properties, and continuous time processes. Applications are dispersed throughout the book. In addition, a whole chapter is devoted to reversible processes and the use of their associated Dirichlet forms to estimate the rate of convergence to equilibrium. These results are then applied to the analysis of the Metropolis (a.k.a simulated annealing) algorithm.

The corrected and enlarged 2^nd edition contains a new chapter in which the author develops computational methods for Markov chains on a finite state space. Most intriguing is the section with a new technique for computing stationary measures, which is applied to derivations of Wilson's algorithm and Kirchoff's formula for spanning trees in a connected graph.

---

I bought this book not because I needed to learn about Markov processes (I deal with them rather often) but because I wanted a text that discussed time-inhomogeneous Markov processes, however briefly. (I have been told that Doob does this, but I can't ever seem to bring myself to put in the work necessary to really appreciate his stochastic processes book.) Stroock gives a nice treatment of this topic in the context of simulated annealing, which is probably where most people would first encounter it these days.I have not read most of the rest of the book, but it is clearly a more rigorous treatment than either Norris or Bremaud (both of which are nice for the beginner or non-mathematician), and it requires a bit more mathematical maturity. Nevertheless my sampling indicates that it is well-written. It is probably better suited for the mathematicians and probabilists (or those who will have to deal with more advanced topics later) than for the average user of Markov processes. I would recommend it, along with Williams' wonderful Probability with Martingales, to anyone who wants a really solid yet concise grounding in mathematical probability at the undergraduate level.