Choice of Basis for Laplace Approximation
Maximum a posteriori optimization of parameters and the Laplace approximation for the marginal likelihood are both basis-dependent methods. This note compares two choices of basis for models parameterized by probabilities, showing that it is possible to improve on the traditional choice, the probability simplex, by transforming to the `softmax' basis. 1 Introduction Laplace's method approximates the integral of a function R d k wf(w) by fitting a Gaussian at the maximum w of f(w), and computing the volume under that Gaussian: Z d k wf(w) ' f( w)(2ß) k=2 j\\Gammarr log f(w)j \\Gamma1=2 : (1) This method is widely used in probabilistic modelling to approximate the value of marginal likelihoods, which are of interest for model comparison (Ripley, 1996; Lindley, 1980; Smith and Spiegelhalter, 1980; MacKay, 1992; Chickering and Heckerman, 1996). In this paper I consider the case of models whose parameters are probabilities, for example, hidden Markov models, mixture models, bel...