Gaussian processes (GPs) are a cornerstone of modern Bayesian methods, used almost wherever one may require nonparametric priors. The most typical use of GPs is in Gaussian process regression, also known as kriging. Quite naturally, the theory of Gaussian Process methods is well-developed. Aside from limited special cases in which Fourier analysis is applicable, GP-based methods have mostly been studied under large-$n$ asymptotics , which involve treating measurement locations as random and letting their number go to infinity. In this paper we report intriguing theoretical results obtained under a different asymptotic, one that treats the data as fixed, rather than random, with fixed sample size. The limit we look at is the so-called ``flat limit'', pioneered by Driscoll \& Fornberg in 2002. The flat limit consists in letting the spatial width of the kernel function go to infinity, which results in the covariance function becoming flat over the range of the data.

Studying Gaussian processes under the flat limit may seem at first sight to be entirely pointless - does that not correspond to a prior that contains only flat functions? Surprisingly, we show that the answer is no. This occurs because covariance functions have a second hyperparameter that sets the vertical scale (pointwise variance). When one lets pointwise variance grow as the covariance becomes wider, the actual function space spanned by Gaussian processes remains interesting and useful. In the cases studied here, they are (multivariate) polynomials and (polyharmonic) splines.