In statistics, shrinkage has two meanings, though a common idea underlying both of these meanings is
the reduction in the effects of sampling variation:
-
in regression analysis, a fitted relationship
appears to perform less well on a new data set than on the data set used for fitting.
In particular the value of the coefficient of determination ‘shrinks’.
This idea is complementary to overfitting
-
to describe general types of estimators, or the effects of some types of estimation,
whereby a naive or raw estimate is improved by combining it with other information
(see shrinkage estimator).
More …
Estimation theory is a branch of statistics that deals with estimating the
values of parameters based on measured empirical data that has a random component.
The parameters describe an underlying physical setting in such a way that their
value affects the distribution of the measured data. An estimator attempts to approximate
the unknown parameters using the measurements. When the data consist of multiple variables
and one is estimating the relationship between them, estimation is known as regression analysis.
In estimation theory, two approaches are generally considered:
- The probabilistic approach (described in this article)
assumes that the measured data is random with probability distribution dependent
on the parameters of interest
- The set-membership approach assumes that the measured data vector belongs to
a set which depends on the parameter vector.
More …
Markov chain Monte Carlo (MCMC)
methods comprise a class of algorithms for sampling from a probability distribution.
By constructing a Markov chain that
has the desired distribution as its equilibrium distribution, one can obtain
a sample of the desired distribution by observing the chain after a number of steps.
The more steps there are, the more closely the distribution of the sample matches the actual
desired distribution.
More …
In mathematics, the Gibbs measure, is a probability measure frequently
seen in many problems of probability theory and statistical mechanics.
It is a generalization of the canonical ensemble to infinite systems.
The canonical ensemble gives the probability of the system X being in state x
(equivalently, of the random variable X having value x) as
\[P(X=x)={\frac {1}{Z(\beta )}}\exp(-\beta E(x))\]
where normalizing constant \(Z(\beta)\) is the partition function.
More …
In Bayesian probability, the Jeffreys prior is a
non-informative (objective) prior distribution for a parameter space;
it is proportional to the square root of the determinant of the Fisher information matrix:
\[p\left({\vec{\theta }}\right) \propto {\sqrt{\det{\mathcal{I}}\left({\vec{\theta }}\right)}}\]
It has the key feature that its functional dependence on the likelihood L is invariant under
reparameterization of the parameter vector \(\vec{\theta }\)
(the functional form of the prior density function itself is not invariant under
reparameterization, of course: only the measure that is identically zero has that property).
This makes it of special interest for use with scale parameters.
More …