8 Statistics Functions
(require math/statistics) |
This module exports functions that compute statistics, meaning summary values for collections of samples, and functions for managing sequences of weighted or unweighted samples.
Most of the functions that compute statistics accept a sequence of nonnegative reals that correspond one-to-one with sample values. These are used as weights; equivalently counts, pseudocounts or unnormalized probabilities. While this makes it easy to work with weighted samples, it introduces some subtleties in bias correction. In particular, central moments must be computed without bias correction by default. See Expected Values for a discussion.
8.1 Expected Values
Functions documented in this section that compute higher central moments, such as variance, stddev and skewness, can optionally apply bias correction to their estimates. For example, when variance is given the argument #:bias #t, it multiplies the result by (/ n (- n 1)), where n is the number of samples.
> (variance '(1 2 3 4 4) #:bias #t) - : Real [generalized from Nonnegative-Real]
17/10
> (variance '(1 2 3 4) '(1 1 1 2) #:bias #t) - : Real [generalized from Nonnegative-Real]
17/10
> (variance '(1 2 3 4) '(1/2 1/2 1/2 1) #:bias 5) - : Real [generalized from Nonnegative-Real]
17/10
Because the magnitude of the bias correction for weighted samples cannot be known without user guidance, in all cases, the bias argument defaults to #f.
procedure
xs : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f
Examples: | ||||||||||||||||||||||||
|
procedure
(variance xs [ws #:bias bias]) → Nonnegative-Real
xs : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f bias : (U #t #f Real) = #f
procedure
(stddev xs [ws #:bias bias]) → Nonnegative-Real
xs : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f bias : (U #t #f Real) = #f
procedure
xs : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f bias : (U #t #f Real) = #f
procedure
(kurtosis xs [ws #:bias bias]) → Nonnegative-Real
xs : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f bias : (U #t #f Real) = #f
Examples: | ||||||
|
procedure
(variance/mean mean xs [ws #:bias bias]) → Nonnegative-Real
mean : Real xs : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f bias : (U #t #f Real) = #f
procedure
(stddev/mean mean xs [ws #:bias bias]) → Nonnegative-Real
mean : Real xs : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f bias : (U #t #f Real) = #f
procedure
(skewness/mean mean xs [ws #:bias bias]) → Real
mean : Real xs : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f bias : (U #t #f Real) = #f
procedure
(kurtosis/mean mean xs [ws #:bias bias]) → Nonnegative-Real
mean : Real xs : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f bias : (U #t #f Real) = #f
8.2 Running Expected Values
The statistics object allows computing the sample minimum, maximum, count, mean, variance, skewness, and excess kurtosis of a sequence of samples in O(1) space.
Example: | ||||||||||||
|
struct
(struct statistics (min max count))
min : Flonum max : Flonum count : Nonnegative-Flonum
The min and max fields are the minimum and maximum value observed so far, and the count field is the total weight of the samples (which is the number of samples if all samples are unweighted). The remaining, hidden fields are used to compute moments, and their number and meaning may change in future releases.
value
Examples: | ||||||||||||||||||||||||
|
procedure
(update-statistics s x [w]) → statistics
s : statistics x : Real w : Real = 1.0
procedure
(update-statistics* s xs [ws]) → statistics
s : statistics xs : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f
Examples: | |||||||||
|
procedure
s : statistics
procedure
(statistics-mean s) → Flonum
s : statistics
procedure
(statistics-variance s [#:bias bias]) → Nonnegative-Flonum
s : statistics bias : (U #t #f Real) = #f
procedure
(statistics-stddev s [#:bias bias]) → Nonnegative-Flonum
s : statistics bias : (U #t #f Real) = #f
procedure
(statistics-skewness s [#:bias bias]) → Flonum
s : statistics bias : (U #t #f Real) = #f
procedure
(statistics-kurtosis s [#:bias bias]) → Nonnegative-Flonum
s : statistics bias : (U #t #f Real) = #f
See Expected Values for the meaning of the bias keyword argument.
8.3 Correlation
procedure
(covariance xs ys [ws #:bias bias]) → Real
xs : (Sequenceof Real) ys : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f bias : (U #t #f Real) = #f
procedure
(correlation xs ys [ws #:bias bias]) → Real
xs : (Sequenceof Real) ys : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f bias : (U #t #f Real) = #f
Examples: | |||||||||
|
> (define ws (map (λ: ([x : Real] [y : Real]) (/ (pdf (normal-dist) y) (pdf (normal-dist x) y))) xs ys))
> (correlation xs ys (ann ws (Sequenceof Real))) - : Real
0.01856334379507927
See Expected Values for the meaning of the bias keyword argument.
procedure
(covariance/means μx μy xs ys [ws #:bias bias]) → Real
μx : Real μy : Real xs : (Sequenceof Real) ys : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f bias : (U #t #f Real) = #f
procedure
(correlation/means μx μy xs ys [ws #:bias bias]) → Real
μx : Real μy : Real xs : (Sequenceof Real) ys : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f bias : (U #t #f Real) = #f
8.4 Counting and Binning
procedure
(samples->hash xs) → (HashTable A Positive-Integer)
xs : (Sequenceof A) (samples->hash xs ws) → (HashTable A Nonnegative-Real) xs : (Sequenceof A) ws : (U #f (Sequenceof Real))
Examples: | ||||||
|
procedure
(count-samples xs)
→ (Values (Listof A) (Listof Positive-Integer)) xs : (Sequenceof A) (count-samples xs ws) → (Values (Listof A) (Listof Nonnegative-Real)) xs : (Sequenceof A) ws : (U #f (Sequenceof Real))
Examples: | ||||||||||
|
struct
(struct sample-bin (min max values weights))
min : B max : B values : (Listof A) weights : (U #f (Listof Nonnegative-Real))
procedure
(bin-samples bounds lte? xs ws) → (Listof (sample-bin A A))
bounds : (Sequenceof A) lte? : (A A -> Any) xs : (Sequenceof A) ws : (U #f (Sequenceof Real))
If n = (length bounds), then bin-samples returns at least (- n 1) bins, one for each pair of adjacent (sorted) bounds. If some values in xs are less than the smallest bound, they are grouped into a single bin in front. If some are greater than the largest bound, they are grouped into a single bin at the end.
Examples: | |||||||||||||||||||||||
|
If lte? is a less-than-or-equal relation, the bins represent half-open intervals (min, max] (except possibly the first, which may be closed). If lte? is a less-than relation, the bins represent half-open intervals [min, max) (except possibly the last, which may be closed). In either case, the sorts applied to bounds and xs are stable.
Because intervals used in probability measurements are normally open on the left, prefer to use less-than-or-equal relations for lte?.
If ws is #f, bin-samples returns bins with #f weights.
procedure
(bin-samples/key bounds lte? key xs ws) → (Listof (sample-bin A B))
bounds : (Sequenceof B) lte? : (B B -> Any) key : (A -> B) xs : (Sequenceof A) ws : (U #f (Sequenceof Real))
Example: | |||||||||
|
procedure
(sample-bin-compact bin) → (sample-bin A B)
bin : (sample-bin A B)
Example: | |||
|
procedure
(sample-bin-total bin) → Nonnegative-Real
bin : (sample-bin A B)
Examples: | ||||||
|
8.5 Order Statistics
procedure
(sort-samples lt? xs) → (Listof A)
lt? : (A A -> Any) xs : (Sequenceof A) (sort-samples lt? xs ws) → (Values (Listof A) (Listof Nonnegative-Real)) lt? : (A A -> Any) xs : (Sequenceof A) ws : (U #f (Sequenceof Real))
Examples: | |||||||||||||
|
procedure
(median lt? xs [ws]) → A
lt? : (A A -> Any) xs : (Sequenceof A) ws : (U #f (Sequenceof Real)) = #f
procedure
(quantile p lt? xs [ws]) → A
p : Real lt? : (A A -> Any) xs : (Sequenceof A) ws : (U #f (Sequenceof Real)) = #f
Examples: | |||||||||
|
If p = 0, quantile returns the smallest element of xs under the ordering relation lt?. If p = 1, it returns the largest element.
For weighted samples, quantile sorts xs and ws together (using sort-samples), then finds the least x for which the proportion of its cumulative weight is greater than or equal to p.
For unweighted samples, quantile uses the quickselect algorithm to find the element that would be at index (ceiling (- (* p n) 1)) if xs were sorted, where n is the length of xs.
procedure
(absdev xs [ws]) → Nonnegative-Real
xs : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f
procedure
(absdev/median median xs [ws]) → Nonnegative-Real
median : Real xs : (Sequenceof Real) ws : (U #f (Sequenceof Real)) = #f