NAME
math::statistics - Basic statistical functions and procedures
SYNOPSIS
package require TTccll 88 package require mmaatthh::::ssttaattiissttiiccss 00..11..11 ::::mmaatthh::::ssttaattiissttiiccss::::mmeeaann data ::::mmaatthh::::ssttaattiissttiiccss::::mmiinn data ::::mmaatthh::::ssttaattiissttiiccss::::mmaaxx data ::::mmaatthh::::ssttaattiissttiiccss::::nnuummbbeerr data ::::mmaatthh::::ssttaattiissttiiccss::::ssttddeevv data ::::mmaatthh::::ssttaattiissttiiccss::::vvaarr data::::mmaatthh::::ssttaattiissttiiccss::::bbaassiicc-ssttaattss data
::::mmaatthh::::ssttaattiissttiiccss::::hhiissttooggrraamm limits values ::::mmaatthh::::ssttaattiissttiiccss::::ccoorrrr data1 data2::::mmaatthh::::ssttaattiissttiiccss::::iinntteerrvvaall-mmeeaann-ssttddeevv data confidence
::::mmaatthh::::ssttaattiissttiiccss::::tt-tteesstt-mmeeaann data estmean eststdev confidence
::::mmaatthh::::ssttaattiissttiiccss::::qquuaannttiilleess data confidence ::::mmaatthh::::ssttaattiissttiiccss::::qquuaannttiilleess limits counts confidence ::::mmaatthh::::ssttaattiissttiiccss::::aauuttooccoorrrr data ::::mmaatthh::::ssttaattiissttiiccss::::ccrroossssccoorrrr data1 data2::::mmaatthh::::ssttaattiissttiiccss::::mmeeaann-hhiissttooggrraamm-lliimmiittss mean stdev number
::::mmaatthh::::ssttaattiissttiiccss::::mmiinnmmaaxx-hhiissttooggrraamm-lliimmiittss min max number
::::mmaatthh::::ssttaattiissttiiccss::::ppddff-nnoorrmmaall mean stdev value
::::mmaatthh::::ssttaattiissttiiccss::::ppddff-eexxppoonneennttiiaall mean value
::::mmaatthh::::ssttaattiissttiiccss::::ppddff-uunniiffoorrmm xmin xmax value
::::mmaatthh::::ssttaattiissttiiccss::::ccddff-nnoorrmmaall mean stdev value
::::mmaatthh::::ssttaattiissttiiccss::::ccddff-eexxppoonneennttiiaall mean value
::::mmaatthh::::ssttaattiissttiiccss::::ccddff-uunniiffoorrmm xmin xmax value
::::mmaatthh::::ssttaattiissttiiccss::::ccddff-ssttuuddeennttss-tt degrees value
::::mmaatthh::::ssttaattiissttiiccss::::rraannddoomm-nnoorrmmaall mean stdev number
::::mmaatthh::::ssttaattiissttiiccss::::rraannddoomm-eexxppoonneennttiiaall mean number
::::mmaatthh::::ssttaattiissttiiccss::::rraannddoomm-uunniiffoorrmm xmin xmax value
::::mmaatthh::::ssttaattiissttiiccss::::hhiissttooggrraamm-uunniiffoorrmm xmin xmax limits number
::::mmaatthh::::ssttaattiissttiiccss::::ffiilltteerr varname data expression ::::mmaatthh::::ssttaattiissttiiccss::::mmaapp varname data expression ::::mmaatthh::::ssttaattiissttiiccss::::ssaammpplleessccoouunntt varname list expression ::::mmaatthh::::ssttaattiissttiiccss::::ssuubbddiivviiddee::::mmaatthh::::ssttaattiissttiiccss::::pplloott-ssccaallee canvas xmin xmax ymin ymax
::::mmaatthh::::ssttaattiissttiiccss::::pplloott-xxyyddaattaa canvas xdata ydata tag
::::mmaatthh::::ssttaattiissttiiccss::::pplloott-xxyylliinnee canvas xdata ydata tag
::::mmaatthh::::ssttaattiissttiiccss::::pplloott-ttddaattaa canvas tdata tag
::::mmaatthh::::ssttaattiissttiiccss::::pplloott-ttlliinnee canvas tdata tag
::::mmaatthh::::ssttaattiissttiiccss::::pplloott-hhiissttooggrraamm canvas counts limits tag
DESCRIPTION
The mmaatthh::::ssttaattiissttiiccss package contains functions and procedures for basic statistical data analysis, such as: +o Descriptive statistical parameters (mean, minimum, maximum, standard deviation) +o Estimates of the distribution in the form of histograms and quantiles +o Basic testing of hypotheses +o Probability and cumulative density functions It is meant to help in developing data analysis applications or doing ad hoc data analysis, it is not in itself a full application, nor is itintended to rival with full (non-)commercial statistical pack-
ages. The purpose of this document is to describe the implemented procedures and provide some examples of their usage. As there is ample literature on the algorithms involved, we refer to relevant text books for more explanations. The package contains a fairly large number of publicprocedures. They can be distinguished in three sets: general proce-
dures, procedures that deal with specific statistical distributions,list procedures to select or transform data and simple plotting proce-
dures (these require Tk). Note: The data that need to be analyzed are always contained in a simple list. Missing values are represented as empty list elements. GGEENNEERRAALL PPRROOCCEEDDUURREESS The general statistical procedures are: ::::mmaatthh::::ssttaattiissttiiccss::::mmeeaann data Determine the mean value of the given list of data.data - List of data
::::mmaatthh::::ssttaattiissttiiccss::::mmiinn data Determine the minimum value of the given list of data.data - List of data
::::mmaatthh::::ssttaattiissttiiccss::::mmaaxx data Determine the maximum value of the given list of data.data - List of data
::::mmaatthh::::ssttaattiissttiiccss::::nnuummbbeerr dataDetermine the number of non-missing data in the given list
data - List of data
::::mmaatthh::::ssttaattiissttiiccss::::ssttddeevv data Determine the standard deviation of the data in the given listdata - List of data
::::mmaatthh::::ssttaattiissttiiccss::::vvaarr data Determine the variance of the data in the given listdata - List of data
::::mmaatthh::::ssttaattiissttiiccss::::bbaassiicc-ssttaattss data
Determine a list of all the descriptive parameters: mean, mini-
mum, maximum, number of data, standard deviation and variance.(This routine is called whenever either or all of the basic sta-
tistical parameters are required. Hence all calculations are done and the relevant values are returned.)data - List of data
::::mmaatthh::::ssttaattiissttiiccss::::hhiissttooggrraamm limits values Determine histogram information for the given list of data. Returns a list consisting of the number of values that fall into each interval. (The first interval consists of all values lower than the first limit, the last interval consists of all values greater than the last limit. There is one more interval than there are limits.)limits - List of upper limits (in ascending order) for the
intervals of the histogram.values - List of data
::::mmaatthh::::ssttaattiissttiiccss::::ccoorrrr data1 data2 Determine the correlation coefficient between two sets of data.data1 - First list of data
data2 - Second list of data
::::mmaatthh::::ssttaattiissttiiccss::::iinntteerrvvaall-mmeeaann-ssttddeevv data confidence
Return the interval containing the mean value and one containing the standard deviation with a certain level of confidence (assuming a normal distribution)data - List of raw data values (small sample)
confidence - Confidence level (0.95 or 0.99 for instance)
::::mmaatthh::::ssttaattiissttiiccss::::tt-tteesstt-mmeeaann data estmean eststdev confidence
Test whether the mean value of a sample is in accordance withthe estimated normal distribution with a certain level of confi-
dence. Returns 1 if the test succeeds or 0 if the mean is unlikely to fit the given distribution.data - List of raw data values (small sample)
estmean - Estimated mean of the distribution
eststdev - Estimated stdev of the distribution
confidence - Confidence level (0.95 or 0.99 for instance)
::::mmaatthh::::ssttaattiissttiiccss::::qquuaannttiilleess data confidence Return the quantiles for a given set of datadata - List of raw data values
confidence - Confidence level (0.95 or 0.99 for instance)
::::mmaatthh::::ssttaattiissttiiccss::::qquuaannttiilleess limits counts confidence Return the quantiles based on histogram information (alternative to the call with two arguments)limits - List of upper limits from histogram
counts - List of counts for for each interval in histogram
confidence - Confidence level (0.95 or 0.99 for instance)
::::mmaatthh::::ssttaattiissttiiccss::::aauuttooccoorrrr dataReturn the autocorrelation function as a list of values (assum-
ing equidistance between samples, about 1/2 of the number of raw data) The correlation is determined in such a way that the first value is always 1 and all others are equal to or smaller than 1. The number of values involved will diminish as the "time" (the index in the list of returned values) increasesdata - Raw data for which the autocorrelation must be determined
::::mmaatthh::::ssttaattiissttiiccss::::ccrroossssccoorrrr data1 data2Return the cross-correlation function as a list of values
(assuming equidistance between samples, about 1/2 of the number of raw data) The correlation is determined in such a way that the values can never exceed 1 in magnitude. The number of values involved willdiminish as the "time" (the index in the list of returned val-
ues) increases.data1 - First list of data
data2 - Second list of data
::::mmaatthh::::ssttaattiissttiiccss::::mmeeaann-hhiissttooggrraamm-lliimmiittss mean stdev number
Determine reasonable limits based on mean and standard deviation for a histogramConvenience function - the result is suitable for the histogram
function.mean - Mean of the data
stdev - Standard deviation
number - Number of limits to generate (defaults to 8)
::::mmaatthh::::ssttaattiissttiiccss::::mmiinnmmaaxx-hhiissttooggrraamm-lliimmiittss min max number
Determine reasonable limits based on a minimum and maximum for a histogramConvenience function - the result is suitable for the histogram
function.min - Expected minimum
max - Expected maximum
number - Number of limits to generate (defaults to 8)
SSTTAATTIISSTTIICCAALL DDIISSTTRRIIBBUUTTIIOONNSS In the literature a large number of probability distributions can befound. The statistics package supports:
+o The normal or Gaussian distribution+o The uniform distribution - equal probability for all data within
a given interval+o The exponential distribution - useful as a model for certain
extreme-value distributions.
+o PM - binomial, Poisson, chi-squared, student's T, F. In princi-
ple for each distribution one has procedures for:+o The probability density (pdf-*)
+o The cumulative density (cdf-*)
+o Quantiles for the given distribution (quantiles-*)
+o Histograms for the given distribution (histogram-*)
+o List of random values with the given distribution (random-*) The
following procedures have been implemented:::::mmaatthh::::ssttaattiissttiiccss::::ppddff-nnoorrmmaall mean stdev value
Return the probability of a given value for a normal distribu-
tion with given mean and standard deviation.mean - Mean value of the distribution
stdev - Standard deviation of the distribution
value - Value for which the probability is required
::::mmaatthh::::ssttaattiissttiiccss::::ppddff-eexxppoonneennttiiaall mean value
Return the probability of a given value for an exponential dis-
tribution with given mean.mean - Mean value of the distribution
value - Value for which the probability is required
::::mmaatthh::::ssttaattiissttiiccss::::ppddff-uunniiffoorrmm xmin xmax value
Return the probability of a given value for a uniform distribu-
tion with given extremes.xmin - Minimum value of the distribution
xmin - Maximum value of the distribution
value - Value for which the probability is required
::::mmaatthh::::ssttaattiissttiiccss::::ccddff-nnoorrmmaall mean stdev value
Return the cumulative probability of a given value for a normal distribution with given mean and standard deviation, that is the probability for values up to the given one.mean - Mean value of the distribution
stdev - Standard deviation of the distribution
value - Value for which the probability is required
::::mmaatthh::::ssttaattiissttiiccss::::ccddff-eexxppoonneennttiiaall mean value
Return the cumulative probability of a given value for an expo-
nential distribution with given mean.mean - Mean value of the distribution
value - Value for which the probability is required
::::mmaatthh::::ssttaattiissttiiccss::::ccddff-uunniiffoorrmm xmin xmax value
Return the cumulative probability of a given value for a uniform distribution with given extremes.xmin - Minimum value of the distribution
xmin - Maximum value of the distribution
value - Value for which the probability is required
::::mmaatthh::::ssttaattiissttiiccss::::ccddff-ssttuuddeennttss-tt degrees value
Return the cumulative probability of a given value for a Stu-
dent's t distribution with given number of degrees.degrees - Number of degrees of freedom
value - Value for which the probability is required
::::mmaatthh::::ssttaattiissttiiccss::::rraannddoomm-nnoorrmmaall mean stdev number
Return a list of "number" random values satisfying a normal dis-
tribution with given mean and standard deviation.mean - Mean value of the distribution
stdev - Standard deviation of the distribution
number - Number of values to be returned
::::mmaatthh::::ssttaattiissttiiccss::::rraannddoomm-eexxppoonneennttiiaall mean number
Return a list of "number" random values satisfying an exponen-
tial distribution with given mean.mean - Mean value of the distribution
number - Number of values to be returned
::::mmaatthh::::ssttaattiissttiiccss::::rraannddoomm-uunniiffoorrmm xmin xmax value
Return a list of "number" random values satisfying a uniform distribution with given extremes.xmin - Minimum value of the distribution
xmin - Maximum value of the distribution
number - Number of values to be returned
::::mmaatthh::::ssttaattiissttiiccss::::hhiissttooggrraamm-uunniiffoorrmm xmin xmax limits number
Return the expected histogram for a uniform distribution.xmin - Minimum value of the distribution
xmax - Maximum value of the distribution
limits - Upper limits for the buckets in the histogram
number - Total number of "observations" in the histogram
TO DO: more function descriptions to be added DDAATTAA MMAANNIIPPUULLAATTIIOONN The data manipulation procedures act on lists or lists of lists: ::::mmaatthh::::ssttaattiissttiiccss::::ffiilltteerr varname data expression Return a list consisting of the data for which the logicalexpression is true (this command works analogously to the com-
mand ffoorreeaacchh).varname - Name of the variable used in the expression
data - List of data
expression - Logical expression using the variable name
::::mmaatthh::::ssttaattiissttiiccss::::mmaapp varname data expression Return a list consisting of the data that are transformed via the expression.varname - Name of the variable used in the expression
data - List of data
expression - Expression to be used to transform (map) the data
::::mmaatthh::::ssttaattiissttiiccss::::ssaammpplleessccoouunntt varname list expressionReturn a list consisting of the counts of all data in the sub-
lists of the "list" argument for which the expression is true.varname - Name of the variable used in the expression
data - List of sublists, each containing the data
expression - Logical expression to test the data (defaults to
"true"). ::::mmaatthh::::ssttaattiissttiiccss::::ssuubbddiivviiddeeRoutine PM - not implemented yet
PPLLOOTT PPRROOCCEEDDUURREESS The following simple plotting procedures are available:::::mmaatthh::::ssttaattiissttiiccss::::pplloott-ssccaallee canvas xmin xmax ymin ymax
Set the scale for a plot in the given canvas. All plot routines expect this function to be called first. There is no automatic scaling provided.canvas - Canvas widget to use
xmin - Minimum x value
xmax - Maximum x value
ymin - Minimum y value
ymax - Maximum y value
::::mmaatthh::::ssttaattiissttiiccss::::pplloott-xxyyddaattaa canvas xdata ydata tag
Create a simple XY plot in the given canvas - the data are shown
as a collection of dots. The tag can be used to manipulate the appearance.canvas - Canvas widget to use
xdata - Series of independent data
ydata - Series of dependent data
tag - Tag to give to the plotted data (defaults to xyplot)
::::mmaatthh::::ssttaattiissttiiccss::::pplloott-xxyylliinnee canvas xdata ydata tag
Create a simple XY plot in the given canvas - the data are shown
as a line through the data points. The tag can be used to manip-
ulate the appearance.canvas - Canvas widget to use
xdata - Series of independent data
ydata - Series of dependent data
tag - Tag to give to the plotted data (defaults to xyplot)
::::mmaatthh::::ssttaattiissttiiccss::::pplloott-ttddaattaa canvas tdata tag
Create a simple XY plot in the given canvas - the data are shown
as a collection of dots. The horizontal coordinate is equal to the index. The tag can be used to manipulate the appearance.This type of presentation is suitable for autocorrelation func-
tions for instance or for inspecting the time-dependent behav-
iour.canvas - Canvas widget to use
tdata - Series of dependent data
tag - Tag to give to the plotted data (defaults to xyplot)
::::mmaatthh::::ssttaattiissttiiccss::::pplloott-ttlliinnee canvas tdata tag
Create a simple XY plot in the given canvas - the data are shown
as a line. See plot-tdata for an explanation.
canvas - Canvas widget to use
tdata - Series of dependent data
tag - Tag to give to the plotted data (defaults to xyplot)
::::mmaatthh::::ssttaattiissttiiccss::::pplloott-hhiissttooggrraamm canvas counts limits tag
Create a simple histogram in the given canvascanvas - Canvas widget to use
counts - Series of bucket counts
limits - Series of upper limits for the buckets
tag - Tag to give to the plotted data (defaults to xyplot)
TTHHIINNGGSS TTOO DDOO The following procedures are yet to be implemented:+o F-test-stdev
+o interval-mean-stdev
+o histogram-normal
+o histogram-exponential
+o test-histogram
+o linear-model
+o linear-residuals
+o test-corr
+o quantiles-*
+o fourier-coeffs
+o fourier-residuals
+o onepar-function-fit
+o onepar-function-residuals
+o plot-linear-model
+o subdivide EEXXAAMMPPLLEESS The code below is a small example of how you can examine a set of data:# Simple example:
# - Generate data (as a cheap way of getting some)
# - Perform statistical analysis to describe the data
#
package require math::statistics
#
# Two auxiliary procs
#
proc pause {time} { set wait 0after [expr {$time*1000}] {set ::wait 1}
vwait wait }proc print-histogram {counts limits} {
foreach count $counts limit $limits {
if { $limit != {} } {
puts [format "<%12.4g\t%d" $limit $count]
set prevlimit $limit
} else {puts [format ">%12.4g\t%d" $prevlimit $count]
} } }#
# Our source of arbitrary data
#
proc generateData { data1 data2 } {upvar 1 $data1 data1
upvar 1 $data2 data2
set d1 0.0 set d2 0.0for { set i 0 } { $i < 100 } { incr i } {
set d1 [expr {10.0-2.0*cos(2.0*3.1415926*$i/24.0)+3.5*rand()}]
set d2 [expr {0.7*$d2+0.3*$d1+0.7*rand()}]
lappend data1 $d1
lappend data2 $d2
} return {} }#
# The analysis session
#
package require Tk console show canvas .plot1 canvas .plot2pack .plot1 .plot2 -fill both -side top
generateData data1 data2puts "Basic statistics:"
set b1 [::math::statistics::basic-stats $data1]
set b2 [::math::statistics::basic-stats $data2]
foreach label {mean min max number stdev var} v1 $b1 v2 $b2 {
puts "$label\t$v1\t$v2"
} puts "Plot the data as function of \"time\" and against each other"::math::statistics::plot-scale .plot1 0 100 0 20
::math::statistics::plot-scale .plot2 0 20 0 20
::math::statistics::plot-tline .plot1 $data1
::math::statistics::plot-tline .plot1 $data2
::math::statistics::plot-xydata .plot2 $data1 $data2
puts "Correlation coefficient:"puts [::math::statistics::corr $data1 $data2]
pause 2 puts "Plot histograms"::math::statistics::plot-scale .plot2 0 20 0 100
set limits [::math::statistics::minmax-histogram-limits 7 16]
set histogramdata [::math::statistics::histogram $limits $data1]
::math::statistics::plot-histogram .plot2 $histogramdata $limits
puts "First series:"print-histogram $histogramdata $limits
pause 2set limits [::math::statistics::minmax-histogram-limits 0 15 10]
set histogramdata [::math::statistics::histogram $limits $data2]
::math::statistics::plot-histogram .plot2 $histogramdata $limits d2
puts "Second series:"print-histogram $histogramdata $limits
puts "Autocorrelation function:"set autoc [::math::statistics::autocorr $data1]
puts [::math::statistics::map $autoc {[format "%.2f" $x]}]
puts "Cross-correlation function:"
set crossc [::math::statistics::crosscorr $data1 $data2]
puts [::math::statistics::map $crossc {[format "%.2f" $x]}]
::math::statistics::plot-scale .plot1 0 100 -1 4
::math::statistics::plot-tline .plot1 $autoc "autoc"
::math::statistics::plot-tline .plot1 $crossc "crossc"
puts "Quantiles: 0.1, 0.2, 0.5, 0.8, 0.9"puts "First: [::math::statistics::quantiles $data1 {0.1 0.2 0.5 0.8 0.9}]"
puts "Second: [::math::statistics::quantiles $data2 {0.1 0.2 0.5 0.8 0.9}]"
If you run this example, then the following should be clear:+o There is a strong correlation between two time series, as dis-
played by the raw data and especially by the correlation func-
tions. +o Both time series show a significant periodic component +o The histograms are not very useful in identifying the nature ofthe time series - they do not show the periodic nature.
KKEEYYWWOORRDDSSdata analysis, mathematics, statistics
math 0.1.1 math::statistics(n)