Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
 PREV CLASS   NEXT CLASS FRAMES    NO FRAMES    
SUMMARY: NESTED | FIELD | CONSTR | METHOD DETAIL: FIELD | CONSTR | METHOD

baseCode.math
Class DescriptiveWithMissing

java.lang.Object
  extended bycern.jet.stat.Descriptive
      extended bybaseCode.math.DescriptiveWithMissing
public class DescriptiveWithMissing
extends cern.jet.stat.Descriptive

Mathematical functions for statistics that allow missing values without scotching the calculations.

Be careful because some methods from cern.jet.stat.Descriptive have not been overridden and will yield a UnsupportedOperationException if used.

Some functions that come with DoubleArrayLists will not work in an entirely compatible way with missing values. For examples, size() reports the total number of elements, including missing values. To get a count of non-missing values, use this.sizeWithoutMissingValues(). The right one to use may vary.

Not all methods need to be overridden. However, all methods that take a "size" parameter should be passed the results of sizeWithoutMissingValues(data), instead of data.size().

Copyright � 2004 Columbia University

Based in part on code from the colt package: Copyright � 1999 CERN - European Organization for Nuclear Research.

Version:
$Id: DescriptiveWithMissing.java,v 1.18 2005/01/05 02:01:02 pavlidis Exp $
Author:
Paul Pavlidis
See Also:
cern.jet.stat.Descriptive

Method Summary
static double autoCorrelation(cern.colt.list.DoubleArrayList data, int lag, double mean, double variance)
          Not supported.
static double correlation(cern.colt.list.DoubleArrayList x, cern.colt.list.DoubleArrayList y)
          Calculate the pearson correlation of two arrays.
static double correlation(cern.colt.list.DoubleArrayList data1, double standardDev1, cern.colt.list.DoubleArrayList data2, double standardDev2)
          Returns the correlation of two data sequences.
static double covariance(cern.colt.list.DoubleArrayList data1, cern.colt.list.DoubleArrayList data2)
          Returns the SAMPLE covariance of two data sequences.
static double durbinWatson(cern.colt.list.DoubleArrayList data)
          Durbin-Watson computation.
static double geometricMean(cern.colt.list.DoubleArrayList data)
          Returns the geometric mean of a data sequence.
static void incrementalUpdate(cern.colt.list.DoubleArrayList data, int from, int to, double[] inOut)
          Not supported.
static void incrementalUpdateSumsOfPowers(cern.colt.list.DoubleArrayList data, int from, int to, int fromSumIndex, int toSumIndex, double[] sumOfPowers)
          Not supported.
static void incrementalWeightedUpdate(cern.colt.list.DoubleArrayList data, cern.colt.list.DoubleArrayList weights, int from, int to, double[] inOut)
          Not supported.
static double kurtosis(cern.colt.list.DoubleArrayList data, double mean, double standardDeviation)
          Returns the kurtosis (aka excess) of a data sequence, which is -3 + moment(data,4,mean) / standardDeviation4.
static double kurtosis(double moment4, double standardDeviation)
          Returns the kurtosis (aka excess) of a data sequence.
static double lag1(cern.colt.list.DoubleArrayList data, double mean)
          Not supported.
static double mean(double[] elements, int effectiveSize)
          Special mean calculation where we use the effective size as an input.
static double mean(cern.colt.list.DoubleArrayList data)
           
static double mean(cern.colt.list.DoubleArrayList x, int effectiveSize)
          Special mean calculation where we use the effective size as an input.
static double meanAboveQuantile(int quantile, cern.colt.list.DoubleArrayList array)
          Calculate the mean of the values above a particular quantile of an array.
static double median(cern.colt.list.DoubleArrayList sortedData)
          Returns the median of a sorted data sequence.
static double moment(cern.colt.list.DoubleArrayList data, int k, double c)
          Returns the moment of k -th order with constant c of a data sequence, which is Sum( (data[i]-c)k ) / data.size().
static double product(cern.colt.list.DoubleArrayList data)
          Returns the product of a data sequence, which is Prod( data[i] ).
static double quantile(cern.colt.list.DoubleArrayList sortedData, double phi)
          Returns the phi- quantile; that is, an element elem for which holds that phi percent of data elements are less than elem.
static double quantileInverse(cern.colt.list.DoubleArrayList sortedList, double element)
          Returns how many percent of the elements contained in the receiver are <= element.
static cern.colt.list.DoubleArrayList quantiles(cern.colt.list.DoubleArrayList sortedData, cern.colt.list.DoubleArrayList percentages)
          Returns the quantiles of the specified percentages.
static double rankInterpolated(cern.colt.list.DoubleArrayList sortedList, double element)
          Returns the linearly interpolated number of elements in a list less or equal to a given element.
static double sampleKurtosis(cern.colt.list.DoubleArrayList data, double mean, double sampleVariance)
          Returns the sample kurtosis (aka excess) of a data sequence.
static double sampleSkew(cern.colt.list.DoubleArrayList data, double mean, double sampleVariance)
          Returns the sample skew of a data sequence.
static double sampleStandardDeviation(int size, double sampleVariance)
          Returns the sample standard deviation.
static double sampleVariance(cern.colt.list.DoubleArrayList data, double mean)
          Returns the sample variance of a data sequence.
static int sizeWithoutMissingValues(cern.colt.list.DoubleArrayList list)
          Return the size of the list, ignoring missing values.
static double skew(cern.colt.list.DoubleArrayList data, double mean, double standardDeviation)
          Returns the skew of a data sequence, which is moment(data,3,mean) / standardDeviation3.
static void standardize(cern.colt.list.DoubleArrayList data)
          Standardize.
static void standardize(cern.colt.list.DoubleArrayList data, double mean, double standardDeviation)
          Modifies a data sequence to be standardized.
static double sum(cern.colt.list.DoubleArrayList data)
          Returns the sum of a data sequence.
static double sumOfInversions(cern.colt.list.DoubleArrayList data, int from, int to)
          Returns the sum of inversions of a data sequence, which is Sum( 1.0 / data[i]).
static double sumOfLogarithms(cern.colt.list.DoubleArrayList data, int from, int to)
          Returns the sum of logarithms of a data sequence, which is Sum( Log(data[i]).
static double sumOfPowerDeviations(cern.colt.list.DoubleArrayList data, int k, double c)
          Returns Sum( (data[i]-c)k ); optimized for common parameters like c == 0.0 and/or k == -2 ..
static double sumOfPowerDeviations(cern.colt.list.DoubleArrayList data, int k, double c, int from, int to)
          Returns Sum( (data[i]-c)k ) for all i = from ..
static double sumOfPowers(cern.colt.list.DoubleArrayList data, int k)
          Returns the sum of powers of a data sequence, which is Sum ( data[i]k ).
static double sumOfSquaredDeviations(cern.colt.list.DoubleArrayList data)
          Compute the sum of the squared deviations from the mean of a data sequence.
static double sumOfSquares(cern.colt.list.DoubleArrayList data)
          Returns the sum of squares of a data sequence.
static double trimmedMean(cern.colt.list.DoubleArrayList sortedData, double mean, int left, int right)
          Returns the trimmed mean of a sorted data sequence.
static double variance(cern.colt.list.DoubleArrayList data)
          Provided for convenience!
static double weightedMean(cern.colt.list.DoubleArrayList data, cern.colt.list.DoubleArrayList weights)
          Returns the weighted mean of a data sequence.
static double winsorizedMean(cern.colt.list.DoubleArrayList sortedData, double mean, int left, int right)
          Not supported.
 
Methods inherited from class cern.jet.stat.Descriptive
checkRangeFromTo, frequencies, geometricMean, harmonicMean, max, meanDeviation, min, moment, pooledMean, pooledVariance, product, rms, sampleKurtosis, sampleKurtosisStandardError, sampleSkew, sampleSkewStandardError, sampleVariance, sampleWeightedVariance, skew, split, standardDeviation, standardError, sumOfSquaredDeviations, variance, variance, weightedRMS
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Method Detail

autoCorrelation

public static double autoCorrelation(cern.colt.list.DoubleArrayList data,
                                     int lag,
                                     double mean,
                                     double variance)
Not supported.

Parameters:
data - DoubleArrayList
lag - int
mean - double
variance - double
Returns:
double

correlation

public static double correlation(cern.colt.list.DoubleArrayList data1,
                                 double standardDev1,
                                 cern.colt.list.DoubleArrayList data2,
                                 double standardDev2)
Returns the correlation of two data sequences. That is covariance(data1,data2)/(standardDev1*standardDev2). Missing values are ignored. This method is overridden to stop users from using the method in the superclass when missing values are present. The problem is that the standard deviation cannot be computed without knowning which values are not missing in both vectors to be compared. Thus the standardDev parameters are thrown away by this method.

Parameters:
data1 - DoubleArrayList
standardDev1 - double - not used
data2 - DoubleArrayList
standardDev2 - double - not used
Returns:
double

correlation

public static double correlation(cern.colt.list.DoubleArrayList x,
                                 cern.colt.list.DoubleArrayList y)
Calculate the pearson correlation of two arrays. Missing values (NaNs) are ignored.

Parameters:
x - DoubleArrayList
y - DoubleArrayList
Returns:
double

covariance

public static double covariance(cern.colt.list.DoubleArrayList data1,
                                cern.colt.list.DoubleArrayList data2)
Returns the SAMPLE covariance of two data sequences. Pairs of values are only considered if both are not NaN. If there are no non-missing pairs, the covariance is zero.

Parameters:
data1 - the first vector
data2 - the second vector
Returns:
double

durbinWatson

public static double durbinWatson(cern.colt.list.DoubleArrayList data)
Durbin-Watson computation. This measures the serial correlation in a data series.

Parameters:
data - DoubleArrayList
Returns:
double

geometricMean

public static double geometricMean(cern.colt.list.DoubleArrayList data)
Returns the geometric mean of a data sequence. Missing values are ignored. Note that for a geometric mean to be meaningful, the minimum of the data sequence must not be less or equal to zero.
The geometric mean is given by pow( Product( data[i] ), 1/data.size()). This method tries to avoid overflows at the expense of an equivalent but somewhat slow definition: geo = Math.exp( Sum( Log(data[i]) ) / data.size()).

Parameters:
data - DoubleArrayList
Returns:
double

incrementalUpdate

public static void incrementalUpdate(cern.colt.list.DoubleArrayList data,
                                     int from,
                                     int to,
                                     double[] inOut)
Not supported.

Parameters:
data - DoubleArrayList
from - int
to - int
inOut - double[]

incrementalUpdateSumsOfPowers

public static void incrementalUpdateSumsOfPowers(cern.colt.list.DoubleArrayList data,
                                                 int from,
                                                 int to,
                                                 int fromSumIndex,
                                                 int toSumIndex,
                                                 double[] sumOfPowers)
Not supported.

Parameters:
data - DoubleArrayList
from - int
to - int
fromSumIndex - int
toSumIndex - int
sumOfPowers - double[]

incrementalWeightedUpdate

public static void incrementalWeightedUpdate(cern.colt.list.DoubleArrayList data,
                                             cern.colt.list.DoubleArrayList weights,
                                             int from,
                                             int to,
                                             double[] inOut)
Not supported.

Parameters:
data - DoubleArrayList
weights - DoubleArrayList
from - int
to - int
inOut - double[]

kurtosis

public static double kurtosis(double moment4,
                              double standardDeviation)
Returns the kurtosis (aka excess) of a data sequence.

Parameters:
moment4 - the fourth central moment, which is moment(data,4,mean).
standardDeviation - the standardDeviation.
Returns:
double

kurtosis

public static double kurtosis(cern.colt.list.DoubleArrayList data,
                              double mean,
                              double standardDeviation)
Returns the kurtosis (aka excess) of a data sequence, which is -3 + moment(data,4,mean) / standardDeviation4.

Parameters:
data - DoubleArrayList
mean - double
standardDeviation - double
Returns:
double

lag1

public static double lag1(cern.colt.list.DoubleArrayList data,
                          double mean)
Not supported.

Parameters:
data - DoubleArrayList
mean - double
Returns:
double

mean

public static double mean(cern.colt.list.DoubleArrayList data)
Parameters:
data - Values to be analyzed.
Returns:
Mean of the values in x. Missing values are ignored in the analysis.

mean

public static double mean(cern.colt.list.DoubleArrayList x,
                          int effectiveSize)
Special mean calculation where we use the effective size as an input.

Parameters:
x - The data
effectiveSize - The effective size used for the mean calculation.
Returns:
double

mean

public static double mean(double[] elements,
                          int effectiveSize)
Special mean calculation where we use the effective size as an input.

Parameters:
elements - The data double array.
effectiveSize - The effective size used for the mean calculation.
Returns:
double

meanAboveQuantile

public static double meanAboveQuantile(int quantile,
                                       cern.colt.list.DoubleArrayList array)
Calculate the mean of the values above a particular quantile of an array.

Parameters:
quantile - A value from 0 to 100
array - Array for which we want to get the quantile.
Returns:
double

median

public static double median(cern.colt.list.DoubleArrayList sortedData)
Returns the median of a sorted data sequence. Missing values are not considered.

Parameters:
sortedData - the data sequence; must be sorted ascending .
Returns:
double

moment

public static double moment(cern.colt.list.DoubleArrayList data,
                            int k,
                            double c)
Returns the moment of k -th order with constant c of a data sequence, which is Sum( (data[i]-c)k ) / data.size().

Parameters:
data - DoubleArrayList
k - int
c - double
Returns:
double

product

public static double product(cern.colt.list.DoubleArrayList data)
Returns the product of a data sequence, which is Prod( data[i] ). Missing values are ignored. In other words: data[0]*data[1]*...*data[data.size()-1]. Note that you may easily get numeric overflows.

Parameters:
data - DoubleArrayList
Returns:
double

quantile

public static double quantile(cern.colt.list.DoubleArrayList sortedData,
                              double phi)
Returns the phi- quantile; that is, an element elem for which holds that phi percent of data elements are less than elem. Missing values are ignored. The quantile need not necessarily be contained in the data sequence, it can be a linear interpolation.

Parameters:
sortedData - the data sequence; must be sorted ascending .
phi - the percentage; must satisfy 0 <= phi <= 1.
Returns:
double

quantileInverse

public static double quantileInverse(cern.colt.list.DoubleArrayList sortedList,
                                     double element)
Returns how many percent of the elements contained in the receiver are <= element. Does linear interpolation if the element is not contained but lies in between two contained elements. Missing values are ignored.

Parameters:
sortedList - the list to be searched (must be sorted ascending).
element - the element to search for.
Returns:
the percentage phi of elements <= element(0.0 <= phi <= 1.0).

quantiles

public static cern.colt.list.DoubleArrayList quantiles(cern.colt.list.DoubleArrayList sortedData,
                                                       cern.colt.list.DoubleArrayList percentages)
Returns the quantiles of the specified percentages. The quantiles need not necessarily be contained in the data sequence, it can be a linear interpolation.

Parameters:
sortedData - the data sequence; must be sorted ascending .
percentages - the percentages for which quantiles are to be computed. Each percentage must be in the interval [0.0,1.0].
Returns:
the quantiles.

rankInterpolated

public static double rankInterpolated(cern.colt.list.DoubleArrayList sortedList,
                                      double element)
Returns the linearly interpolated number of elements in a list less or equal to a given element. Missing values are ignored. The rank is the number of elements <= element. Ranks are of the form {0, 1, 2,..., sortedList.size()}. If no element is <= element, then the rank is zero. If the element lies in between two contained elements, then linear interpolation is used and a non integer value is returned.

Parameters:
sortedList - the list to be searched (must be sorted ascending).
element - the element to search for.
Returns:
the rank of the element.

sampleKurtosis

public static double sampleKurtosis(cern.colt.list.DoubleArrayList data,
                                    double mean,
                                    double sampleVariance)
Returns the sample kurtosis (aka excess) of a data sequence.

Parameters:
data - DoubleArrayList
mean - double
sampleVariance - double
Returns:
double

sampleSkew

public static double sampleSkew(cern.colt.list.DoubleArrayList data,
                                double mean,
                                double sampleVariance)
Returns the sample skew of a data sequence.

Parameters:
data - DoubleArrayList
mean - double
sampleVariance - double
Returns:
double

skew

public static double skew(cern.colt.list.DoubleArrayList data,
                          double mean,
                          double standardDeviation)
Returns the skew of a data sequence, which is moment(data,3,mean) / standardDeviation3.

Parameters:
data - DoubleArrayList
mean - double
standardDeviation - double
Returns:
double

sampleStandardDeviation

public static double sampleStandardDeviation(int size,
                                             double sampleVariance)
Returns the sample standard deviation.

This is included for compatibility with the superclass, but does not implement the correction used there.

Parameters:
size - the number of elements of the data sequence.
sampleVariance - the sample variance .
See Also:
Descriptive.sampleStandardDeviation(int, double)

sampleVariance

public static double sampleVariance(cern.colt.list.DoubleArrayList data,
                                    double mean)
Returns the sample variance of a data sequence. That is Sum ( (data[i]-mean)^2 ) / (data.size()-1).

Parameters:
data - DoubleArrayList
mean - double
Returns:
double

standardize

public static void standardize(cern.colt.list.DoubleArrayList data,
                               double mean,
                               double standardDeviation)
Modifies a data sequence to be standardized. Mising values are ignored. Changes each element data[i] as follows: data[i] = (data[i]-mean)/standardDeviation.

Parameters:
data - DoubleArrayList
mean - mean of data
standardDeviation - stdev of data

standardize

public static void standardize(cern.colt.list.DoubleArrayList data)
Standardize. Note that this does something slightly different than standardize in the superclass, because our sampleStandardDeviation does not use the correction of the superclass (which isn't really standard).

Parameters:
data - DoubleArrayList

sum

public static double sum(cern.colt.list.DoubleArrayList data)
Returns the sum of a data sequence. That is Sum( data[i] ).

Parameters:
data - DoubleArrayList
Returns:
double

sumOfInversions

public static double sumOfInversions(cern.colt.list.DoubleArrayList data,
                                     int from,
                                     int to)
Returns the sum of inversions of a data sequence, which is Sum( 1.0 / data[i]).

Parameters:
data - the data sequence.
from - the index of the first data element (inclusive).
to - the index of the last data element (inclusive).
Returns:
double

sumOfLogarithms

public static double sumOfLogarithms(cern.colt.list.DoubleArrayList data,
                                     int from,
                                     int to)
Returns the sum of logarithms of a data sequence, which is Sum( Log(data[i]). Missing values are ignored.

Parameters:
data - the data sequence.
from - the index of the first data element (inclusive).
to - the index of the last data element (inclusive).
Returns:
double

sumOfPowers

public static double sumOfPowers(cern.colt.list.DoubleArrayList data,
                                 int k)
Returns the sum of powers of a data sequence, which is Sum ( data[i]k ).

Parameters:
data - DoubleArrayList
k - int
Returns:
double

sumOfSquares

public static double sumOfSquares(cern.colt.list.DoubleArrayList data)
Returns the sum of squares of a data sequence. Skips missing values.

Parameters:
data - DoubleArrayList
Returns:
double

sumOfSquaredDeviations

public static double sumOfSquaredDeviations(cern.colt.list.DoubleArrayList data)
Compute the sum of the squared deviations from the mean of a data sequence. Missing values are ignored.

Parameters:
data - DoubleArrayList
Returns:
double

sumOfPowerDeviations

public static double sumOfPowerDeviations(cern.colt.list.DoubleArrayList data,
                                          int k,
                                          double c)
Returns Sum( (data[i]-c)k ); optimized for common parameters like c == 0.0 and/or k == -2 .. 4.

Parameters:
data - DoubleArrayList
k - int
c - double
Returns:
double

sumOfPowerDeviations

public static double sumOfPowerDeviations(cern.colt.list.DoubleArrayList data,
                                          int k,
                                          double c,
                                          int from,
                                          int to)
Returns Sum( (data[i]-c)k ) for all i = from .. to; optimized for common parameters like c == 0.0 and/or k == -2 .. 5. Missing values are ignored.

Parameters:
data - DoubleArrayList
k - int
c - double
from - int
to - int
Returns:
double

sizeWithoutMissingValues

public static int sizeWithoutMissingValues(cern.colt.list.DoubleArrayList list)
Return the size of the list, ignoring missing values.

Parameters:
list - DoubleArrayList
Returns:
int

trimmedMean

public static double trimmedMean(cern.colt.list.DoubleArrayList sortedData,
                                 double mean,
                                 int left,
                                 int right)
Returns the trimmed mean of a sorted data sequence. Missing values are completely ignored.

Parameters:
sortedData - the data sequence; must be sorted ascending .
mean - the mean of the (full) sorted data sequence.
left - int the number of leading elements to trim.
right - int number of trailing elements to trim.
Returns:
double

variance

public static double variance(cern.colt.list.DoubleArrayList data)
Provided for convenience!

Parameters:
data - DoubleArrayList
Returns:
double

weightedMean

public static double weightedMean(cern.colt.list.DoubleArrayList data,
                                  cern.colt.list.DoubleArrayList weights)
Returns the weighted mean of a data sequence. That is Sum (data[i] * weights[i]) / Sum ( weights[i] ).

Parameters:
data - DoubleArrayList
weights - DoubleArrayList
Returns:
double

winsorizedMean

public static double winsorizedMean(cern.colt.list.DoubleArrayList sortedData,
                                    double mean,
                                    int left,
                                    int right)
Not supported.

Parameters:
sortedData - DoubleArrayList
mean - double
left - int
right - int
Returns:
double
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
 PREV CLASS   NEXT CLASS FRAMES    NO FRAMES    
SUMMARY: NESTED | FIELD | CONSTR | METHOD DETAIL: FIELD | CONSTR | METHOD
Copyright © 2003-2005 Columbia University. All Rights Reserved.