lind.analysis package

Submodules

lind.analysis.freq module

Frequentist Tests

Note on estimating the population variance: We often use n-1 instead of n when estimating the population variance (Bessel’s correction), where n is the number of samples. This method corrects the bias in the estimation of the population variance. It also partially corrects the bias in the estimation of the population standard deviation. However, the correction often increases the mean squared error in these estimations. When n is large this correction is small.

TODO: add continuity corrections to unpooled z tests

lind.analysis.freq.find_p_value(test_statistic: float, df: float = inf, tails: bool = True) → float

A convenience function for finding p values for t tests and z tests.

Notes

  • sf is 1 - cdf

Parameters

  • test_statistic (float) – The t or z test statistic

  • df (float) – The degrees freedom. If infinity (np.inf), this is assumed to be a z test. Otherwise it is assumed to be a t-test.

  • tails (bool) – An indicator for two tailed tests. If True, this assumes a two tailed test. If False, this assumes a one tailed test.

Returns

The p value corresponding to the test statistic.

Return type

float

lind.analysis.freq.find_test_statistic(p_value: float, df: float = inf, tails: bool = True) → float

A convenience function for recovering t and z test statics from p-values

Parameters

  • p_value (float) – The p-value of interest

  • df (float) – The degrees freedom. If infinity (np.inf), this is assumed to be a z test. Otherwise it is assumed to be a t-test. The degrees freedom is usually the number of total samples minus one for the t test.

  • tails (bool) – An indicator for two tailed tests. If True, this assumes a two tailed test. If False, this assumes a one tailed test.

Returns

The corresponding test statistic

Return type

float

lind.analysis.freq.find_confidence_interval(se: float, df: float = inf, alpha: float = 0.05, tails: float = True) → float

A convenience function for finding the confidence interval based on the standard error.

Parameters

  • se (float) – The standard error of the measurement (estimate).

  • df (float) – The degrees freedom. If infinity (np.inf), this is assumed to be a z test. Otherwise it is assumed to be a t-test.

  • alpha (float) – The probability of making a type I error. A 95% credible interval has alpha = 5% or .05.

  • tails (bool) – An indicator for two tailed tests. If True, this assumes a two tailed test. If False, this assumes a one tailed test.

Returns

The width of the confidence interval (absolute units).

Return type

float

lind.analysis.freq.one_samp_z_prop(sample: numpy.ndarray, null_h: float = 0.5) → Tuple[float, float]

Function for one sample z test of proportions.

Parameters

  • sample (ndarray) – An array of samples (observations).

  • null_h (float) – The point null hypothesis to use when comparing the means.

Returns

  • float – test statistic (t statistic)

  • float – standard error

lind.analysis.freq.two_samp_z_prop(sample1: numpy.ndarray, sample2: numpy.ndarray, null_h: float = 0.0, pooled: bool = False) → Tuple[float, float]

Function for two sample z test of proportions.

Parameters

  • sample1 (ndarray) – A numpy array with the unit level data from the first sample.

  • sample2 (ndarray) – A numpy array with the unit level data from the second sample.

  • null_h (float) – The point null hypothesis to use when comparing the means.

  • pooled (bool) – Indicates whether to use the assumption that the sample variances are equal or not. Pooled = True assumes that the variances are equal. It is common to use the pooled assumption given that the unpooled assumption yields over confident estimates in practice (barring the appropriate corrections).

Returns

  • float – test statistic (t statistic)

  • float – standard error

lind.analysis.freq.one_samp_z(sample: numpy.ndarray, null_h: float = 0.0) → Tuple[float, float]

Function for one sample z test.

Parameters

  • sample (ndarray) – An array of samples (observations).

  • null_h (float) – The point null hypothesis to use when comparing the means.

Returns

  • float – test statistic (t statistic)

  • float – standard error

lind.analysis.freq.two_samp_z(sample1: numpy.ndarray, sample2: numpy.ndarray, null_h: float = 0.0, pooled: bool = False) → Tuple[float, float]

Function for a two sample z test.

Parameters

  • sample1 (ndarray) – A numpy array with the unit level data from the first sample.

  • sample2 (ndarray) – A numpy array with the unit level data from the second sample.

  • null_h (float) – The point null hypothesis to use when comparing the means.

  • pooled (bool) – Indicates whether to use the assumption that the sample variances are equal or not. Pooled = True assumes that the variances are equal. It is common to use the pooled assumption given that the unpooled assumption yields over confident estimates in practice (barring the appropriate corrections).

Returns

  • float – test statistic (t statistic)

  • float – standard error

lind.analysis.freq.one_samp_t(sample: numpy.ndarray, null_h: float = 0.0) → Tuple[float, float, float]
Parameters

  • sample (ndarray) – An array of samples (observations).

  • null_h (float) – The point null hypothesis to use when comparing the means.

Returns

  • float – test statistic (t statistic)

  • float – standard error

  • float – degrees freedom

lind.analysis.freq.two_samp_t(sample1: numpy.ndarray, sample2: numpy.ndarray, null_h: float = 0.0, pooled: bool = False) → Tuple[float, float, float]

A simple function for running two sample student t tests.

Parameters

  • sample1 (ndarray) – A numpy array with the unit level data from the first sample.

  • sample2 (ndarray) – A numpy array with the unit level data from the second sample.

  • null_h (float) – The point null hypothesis to use when comparing the means.

  • pooled (bool) – Indicates whether to use the assumptions that the sample variances are equal or not. Pooled = True assumes that the variances are equal. The un-pooled t-test is sometimes called Welch’s t-test

Returns

  • float – test statistic (t statistic)

  • float – standard error

  • float – degrees freedom

Examples

>>> t_stat, se, df = two_samp_t(
>>>     sample1 = np.asarray([1, 2, 3]),
>>>     sample2 = np.asarray([1, 2, 3]),
>>>     null_h = 0.0, pooled = False
>>> )

References

Bernard Rosner

  • Eq 8.11, 8.21 Fundamentals of Biostatistics

lind.analysis.multiple_comparison_procedures module

multiple_comparison_procedures: This module contains functions to correct for multiple comparisons in hypothesis testing. Corrected p values will be referred to as q values. Multiple comparison procedures (MCPs) are sometimes referred to as correction factors.

MCPs tend to correct for family wise error rate (FWER) or false discovery rate (FDR).

Author’s Note: This code draws from example code referenced in Rosetta Code. That code is a translation of code in the R stats package and falls under the R 4.1.0 license (GPL v2).

Most of the code in this module is based on R source code covered by the GPL license. It is thus a modified version covered by the GPL.

FDR-controlling procedures are designed to control the expected proportion of “discoveries” (rejected null hypotheses) that are false (incorrect rejections of the null). FDR-controlling procedures provide less stringent control of Type I errors compared to family-wise error rate (FWER) controlling procedures, which control the probability of at least one Type I error.

FDR-controlling procedures have greater power, at the cost of increased numbers of Type I errors. The power of a binary hypothesis test is the probability that the test rejects the null hypothesis when a specific alternative hypothesis is true (i.e. the probability of avoiding a type II error).

A type I error is the rejection of a true null hypothesis (“false positive”), while a type II error is the non-rejection of a false null hypothesis (“false negative”).

Sometimes experimenters prefer to use the language of sensitivity ans specificity. Sensitivity measures the proportion of positives that are correctly identified. Specificity measures the proportion of negatives that are correctly identified.

Recommended import style: >>> from lind.analysis import multiple_comparison_procedures as mcp

TODO: There are parts of this code written by convenience that could be vectorized for increased

speed. Will need to update code in the future to better optimize for speed.

TODO: Add better docstrings to private utility functions

lind.analysis.multiple_comparison_procedures.bh(p_values: Union[List, numpy.ndarray]) → numpy.ndarray

bh: Benjamini-Hochberg

The Benjamini and Hochberg method controls the false discovery rate (FDR), the expected proportion of false discoveries amongst the rejected hypotheses. The FDR is a less stringent condition than the family-wise error rate (so bh is more powerful than many other correction methods).

Parameters

p_values (Union[List[float], ndarray[float]]) – List or numpy array of p values to be converted into q values

Returns

the q values for the respective p value inputs

Return type

ndarray[float]

Examples

>>> q_values = bh([0.05, 0.002, 0.006])

References

Benjamini and Hochberg

  • Controlling the false discovery rate: a practical and powerful approach to multiple testing

lind.analysis.multiple_comparison_procedures.by(p_values: Union[List, numpy.ndarray]) → numpy.ndarray

by: Benjamini-Yekutieli

The Benjamini and Yekutieli method controls the false discovery rate (FDR), the expected proportion of false discoveries amongst the rejected hypotheses. The FDR is a less stringent condition than the family-wise error rate (so bh is more powerful than many other correction methods).

Parameters

p_values (Union[List[float], ndarray[float]]) – List or numpy array of p values to be converted into q values

Returns

the q values for the respective p value inputs

Return type

ndarray[float]

Examples

>>> q_values = by([0.05, 0.002, 0.006])

References

Benjamini and Yekutieli

  • The control of the false discovery rate in multiple testing under dependency.

lind.analysis.multiple_comparison_procedures.bonferonni(p_values: Union[List, numpy.ndarray]) → numpy.ndarray

bonferonni: Bonferonni

Correction to control for family wise error rate (FWER).

Recommended to use Hold instead of the uncorrected Bonferonni. Holm is more powerful and valid under the same assumption.

Parameters

p_values (Union[List[float], ndarray[float]]) – List or numpy array of p values to be converted into q values

Returns

the q values for the respective p value inputs

Return type

ndarray[float]

Examples

>>> q_values = bonferonni([0.05, 0.002, 0.006])

References

Bonferroni

  • Teoria statistica delle classi e calcolo delle probabilità

Shaffer

  • Multiple Hypothesis Testing

lind.analysis.multiple_comparison_procedures.holm(p_values: Union[List, numpy.ndarray]) → numpy.ndarray

holm: Holm

Correction to control for family wise error rate (FWER).

More powerful than unmodified Bonferonni and valid under the same assumption.

Parameters

p_values (Union[List[float], ndarray[float]]) – List or numpy array of p values to be converted into q values

Returns

the q values for the respective p value inputs

Return type

ndarray[float]

Examples

>>> q_values = holm([0.05, 0.002, 0.006])

References

Holm

  • A simple sequentially rejective multiple test procedure

lind.analysis.multiple_comparison_procedures.hommel(p_values: Union[List, numpy.ndarray]) → numpy.ndarray

hommel: Hommel

Correction to control for family wise error rate (FWER).

Valid when the hypothesis tests are independent or when they are non-negatively associated (see Sarkar). Slightly more powerful than Hochberg but also slower.

Parameters

p_values (Union[List[float], ndarray[float]]) – List or numpy array of p values to be converted into q values

Returns

the q values for the respective p value inputs

Return type

ndarray[float]

Examples

>>> q_values = hommel([0.05, 0.002, 0.006])

References

Hommel

  • A stagewise rejective multiple test procedure based on a modified Bonferroni test

Sarkar

  • Some probability inequalities for ordered MTP2 random variables: a proof of Simes conjecture

  • The Simes method for multiple hypothesis testing with positively dependent test statistics

lind.analysis.multiple_comparison_procedures.hochberg(p_values: Union[List, numpy.ndarray]) → numpy.ndarray

hochberg: Hochberg

Correction to control for family wise error rate (FWER).

Valid when the hypothesis tests are independent or when they are non-negatively associated (see Sarkar). Slightly less powerful than Hochberg but also faster.

Parameters

p_values (Union[List[float], ndarray[float]]) – List or numpy array of p values to be converted into q values

Returns

the q values for the respective p value inputs

Return type

ndarray[float]

Examples

>>> q_values = hochberg([0.05, 0.002, 0.006])

References

Hochberg

A sharper Bonferroni procedure for multiple tests of significance by

Sarkar

  • Some probability inequalities for ordered MTP2 random variables: a proof of Simes conjecture

  • The Simes method for multiple hypothesis testing with positively dependent test statistics

Module contents

Analysis

This module is dedicated to analysis of experiments (AOE).