DataViz

dt <- dt[!is.na(value)]

# Histogram is used to show the distribution of a variable (mostly continuous),

# while barplot is used to compare between groups.

# In histogram, each bar represent a group of binned quantitative data,

# while in barplot, each bar represent a discrete category.

# Pearson correlation measures the linear relationship of two continues variables, while spearman rank correlation measures the monotonic relationships between two continuous or ordinal variables.

Use pearson correlation when the two variables to compare are linearly related, otherwise use spearman correlation.

# 1. Boxplot, with computed statistics (quantiles, median, outliers)

# 2. Violin plot

# 3. Histogram or density plot

# 4. Ecdf plot

1. Choose the K initial centroids (one for each cluster). Different methods such as sampling random observations are available for this task.

2. Assign each observation xi to its nearest centroid by computing the Euclidean distance between each observation to each centroid.

3. Update the centroids μk by taking the mean value of all of the observations assigned to each previous centroid.

4. Repeat steps 2 and 3 until the difference between new and former centroids is less than a previously defined threshold.

• The number of clusters K is properly selected

• The clusters are isotropically distributed, i.e., in each cluster the variables are not correlated and have equal variance

• The clusters have equal (or similar) variance

• The clusters are of similar size

• normalized data

it relies on a predefined number of clusters.

We describe bottom-up (a.k.a. agglomerative) hierarchical clustering:

1. Initialization: Compute all the n(n−1)/2�(�−1)/2 pairwise dissimilarities between the n� observations. Treat each observation as its own cluster. A typically dissimilarity measure is the Euclidean distance. Other dissimilarities can be used (1-correlation), Manhattan distance, etc.

2. For i=n,n−1,...,2�=�,�−1,...,2:

• Fuse the two clusters that are least dissimilar. The dissimilarity between these two clusters indicates the height in the dendrogram at which the fusion should be placed.

• Compute the new pairwise inter-cluster dissimilarities among the i−1�−1 remaining clusters using the linkage rule.

The linkage rules define dissimilarity between clusters. Here are four popular linkage rules:

• Complete: The dissimilarity between cluster A and cluster B is the largest dissimilarity between any element of A and any element of B.

• Single: The dissimilarity between cluster A and cluster B is the smallest dissimilarity between any element of A and any element of B. Single linkage can result in extended, trailing clusters in which single observations are fused one-at-a-time.

• Average: The dissimilarity between cluster A and cluster B is the average dissimilarity between any element of A and any element of B.

• Centroid: The dissimilarity between cluster A and cluster B is the dissimilarity between the centroids (mean vector) of A and B. Centroid linkage can result in undesirable inversions.

The time complexity of K-Means clustering is linear, while that of hierarchical clustering is quadratic.

In K-Means clustering, we start with a random choice of centroids for each cluster. Hence, the results produced by the algorithm depend on the initialization.

K-Means clustering requires the number of clusters a priori.

The Rand index is a measure of the similarity between two partitions.

For Principal Component Analysis (Pearson, 1901), this representation is the projection of the data on the subspace of dimension q that is closest to the data according to the sums of the squared Euclidean distances.

The P-value is the probability of obtaining a test statistic the same as or more extreme than the one we actually observed, under the assumption that the null hypothesis is true.

The formal definition of the P-value depends on whether we take “more extreme” to mean greater, less, or either way:

For a univariate data these are histograms, single boxplots or violin plots. For multivariate data these are clustered heatmaps, PCA projections, etc.

Associative plots show how a variable depends on another variable.

Suitable plots are side-by-side boxplots, scatter plots, etc.

This phenomenon, where a variable X seems to relate to a second variable Y in a certain way, but flips direction when the stratifying for another variable Z (grouping by Z), has since been referred to as Simpson’s paradox.

Reverse causality states that if A and B correlate one might falsely interpret that A causes B even though in reality B causes A

We take a sample of size n, with replacement, from our observed data, to make a new dataset.

A confidence interval of confidence level 1−α for a parameter θ is an interval C=(a,b) which would the data generation process be repeated, would contain the parameter with probability 1−α, i.e. p(θ∈C)=1−α. A typical value is α=0.05 which leads to 95% confidence intervals.

(a) is wrong. p is not random and thus it either is in the interval or it is not.

(b) and (c) are wrong for the same reason as (a). Moreover, we did not even specify a null hypothesis This is not necessary to compute a confidence interval.

(d) is correct. In fact, this is a restatement of the definition of a 95% confidence interval

(a) is correct

(b) is wrong. The alternative=greater

# asks whether the upper left cell

# is significantly bigger than we expect

# This is not the case here

(c) is correct

is wrong. It depends on the difference in mean

is wrong. Bimodality is a deviation from normality # In a bimodal distribution, the mean might not be meaningful

Correct

is wrong. We never accept the null! # There may be a difference # But we may have too little data # Or too much noise to detect it

X and Y come from the same distribution, i.e. values do not depend on the group.

two quantitatives: X and Y do not correlate

multiple testing refers to the issue that, if we test enough hypotheses at a given significance level, say α=0.05, we are bound to eventually get a significant result, even if the null hypothesis is always true.

Family-wise error rate (FWER): p(V>0)�(�>0), the probability of one or more false positives

The false discovery rate

Controlling the Family-wise error rate ensures we have few false positives, but it comes at the cost of many false negatives.

the expected fraction of false positives among all discoveries:

, where max(R, 1) ensures the denominator to not be 0.

As expected, we see that the nominal P−value cutoff is the most lenient, the FWER one (Bonferroni) the most stringent and the FDR ones (Benjamini-Hochberg) is intermediate.

As the sample size increases, even very small effects may become significant, but that does not mean that they actually matter.

Moreover, we saw that when we run many tests, some are bound to reject the null, even if the null is always true. We thus need to apply a correction.

The effect size is a quantitative measure of the magnitude of a phenomenon. Examples of effect sizes are the correlation between two variables, the regression coefficient in a regression, the mean difference, or even the risk with which something happens, such as how many people survive after a heart attack for every one person that does not survive.

The effect size determines whether an effect is actually important.

10

is correct.

# The bonferroni is generally

# more conservative # Thus it demands lower P-values to reject

# Lower P-values require more permutations

less than 1%

wrong. We expect that less than 1% of the positives will be false positives

• To test conditional dependence. This is done by testing the null hypothesis:

  • H0:βj=0.

• To estimate the effects of one variable on the response variable. This is done by providing an estimate of the coefficient βj.

• To predict the value of the response variable given values of the explanatory variables. The predicted value is then an estimate of the conditional expectation E[y|x].

• To quantify how much variation of a response variable can be explained by a set of explanatory variables.

The proportion of variance explained by the model

The expected values of the response are a linear combinations of the explanatory variables.

Errors are identically and independently distributed.

Errors follow a normal distribution with constant variance across all explanatory variable values

the residuals also follow a Gaussian distribution. The Q-Q plot supports such distribution.

# Linear models minimize the RSS along the predicted coordinate.

# In this specific case, such coordinate is either the male student's height (first model of the students heights) or the fathers height (second model predicting the fathers heights).

# PCA minimizes the sum of squares (along both coordinates). And hence, the distance perpendicular to the first principal component. PC1 is the line with respect to which the sum of

# PC1 represents the line such that the sum of the squared distances of every point (x,y) to it is the minimum across all possible lines

The sensitivity refers to the fraction of actual positives that is predicted to be positive:

The sensitivity is also referred to as “recall,” “true positive rate,” or “power.”

The specificity refers to the fraction of actual negatives that is predicted to be negative:

The specificity is also known as “true negative rate” or “sensitivity of the negative class”

The precision refers to the fraction of predicted positives that are indeed positives:

The precision is also called the positive predictive value. Note that, in the hypothesis testing context, we discussed a related concept, the false discovery rate (FDR). The FDR relates to the precision as follows:

The receiver operating characteristic curve or ROC curve is a way of evaluating the quality of a binary classifier at different cutoffs. It describes on the x axis the false positive rate (1-specificity), and on the y axis the true positive rate (sensitivity).

c

The odds for a binary variable y are defined as

The residual is the difference between the observed value and the estimated value of the quantity of interest (for example, a sample mean)

The error of an observation is the deviation of the observed value from the true value of a quantity of interest (for example, a population mean).

The welch test assumes unequal variance

The idea of the the Wilcoxon rank-sum test is that under the null hypothesis, the xi’s and yi’s should be well interleaved in this ranking.

Variance is not constant: heteroscedascity

# transformation of the response y

# - log transformation

# - square root transformation

# - variance stabilizing transformation

# Use a generalized linar model

Linearity

# investigate further terms e.g. x**2

boxplots are not well suited for bimodal data, since they only show one mode (the median)

Boxplots are also not suited for categorical data and discrete data with very few values, for which bar plots are preferred