02_Exploratory data analysis (EDA)

Numerical (or quantitv):

• discrete data: countable

• continuous data: measurable (length, weight, temperature)

Categorical (or qualitiv)

• nominal data: if you can brand it, tick it off in a checkbox (gender, color)

• oridnal data: if you can rank or order it ( eg. a checkbox with options -> bad, okay, good, very good)

= numercial, because

• arithmetic operations are possible

• enumeration is possible

• consistent interval differences

• suitable for statistical analysis

• visualization possible: pie charts, bar charts, various plots

-> if data is not numerical we often transform it to a numerical data type

• arithmetic operations are not possible

• enumeration is possible

• no consistent interval differences

• partly suitable for statistical analysis (eg. test statistics, regression)

• visualization possible: pie charts, bar charts, tables

-> can often be converted to a numerical scale (with assumptions)

• arithmetic operations are not possible

• enumeration is not possible

• no consistent interval differences

• partly suitable for statistical analysis (eg. test statistics, regression)

• visualization possible: pie charts, bar charts, tables

-> often converted to a numerical data by One-hot-encoding

• technique to transform categorical into numerical features

• Example: matrix of cat, dog, mouse as feature with one when true

  
  

  
  

• structured: well defined and highly organized (e.g. SQL requests)

• unstructured: can be anything that is not in a specific format (e.g. audio or video format, emails)

• both data types eed preprocessing/ feature extraction

• CSV: comma separated values

• JSON: JavaScript Object Notation

• it is a table or a 2D array-like structure

• visualization can be done with package pandas

• data frames can be created from dictionaries or importing files such as CSVs

1. A plot or table conveys a message

2. Show the data (use appropriate plot and plot should be clear

3. Compare things directly (Plot on the same graph and plot (log) ratio)

4. Be careful with scale (use of common scales and consider starting a zero)

• no proper axes description

• comparing without using the same sclae for the x or y axis

• inconsistent use of color

• too much information in one plot

• sample mean (dt. Stichprobenmittelwert)

• is the maximum likelihood estimator for the expected value of the gaussian distribution (für den Erwartungswert einer normalverteilten Grundgesamtheit)

• certainty (Konfindenzniveau): depends on the number of underlying samples -> use with standard deviation or percentiles

• not robust (highly sensitiv to outliers) -> consider using median

• often oversimplyfying complex relationships -> start with mean or median, the go more complex if need

• standard deviation: measure of variability of data

• standard error (of the mean): its quality also depends on the number of samples (explanation missing)

= the x-th percentile means that x% of values fall at or below that number.

• the process of classifying your data into different groups so you can do separate analyses

• typically by age, gedenr, ethnicity, income, education etc.

Example: My classifier went from 30% to 60% accuracy

• Its accuracy increased by 30 percentage points

• it i 100% more accurate

• Business problem: Who is user, stakeholder? What are their pain points?

• proof if value: Is it possible to solve the problem?

• MVP (minimal viable product): How would the customer use the solution?

• Product definition: Clear product vision, Roadmap, Budget

• Business problem: used as a challenger, for documentation

• proof if value: sparring partner, accelerator

• MVP (minimal viable product): Builder or Multiplier

• Product definition: Documentation, Mock Ups, Requirements