3. Data Preprocessing

• Definition: transfer of data between devices via streams/channels.

• Increasing importance due to growing data volumes.

• Reliability & high-speed transfer critical for businesses.

• Example (IoT in logistics): GPS/sensors transmit location, temperature, humidity → enable real-time monitoring & fast responses to issues.

• Mathematical study of data transmission & processing limits.

• Any information can be represented in binary (0/1).

• Defined channel capacity considering noise.

• Maximized channel efficiency with encoding schemes.

• Developed communication model with 6 elements:

  1. Source (origin of message).

  2. Encoder (message → signals, adds disguise/adaptation).

  3. Channel (medium: wires, air, fiber).

  4. Noise (interference: sounds, weather).

  5. Decoder (signal → understandable format).

  6. Receiver (final entity).

• minimum compressed data volume, measure of information content.

• defines limits of data compression.

• error detection in noisy transmissions.

• Framework describing interaction of hardware & software for network communication.

1. Physical layer → physical resources (cables, hubs, modems), transmission of signals.

2. Data link layer → LLC (protocols, flow/error control) + MAC (linking physical & LLC). Detects/corrects errors.

3. Network layer → IP addressing, routing, forwarding packets.

4. Transport layer → packet order, protocols: TCP (reliable) & UDP (fast, no guarantee).

5. Session layer → maintains device communication (authentication, reconnections).

6. Presentation layer → “translator” (formatting, syntax, encryption/decryption).

7. Application layer → user interface for network resources (browsers, email, FTP).

• Data from multiple sources often suffers from issues: noisy, inaccurate, incomplete, inconsistent, missing, duplicate, or outlying values.

• Key quality factors: accuracy, interpretability, consistency, completeness.

• Poor quality affects ML models → inaccurate predictions & unreliable results.

1. Data cleaning (fix missing values, outliers, duplicates).

2. Data transformation.

3. Dimensionality reduction.

• Missing values: unobserved or incorrectly recorded data.

• Outliers:

  • “True” outliers → real but unusual events.

  • “Fake” outliers → data errors.

1. Removal of records (valid for large datasets; risky if important info is lost).

2. Linear interpolation (estimate missing values from neighboring points).

3. Mean imputation (replace with average of variable).

4. Mode imputation (replace with most frequent value).

add indicator variable (0 = original, 1 = modified).

• Common in datasets from multiple sources (e.g., separate regional customer databases).

• Duplicates can distort analysis → e.g., splitting customer history across records.

• Leads to inefficiency, wrong insights, longer computing time.

• Good practice: detect & remove duplicates to improve accuracy & efficiency.

• consider unit of measurement

• during preprocessing stage

• transforming the data in such way that the new values fall within a smaller or common rage, e. g. between 0 and 1

• gives all the attributes an equal weight

• prevents one attribute from dominating the results due to its larger magnitude

• linear transformation of the original data

• scales the data into a fixed range, typically between the values 0 (min) and 1 (max)

• transforms numerical data so that it has a mean of 0 and a standard deviation of 1

• how many standard deviations a particular data point is away from the mean

• shift the decimal point of the original data value by using a factor of 10

• Definition: X= log(x)

• Purpose: Applied in linear regression when linearity between input variables is missing.

• Example: Population data → log transformation makes population vs. area relationship more linear.

• Effect:

  • Compresses large values.

  • Stretches small values.

  • Produces a more balanced range of values.

• Definition: Converts continuous values (e.g., age) into intervals (0–18, 19–30) or conceptual labels (child, adult).

• Purpose:

  • Handles non-linear continuous variables.

  • Required when algorithms need categorical inputs.

• Types:

  • Supervised: Uses class information.

  • Unsupervised: No class information used.

• Binning (unsupervised)

• Histogram Analysis (unsupervised)

1. Binning (unsupervised)

   • Groups continuous values into intervals (“bins”).

   • Values replaced by bin mean or median (smoothing).

   • Example: Prices [1,1,3,4,5,6,7,7,9,9] → bins.

     • Equal width: same interval size (1–3, 4–6, 7–9).

     • Equal frequency: bins contain same number of values.

1. Histogram Analysis (unsupervised)

   • Graphical representation of value distribution.

   • Divides values into disjoint buckets.

   • Buckets with one value are “singleton buckets.”

• Goal: Reduce number of variables without losing main dataset properties.

• Benefits:

  • Less execution time & storage.

  • Removes irrelevant features → higher model accuracy.

  • Easier interpretation & visualization.

• Feature Selection

• Correlation Analysis

• Feature Extraction

• Selects a subset of relevant features.

• Removes noisy or biased data.

• Lowers model complexity → more efficient & accurate.

• Measures relationships between feature pairs.

• Correlation coefficient (ρ):

  • Range: -1 to 1.

  • ρ = 1 → strong positive correlation.

  • ρ = 0 → no correlation.

  • ρ = -1 → strong negative correlation.

• Highly correlated features → one can be removed.

• Methods: Pearson, Kendall, Spearman, Point-Biserial, Chi-squared test.

• Transforms original features into new, lower-dimensional ones.

• Goal: Keep relevant info, reduce overfitting & complexity.

• Examples:

  • PCA (Principal Component Analysis): Creates new “principal components” capturing maximum variance.

  • LDA (Linear Discriminant Analysis).

  • KPCA (Kernel PCA).

• Applications: healthcare data, image resizing, stock data analysis.