2. UC & Performance Eval

Applications of data science to achieve business goals via prediction and insights from data.

• Value – potential ROI, operational improvement, revenue growth.

• Effort – time, resources, human capital needed.

• Risk – market, technology, regulatory uncertainties.

• Bubble diagrams in project portfolio management show value (size), effort & risk (position).

• Lack of right data → secure, unbiased data, address ethics.

• Unclear project goals → define problem, ask right questions, set final goal.

• Wrong team composition → include data scientists, engineers, analysts, domain experts.

• No clear methodology → combine DS project lifecycle with agile.

• Stop after deployment → continuously improve, ensure security.

• What is the value of the knowledge gained?

• What will be learned about dataset & hypothesis?

• How valuable are results for positive or negative predictions?

• customer-related

• operational-related

• fraud detection/security

• Goals: loyalty, acquisition, reduced costs, churn prevention.

• Questions:

  • Deal size drivers?

  • Customer journey?

  • Cost-effective acquisition?

  • Problematic features?

• Example: Linking purchase history + social media for targeted marketing → higher conversion, lower acquisition costs.

• Goals: cost reduction, service quality, asset optimization.

• Questions:

  • Predict maintenance/failures?

  • Impact of production changes?

  • Minimize processing time?

• Example: Vivint uses IoT data to reduce false alarms, improve efficiency.

• Goals: detect unauthorized access, fraud patterns, high-risk customers.

• Example: Detecting small fraudulent credit card transactions

1. Data collection: Internal/external DBs, sensors, web scraping, etc.

2. Preprocessing: Clean noise, remove redundant/missing values, select relevant features.

3. Training and testing datasets:

   • Training set: build and learn the model

   • testing set: evalutate the model’s accuracy

4. Prediction types:

   • Classification: categorized into classes (e.g., {fraud, not fraud}).

   • Regression (e.g., acquisition cost prediction).

5. E

• Evaluation:

  • Classification → threshold setting.

  • Regression → error margin (e.g., <5%).

• Updating: Retrain model with new data or feature changes.

• true positive (TP)

• true negative (TN)

• false positive (FP) —> type I classification error

• false negative (FN) —> type II classification error

• accuracy

• precision

• recall

• ration of the number of correct predictions to total predictions

• how correct the model is when returning a positive result

• how often the model produces true positives

• used if we are more tolerant to false positives thatn false negatives

• trade-off between the true positive rate and the false positive rate at every possible cutoff value

• ROC helps to find the best possible realistic treshold value which results in the highest TP rate and lowes FP rate

1. binary classifier trained on a labeled dataset produces the probability scores for each instance —> lowest predicted probability score is used to set the initial value of the treshold

2. assignt the testing set according to classes and count the TP, TN, FP, and FN values

3. Calculate FP rate and TP rate

4. point on ROC curve with coordinate: FP rate/TP rate

5. increase threshold value to the next predicted probability score, repeat steps 2-4

• evaluate, how close the model output (y) is to the actual output (d)

• absolute error

• relative error

• mean absolute percentage error

• square error

• mean square error

• mean absolute error

• root mean square error

• absolute difference between the model’s output (d) and the desired output (y)

• absolute error with respect to the desired output to obtain a unit-less percentage

• average relative error calculated over the entire testing set of n data records

• especially useful, when probability density distribution of the values is sufficiently far from zero, so that zero does not have a significant impact

• ensures that a positive quantity is obtained

• adds significant weight to large error values

• average square error ocer the entire testing set for n data records

• can be dominated by outliers

• more robust than the mean squared error to outliers

• square root of the mean square error

• easier to interpret than mean square error

• same scale of the desired outputs

• After prediction model evaluation → implement to deliver DSUC value.

• End users/decision-makers must check if DSUC value is achieved.

• Assessment via performance measures → Key Performance Indicators (KPIs).

• Distinction (Parmenter, 2020):

  • Result indicators – measure outcomes.

  • KPIs – focus on critical success factors, both current & future.

1. Easy to understand by all staff.

2. Measured frequently (daily/weekly).

3. Assigned to relevant task manager (including CEO).

4. Show positive/negative deviations from business objectives.

5. Non-financial (not in $/€).

6. Significant impact on organizational performance.

7. Clear responsibility assigned (specific teams accountable).

• Note: Non-financial criterion (5) often ignored in practice.

• Planned innovations.

• Late customer deliveries.

• Number of late projects.

• Downtime due to breakdowns or staff absence.

• Unresolved customer complaints.

• Number of vacant positions.

• New initiatives reported weekly.

• Employees in critical roles resigning.

• Measures % of employees leaving the company.

• Formula:

• Can be calculated for specific teams/time periods.

• Helps estimate satisfaction across organization.

• Both experts & non-experts are vulnerable to biases (Montibeller & Winterfeldt, 2015).

• Bias = prejudice influencing unfair judgment/decisions.

• Types:

  • Cognitive bias → systematic deviation due to culture/experience.

  • Motivational bias → influenced by desire for a preferred outcome.

• In data science: biases can distort datasets, processing & prediction models → inaccurate decisions.

• Awareness & de-biasing techniques are essential.

• action-oriented biases

• stability biases

• pattern-recognition biases

• interest and socail biases

• Favor action > inaction.

• Examples: overoptimism, overconfidence, bizarreness effect.

• Overconfidence → excessive belief in one’s judgment → ignores alternatives.

• Mitigation: humility, diverse perspectives, feedback, objective assessments.

• Preference for status quo / resistance to change.

• Examples: status quo bias, loss aversion, anchoring effect.

• Anchoring effect → first info strongly influences estimates (e.g., population guesses, negotiations).

• Mitigation: avoid anchors or use different expert anchors.

• Brain creates faulty rules/patterns.

• Example: Confirmation bias → seek info confirming existing beliefs.

• Mitigation: consult diverse experts, evaluate alternative hypotheses, apply probability assessments.

• Rooted in personal interests, desires, preferences.

• Influence analysis, data interpretation, emphasis.

• Social bias: conforming to group/social expectations.

• Probability training → improves understanding of statistics.

• Counterfactuals → rebalance data (e.g., replace “He is a doctor” → “She is a doctor”).

• Caution: removing one bias may introduce costs or new issues.

• Best strategy → prevent bias at dataset level + ensure data diversity.