Business Intelligence II

1. Data

2. Information

3. Knowledge

4. Insights

5. Wisdom

• Subtree-Replacement

• Subtree-Raising

information assets with volumes, velocities and/or variety requiring innovative forms of information processing for enhanced insight discovery, decision-making and process automation.

Volume

Velocity sources of data are growing rapidly

Variety heterogeneous sources of data

Value value after applying data analysis

• How the smart and intelligent machines are designed

• mathematical modelling

• Optimization of objective/target functions

• methods and algorithms

• ability to learn without being explicitly programmed

• teach themselves to grow and change when exposed to new data

• Discovering a hidden knowledge and patterns

• data, finding patterns

• integration of strategies, processes and technologies

• in order to deliver strategic knowledge about status, potentials and perspectives

• from distributed and heterogeneous enterprise, market and competitive data

iterdisciplinary field about processes and systems to extract knowledge or insights from data in various forms, either structured or unstructured

• Supervised Learning

• Unsupervised Learning

• Reinforcement Learning

Knowledge Discorvery in Databases

Cross Industry Standard Process for Data Mining

• generic process model

• describes commonly used approaches that data mining and machine learning experts use to tackle problems

• People/Team

• Data

• Soft-/Hardware

• Many stakeholders

• High/wrong expectations

• wrong datasets (biased, noisy, wrong, …)

• company has no data driven decision culture

• lack of expertise in DA/ML/DL/DE

• maps data to output

• Learning Algrorithms can be used to aqquiere this Model with a Learning Data

• Bias

• Cherry Picking

• False Causality

• McNarmara Fallacy

• Danger of Summary

• Simpson’s Paradox

• Overfitting

Drawing conclusions from a set of data that isn’t representative of the population you’re trying to understand.

The practice of selecting results that fit your claim and excluding those that don’t. The worst and most harmful example of being dishonest with data.

To falsely assume when two events occur together that one must have caused the other

Relying solely on metrics in complex situations can cause you to lose sight of the bigger picture.

It can be misleading to only look at the summary metrics of data sets.

A phenomenon in which a trend appears in different groups of data, but disappears or reverses when the groups are combined.

A more complex explanation will often describe your data better than a simple one. However, a simpler explanation is usually more representative of the underlying relationship.

• use reallife based models (instead of playground data)

• Feature Scaling (standartisation)

• Feature Selection (preprocessing)

• cross-validation (use a test learning set)

• do just right fit (model interpration)

• Look at data before results

• Naive Baseline Model

• Understand costfunction

correct, complete and (as far as possible) redundancy-free data combined from different data sources and stored in a (semi-)structured data store

process of obtaining information resources that are relevant to an information need (e.g. full-text searches)

• File System

• File

• Markup File

• Database

• Web Forms

• Web Services

• Applications

• Data Warehouse Systems

• Federated database system

• Portals, Integration of News

• (Meta-) Search engines

subject-oriented, integrated, non-volatile, and time-variant collection of data in support of managements decisions

• iterative process

• over long period of time

• may have various evaluation perspectives / goals

• requires cooperation and collaboration of different actors (e.g., stakeholders, IT systems, etc.)

• Methodology

• Expertise

• Domain knowledge and intuition

• Data Description

• Exploration of Data

• Dimension reduction

• Model Development

• Improvement of the existing models

• Prediction and classification

• Comparison of different groups

• Hypothesis Testing

Exploratory Data Analysis

• Exporative Data Analysis (EDA)

• Knowledge Discovery in Database

• Statistical Analysis

• the nontrivial process of identifying valid, novel, potentially, useful and ultimately understandable patterns in data

• is an interactive, multi-step and iterative process

• data mining is a central step

1. Pre-Processing

2. Scheme Comparison

3. Scheme Adaptation

4. Integration & Restructering

• Bottum-Up

• To-Down

• Explicit Value Mapping

• similarity measures

• original/reference data source

• Usage of general knowledge or domain-specific knowledge

• Completeness

• Correctness

• Sufficiency

• Complexity

The integrated schema must include all concepts contained in any local schema; no information contained in a local schema should be lost or ignored

• all information contained in the integrated schema must be semantically equivalent to at least one local schema

• inter-schema relationships newly added during integration must not contradict information from the local schemas i.e. only consistent extensions to the existing schemas allowed

Real-world concepts modeled in multiple local schemas should be represented only once in the integrated schema

The integrated scheme should be easy to understand

• Stem-and-Leaf Display ()

• Histogram

• Box-Plot

• Scatter

• Curves

Time-Series Data

Spatially structured data - Special Maps

1. Business Understanding

2. Data Preparation

3. Modelling

4. Evaluation

5. Deployment

1. task analysis

2. preprocessing

3. data mining

4. postprocessing

5. deployment

• task analysis

• preprocessing

• postprocessing

analysis of relationships in transactions (e.g. crossselling)

to form a cluster from similar objects

set of objects, which have

• high degree of similarity to each other

• lowest possible degree of similarity to other objects outside the cluster

• partioning clustering

• hierachical clustering

• distibution-based clustering

• density-based clustering

• grid-based clustering

• model-based clustering

assign an object to (predefined) classes

• Training

• Application

extrapolation of a given time series into the future

reduced interpretability (black box)

• incremental

• non-incremental

a particular subsets of a training data could be processes multiple times by the algorithm for the decision tree construction

operate on the whole dataset at once

1. Form Tree Process

2. Growth Tree Process

3. Prune Tree Process