Data Mining Process

• Reproducibility

• Verifiability: Can resuls be verified?

=> Formal process with comprehensive documentation to remove some of the “arts” stuff

Fayyads Knowledge Dicovery in Databases Process:

• Framework for the process of dicovering knowledge from data

  -> Data Mining is therefore one major key activity in the KDD process

  • Consists of Applying data analysis and discovery algorithms to discover patterns, relationships and other knowledge from prepocessed and transformed data

1. Data Selection

2. Preprocessing

3. Transforamtion

4. Data Mining Interpretation

5. Gain of knowledge

Sample, Explore, Modify, Model and Assess Process:

-> Similar to Fayyad: Focus only on core data mining processes

1. Sample: Selection of dataset for modeling

2. Explore: Understanding of the data (incl. visualization)

3. Modify: Preparation for data modeling (Preprocessing, Cleaning, Variable selection, etc.)

4. Model: Application fo various modeling (data mining) techniques

5. Assess: Evaluation of modeling results

Cross-Industry Standard Process for Data Mining:

1. Business Unterstanding (Business Objectives, Assessment of Situation, Data Mining Goalds and Project Plan)

2. Data Understanding (Collection of initial data, Data Description, Data Exploration, Verify Data Quality)

3. Data Preparation (Data Selection, Data Cleaning, Data Construction, Integration of Data, Formatting of Data)

4. Modeling (Selection of Modeling Technique, Generation of Test Design, Building of model(s), Assessment of model)

5. Evaluation (Evaluation of Results

   -> Business Success Criteria, Review of Process, Determination of next Steps)

6. Deployment (Planning of Deployment, Planning of Monitoring and Maintenance, Production of Final Report, Review of Project)

Analytics Solutions Unified Method for Data Mining/ Predictive Analytics

-> Extension of CRISP-DM

-> Additionaly inclues Infrastructure/ operations aspects, Project Management, Detailed Deployment

1. Create Production Data Files: Load all data needed for operation into production environment

2. Create and Perform Operational Readiness Testing

3. Migrate/ Restore QU Model into Production

Standard for Data Mining which includes:

• Terminologies and Definitions

• Machine Learning Concepts and algorithms

• Most important and generalized Stakeholder roles

• AI System Lifecycle (+ Re-evaulation and Retirement of AI Systems)

• Business Understanding

• Testing in the Modeling phase

• Unacceptable risk: forbidden

  • Significant potential to manipulate persons, real-time biometric identification in public space, Social-Scoring

• High Risk: Compliance required

  • Biometric identificationa and categorization of natural persons, management and operation of critical infrastructure, education, law enforcement

• ‘Transparency Risk‘: Information and Transparency obligations

• Minimal or no risk: Permitted

• User-to-Data

• Data-to-Algorithm

• Algorithm-to-User

• Human cognitive biases

• Data biases

Measurement/ description fo an instance

• Nominal: Distinct labels from a defined vocabulary

• Ordinal: Impose an order on discrete categories (can be distinct labels from a defined vocabulary, numeric or strings)

• Interval: Ordered elements with fixed distance in-between (can be distinct labels from a defined vocabulary, numberic (discrete or continiuous) or strings)

• Ratio: Continiuous or zero-defined