spark

Replication involves copying data across multiple connected machines to reduce latency, increase availability (fault tolerance), and improve throughput.

One leader replica handles all writes, and changes are streamed to follower replicas, which handle read-only queries.

• Cluster-computing framework

• Implicit parallel processing (handles node failures)

• In-memory data processing (avoids frequent read/write operations)

• Lazy loading of data

• Supports computations on Resilient Distributed Datasets (RDDs)

• Open-source (Apache project)

• Main entry point for Spark functionality

• Represents the connection to a Spark cluster

• Used to create RDDs, accumulators, and broadcast variables

• Sends tasks to the executors in the cluster

• Creates RDDs which form the basis for task definitions

• Defines a dataset and the operations on it (where to get data, what to do)

• Does not contain the actual data

• Lazy loading: data is only loaded when computation starts

• Distributed: operations can be parallelized across cluster nodes

• Resilient: automatically handles node failures

• Custom RDDs can be implemented

• Distributes data across multiple nodes

• Breaks data into partitions (shards), each a small database

• Enables scalability

• Limitation: may result in duplicate data copies across nodes (especially with large datasets and high throughput needs)

• Goal: Spread data evenly across nodes

• Issue: Uneven distribution (skew) can create hot spots

• Naive approach: Random distribution, lacks location awareness

• Solution: Use key-value data to better control placement

• Assigns a continuous range of keys to each partition

• Distribution may be uneven; boundaries can be adapted to data

• Keys within each partition are kept sorted

• Cluster Manager: Assigns resources (executors) to each Spark application (SparkContext)

• SparkContext: Interfaces with the cluster manager and coordinates execution

• Tasks: Each calculation step is divided into smaller tasks for data partitions

• Worker Nodes: Execute tasks using executors, which may cache data for efficiency