05. Cloud Monitoring

• make best use of rented resources

• -> to reduce cost

• -> and increase satisfaction of users of your services

• exposes enough data about itself

• so that generating inforamtion (finding answers to questions yet to be formulated)

• and easily accessing this informaiton

• becomes simple

• monitoring in the cloud is:

• the process of collecting

  • status information of

  • applications and resources

• data can be sued to observe application and infrastructure

• consists of all components for gathering monitoring data at runtime

• all (raw) data captured by the monitoring system

• infoirmation is gained by

  • processing

  • interpreting

  • organizing

  • visualizing

• raw data

• -> it increases knowledge abotu the observed system

example:

• raw data are CPU and memory utilization

• informatoin is that there is a trand for an overload or memory leak

• No!

• -> collect any data available

• proactively creating

  • -> continuous analysis to trigger alkarms of give overview over the status of the system

• reactively creating

  • triggered through events such as incidents

• infrastrucure level

• application level

• resource management

• incident detection

• root cause analysis

• accounting or metering for payment

• intrusoin detection

• auditing

• performance analysis

• resource management (e.g. scaling decisions)

• failure detection and resolution

• SLA verification

• auditing

• parelell systems

• cloud

• batch system

• data is collectd during an application run

• analysis happens post mortem

• execution is reproducable

• interactive system

• data is continuously produced - realtime data

• realtime analysis

• data used for

  • immediate action

  • study past system behavior

• metrics

• logs

• traces

• monitoring metrics

• metrics we monitor

• metric itself (e.g. execution time)

  • semantics

  • unit

• context

  • server, application service,…

• representation

• aggregation

  • sum, min, max, mean, percentiles, histogram

• measurement frequency

  • every second, minute, 5 minutes,…

• latency

• throughput or traffic

• error rate

• utilization or saturation

  
  

• time it takes to service a request

• -> selectively measure successful and error request

• web service: requests/second

• streaming system: network I/O rate or concurrent sessions

• database: transactions/second or retrievals per second

• rate of requests that fail

• e.g.

  • explicitly (HTTP 500)

  • implicityl (wrong reply contents)

  • or by violating an SLA

• percentage of capacity

• CPU, memory, I/O

• client

• applicatoin

• platform

• infrastructure

• hardware

Monitor:

• requests

Context:

• request type

Metric:

• # requests

• latency

• availablity

purpose:

• SLA checking

• alerting

Monitor:

• microservices

Context:

• service name

• service id

Metric:

• # requests

• request rate

• latency

• #replicas

• CPU time

• memory usage

purpose:

• autoscaling

• performance tuning

Monitor:

• kubernetes

• docker

Context:

• container id

Metric:

• CPU & memory quota

• utilization

• incoming & outgoing bytes

purpose:

• container distribution

• autoscaling VM cluster

Monitor:

• VM

• volumes

• queuing services

Context:

• VM id

• volume id

• service name

Metric:

• CPU & memory

• #read/write

• I/O latency

• # requests

• size of requests of infrastructure service

• disk utilization

• traffic

purpose:

• root cause analysis

  
  

Monitor:

• servers

• network

• SAN

• disks

Context:

• server id

• switch id

Metric:

• disk utilization

• traffic

purpose:

• management of VMs

• comprehensive

• low intrusion

• extensibility

• scalability

• elasticity

• accuracy

• resilience

• white box

• black box

• data is from in and outside of the system

• -> gives more context and more detailed insights

• e.g. internal organization of a service is visible e.g. asynchronous internal handling of requests

• monitored system handled as black box

• no data gained from the inside of a system

• e.g. only the request interface of service is visible

  • => nothing about the internal structure

• lead to intrusion

• instrumentation

• computation for aggregations

• memory overhead for buffering

• time to push to disk or transfer to collector

• storage overhead for long-term storage

• number of metrics

• measurement frequency

• representation

• batching

• sampling

• long-term coarsening

• monitoring and management service

• collects:

  • metrics

  • logs: clouod watch log insights

• online

• -> different frequency depending on data and accout

• -> different storage time for different granularity (lower garnularity -> shorter storage)

• management console web interface

• CLI

• libraries (e.g. java, script languages, windows .net)

• web service API

• vie graphs and statistics

• set alarms

• open source monitoring system

• features:

  • metric collection in form of time seris

  • storage by a time-series database

  • oquery language for accessing the time-series

  • alerting

  • visualization

• sequence of immutable records of discrete events

• generated by applicatoins, system eve, infrastrucure, any devices…

• plaintext -> most comman format of logs

• structured -> much evangelized, typically json

• ASCII

  • easily readable

  • inefficient w.r.t. space and time

• binary

  • more efficient

  • e.g. protobuf from google

• huge!

• -> can be configured to levels

• allows to drill down

• -> difficult to analyze!"

• bunary format for logs

• -> language neutral

• -> platform neutral

• -> extensible mechanism for serializing structured data

• backward compatible

• forward compatible

• stack of FOSS tools

• elasticsearch, logstash, kibana

• suited for logs and metrics

• ELK + beats and X-pack

• parsing from predefined patterns (grok patterns), transforms and filters

• derive structure

• anonymize personal data

• geo-location lookups

• extension to ELK

  • authenticaoin and authorizatoin

  • monitor ELK

  • alerting

  • report generation of kibana contents

  • machine learning, e.g. anomaly detection and forecasting

  • SQL interface for elastic search

• visualization dashboard

• beside from debugging and performance monitoring:

• properly structured logs help to:

  • incident root cause analysis

  • anomaly detection

  • fault prediction and predictive maintenance

  • detect and respond to data breaches and other security incidents

  • ensure compliance with serucity policies, regulations & audits

• pattern detection and recognition

• log normalization

• classificatoin and tagging

• correlation analysis

• artificial ignorance

• agents to collect data

  • -> filebeat for logs

  • -> metricbeat for metrics

• allows to store and analyze logs from amazon services

• and your applicatoin services

• grouping logs

• metrics

  • -> extract from log statements automatically and insert into cloud watch metircs

• control retention period

  • by default never deleted

• analysis

• real-time processing

• allow to cluster several log streams from multiple sources

• capture interactoins of different services, the life of a request

• captuer individual events -> e.g. subint request, receive request, start processing, …, submit answer, receive answer

• associate events with given request to be able to analyze the execution of this request

• tool for distributed tracing

• goals:

  • continuous and ubiquitous tracing

  • low overhead

  • application transparency

  • scalability

• with tree structures

-> can be transformed in a diagram over time

• span has its own id (i.e. id for the sub-request)

• each sub-resuest handling has also a parent id (which refers to the span id of the procedure that called…)

• the lifetime of a request

• -> i.e. the time it takes from receiving the request until the frontend answers…

• to annotate events

• -> unique trace id cerated at frontend service

• -> passed to sub-requests (so that one can identify to which initial request the execution of each sub-request belongs…)

• -> requires manual or automatic instrumentation (adjusting the code so that this passing happens…)

• dapper trace trees

• -> nodes are called spans: lifetime of a request

• -> edges indicate the temporal relationship

• represent a RPC (remote procedure call)

  
  

• span id

  • identifies a span

• parent id:

  • span id of triggering span

• trace id:

  • identifies triggering request

• no

• root span: frontend request (span that receives the user resquest)

• annotaions

• applicatoin level events

• above: regular span attributes (i.e. name, trace id, parent id, span id)

• annotatoins include points in time of the span indicating what happens at the client and server side (w.r..t the RPC)

• be aware of clock skews

• -> as events are created on different systems!

• in the thread-local storage of the thread executing the span

• issue RPC but no direct execution -> asynchronous…

• => have callback function that stores the trace context

• => when callback is invoked: trace context copied to executing thread

• span and trace id are automatically transmitted

• all applicatoins use same control flow and RPC library

• instrumentation thus automatic…

• added by the appliocation owner

• e.g. proper use of authenticatoin or encryption

• or policy based isolation

• => verifiable by looking at what is executed…