Chapter 6 Storage

by abdullah S.

Storage Account = Unique Azure Namespace

• Every object in Azure has its own web address.

Example: acloudguru.<storage-type>.core.windows.net

Blob

Blob means “Binary Large Object” Blob is a term used widely in technology for "Binary Large Object." In other words, pretty much anything made up of bits and bytes. These blobs of data are stored in containers, inside the storage account. That means there are three layers to Blob storage: storage account, container, and then blob.

Let me explain blobs further. Storage account can have multiple blob containers that you can store data in. So let's say this container here is a blob container. Doesn't matter what kind of blobs or items that are in here and what I placed in it. I could have my little birdie as a blob item. I could have my PS4 controller, or even some scissors. They will all fit, it doesn't matter what they are or what size they are.

Each of these blobs, or items, will have a unique address. So if, for example, I want to get the scissors, I can get them directly and quickly with no fuss. I know exactly where they are. So think of your blob container as a mixed bag of items that are all labeled in a sensible manner.

From the ground up, Blob storage was designed to be versatile and suit a ton of scenarios.

Blob Scenarios

Images:

Store various sizes and formats as a single image storage.

All Types:

Store any kind of files and have distributed access through the Azure cloud storage.

Streaming:

Stream audio and video directly from your blob storage.

Log Files:

Write to log files regardless of size and frequency.

Data Store:

Store any kind of data at scale, such as for archiving, backup, restore and disaster recovery

Images are often various different sizes and formats, and Blob storage makes storing and serving them to a browser straightforward. You can store any kind of files to provide distributed access, streaming video and audio directly, writing to log files as they grow at various rates. Any kind of data stores, such as backup and restore, disaster recovery, and archiving of data.

Blob Types

Block

Store text and binary data up to 4.7TB. Made up of individually managed blocks of data.

Append

Block blobs that are optimized for append operations. Works well for logging where data is constantly appended.

Page

Store files up to 8TB. Any part of the file could be accessed at any time, for example a virtual hard drive.

Azure storage supports three types of blobs. Block blobs store text and binary data up to about 4.7 terabytes. Yeah, huge blobs. Block blobs are made up of blocks of data that can be managed individually. Append blobs are made of blocks, like block blobs, but are optimized for append operations. Append blobs work really well for scenarios such as logging data from virtual machines.

Page blobs store random access files up to eight terabytes in size. These are files where any part of the file could be accessed at any time. These page blobs could store a virtual hard drive (a VHD file) and serve as a disk for a virtual machine on Azure, for example.

Blob Pricing Tiers

Hot:

Frequently accessed files. Lower access times and higher access costs

Cool:

Lower storage costs and higher access times. Data remains here for at least 30 days.

Archive:

Lowest costs and highest access times.

Blob storage comes in three main pricing tiers. Hot for frequently accessed files. This tier has lower access times and higher access costs. The Cool access tier has lower storage costs and higher access times compared to Hot storage. This tier is intended for data that will remain in the Cool tier for at least 30 days. And then the Archive tier has the lowest cost and highest access time. More on Archive storage later in this chapter.

Disk

Managed Disk

Azure Manages

You don’t have to worry about backup and uptime.

Size and Performance

Microsoft and Azure guarantees size and performance as per your agreement with them.

Upgrade

Easy to upgrade your disk size and type.

Disk storage is pretty much what it sounds like; it's a disk that you can store data on. But of course, there is more to it. A managed disk, which is also what disk storage is referred to, is what you attach to your VMs. "Managed" is the keyword here; it means that Azure will look after the physical disk for you and guarantees its uptime and backup. You don't have to worry about size or performance either; this is all part of your agreement with Microsoft for using Azure. It also means you can easily upgrade the disk size and the disk type.

Disk Types

HDD

Spinning hard drive. Low cost and suitable for backups.

Standard SSD

Standard for production. Higher reliability, scalability and lower latency over HDD.

Premium SSD

Super fast and high performance. Very low latency. Use for critical workloads.

Ultra Disk

For the most demanding, dataintensive workloads. Disks up to 64TB.

Speaking of disk type, this is important when it comes to performance, and there are four main disk types that you can choose for your VM, one more expensive than the next. Let's start with HDD, which is your old-school spinning hard drive. This is a low-cost storage option for having data that you can access infrequently. This is for scenarios like backup or for testing any scenarios you're developing.

The next step up is a Standard SSD or Solid State Drive; this is generally the norm for any production environment. Running a VM with a Standard SSD gives you lower latency than an HDD, improved reliability, and better scalability.

Then we get to Premium SSD, and premium is always better. Right? This is super-fast, high-performance storage for your critical workloads. Premium SSDs are recommended for database installations, in particular, as the latency is very low. There are lots of large numbers to measure its performance, but all you need to know is that it's better than Standard SSD.

Finally, there is Ultra Disk, which has unicorns flying out of it. Well, not quite, but it is the top tier of storage. This is for your most demanding, data-intensive workloads, delivering unprecedented and extremely scalable performance with sub-millisecond latency. Those are Microsoft's words, but they describe it. You can even have disks up to 64 terabytes in size; huge disks! Microsoft recommends this tier for transaction-heavy workloads, complex analytical modeling, gaming, rendering, and low queue depth databases. So, that's disk storage, a flexible, managed solution for your VM and other services.

File

There are many advantages of using cloud computing, and, of course, those advantages also apply to storage accounts. Let's look at how this works with file storage. Most companies will have file shares that are used to store all of the company's assets. Users will upload, delete, synchronize, and do what they need to do to work with the files. Nothing unusual about that.

However, having on-premises storage can have some issues, such as disk space constraints. You only have a limited amount of storage. Backups need to be configured and maintained. Security of data can be hard to maintain and requires specialists to enforce. Extending file sharing to other teams, organizations, and entities requires more infrastructure to both supply and manage. All these issues can be resolved by using a file storage account on Azure, due to the cloud benefits that include the ability to share access across multiple machines and seamlessly connect to your existing infrastructure.

Azure's file storage is fully managed, which means you don't have to worry about hardware or the operating system; Azure takes care of it. File storage is super resilient and always available. Network issues or power outages will no longer take out your file storage, as it can on your on-premises networks. Azure is always on, with redundancies built in.

File Benefits

Sharing:

Share access to the Azure file storage across machines and provide access to your on-premise

infrastructure.

Managed:

You don’t have to worry about hardware or operating system.

Resilient:

Network and power outages won’t affect your storage.

File Scenarios

Hybrid:

Supplement or replace your existing on-premises file storage solution.

Lift and Shift:

Move your existing file storages and related services to Azure.

Some useful scenarios for using file storage in Azure are, for example, if your on-premises file storage is running out of space, you can use Azure file storage to supplement it. You can also completely replace the on-premises file storage solution. Azure comes with redundancy and backup features as standard.

Another approach is "lift and shift," which is a buzzwordy way of saying that Microsoft would very much like you to move all your infrastructure to Azure. Because you can almost take your existing infrastructure and move it service by service, it is called lift and shift. The file storage solution is a good match for doing this with your existing file storage.

So that's file storage on Azure. It's your friend! Let's move on to the kind of storage you hide in the basement.

Archive

What do you do with data you just have to archive and want to pay very little to store? Well, you use Azure Archive Storage, of course. For a number of reasons, including company policies, legislation, and recovery scenarios, storing a large amount of archived data is often something that is required. The Azure Archive Storage service is here at your service. It is the lowest-priced storage tier within Azure, and you can store terabytes of data for just a few dollars a month.

Archive Overview

Requirement:

Policies, legislation and recovery can be requirements for archiving data. These can be very large amounts of data.

Lowest Price

The archive tier is the lowest price for storage on Azure. A few dollars a month can get you terabytes of space.

Features

Durable, encrypted and stable. Perfectly suited for data that is accessed infrequently.

Free Up premium storage

With cheap archive storage you can free up your more premium on-premises storage.

Secure

Fully secure to allow for any personal data such as financial records, medical data and more.

Blob

Archive storage is blob storage, so the same tools will work for both

It isn't very fast, nor does it perform very well, but it doesn't have to either. It is low cost, durable, and encrypted, perfectly suited for that data that used to be in a basement B in the old archive boxes in the corner. A really good reason to place archive data in the cloud, apart from low cost, is that it can free up your more valuable on-premises storage. By moving data to an archive storage, you can repurpose storage infrastructure for more important things.

The archive data is secure, as it could be sensitive data such as financial records, medical data, or other personal information that's stored. As I mentioned earlier in this chapter, Archive storage is a Blob storage tier. This means any tooling used to access blob storage will also work for archive data, making it easier to manage and integrate into existing workflows.

Storage Redundancy

Data redundancy protects against unplanned failures.

Storage Redundancy: Multiple, Replicated Copies of Data

If one copy fails/is inaccessible, data is still available. Azure Storage always creates multiple copies of your data:

Multiple Redundancy Options

• Automatic

• Minimum of three copies

• Invisible to end user

Multiple Redundancy Options

Locally Redundant Storage (LRS)

Three copies in single location (datacenter/zone)

1 Lowest-cost option

2 Protect against single disk failureDoes not protect against

3 zone or regional outage

Zone-Redundant Storage (ZRS)

1 One copy in each zone Protect against zone outage

2 but not regional outage

Geo-Redundant Storage (GRS)

Three copies in two different regions

1 Three copies in primary regional physical location (LRS)

2 Three copies in secondary (paired) region physical location (LRS)

3 Protect against primary region failure but no primary region zone redundancy

4 Can configure read access from secondary region for high availability

Geo-Zone-Redundant Storage (GZRS)

Maximum redundancy!

1 Copy across three availability zones in primary region (ZRS)

2 Three copies in secondary region physical location/zone (LRS)

3 Protect against primary region failure AND primary region zone failure

4 Can also configure read access from secondary region for high availability

Moving Data

Concept: Moving Data into and Out of Azure Storage

Different solutions based on:

• Transfer frequency (occasional/continuous)

• Data size

• Network bandwidth

AZ-900 focus for smaller, occasional transfers:

AzCopy

Azure Storage Explorer

Azure File Sync

When moving data into and out of Azure Storage, whether from an on-premises location or another cloud provider, different solutions are available based on factors such as transfer frequency, the amount of data being transferred, and the available network bandwidth. This lesson will focus on objectives for smaller occasional data transfers, exploring three different solutions: AzCopy, Azure Storage Explorer, and Azure File Sync.

Moving Data AzCopy

Command-Line Utility

Transfer blobs and Azure Files
Useful for scripting data transfers

AzCopy is a command-line utility designed for occasional data transfers. It is a downloaded application that integrates with your command-line environment, capable of transferring both blob and Azure Files storage types. AzCopy is particularly useful for automated scripting routines, compatible with Bash and PowerShell scripts. An example command involves copying a file (e.g., "funny-llamas.mp4") to a storage account named "mystorageaccount," specifically into a blob container called "my-container."

Moving Data Storage Explorer

Graphical User Interface (GUI) Interaction

Downloaded application
User-friendly graphical interface
1. Drag-and-drop interaction
Move all storage account formats

Next is Storage Explorer, a graphical user interface (GUI) interaction method for storage transfers. Unlike AzCopy, Storage Explorer is a downloaded application with a more user-friendly graphical interface, allowing users to drag and drop different data types into and out of Azure Storage. It supports all storage account formats, not just Blob and Azure Files formats.

Moving Data Azure File Sync

Synchronize Azure Files with On-Premises File Servers

Local file server performance + cloud availability

Use cases:

• Back up local file server

• Synchronize files between multiple on-premises locations

• Remote users access Azure Files

• Transition to only Azure Files for file server

The third method discussed is Azure File Sync, specifically working with Azure Files storage format. Azure File Sync automatically synchronizes the Azure File Service with on-premises file servers. This is ideal for scenarios where on-premises locations have local file servers, but maintaining cloud availability of Azure files is necessary. Azure File Sync keeps both locations synchronized, offering local file server performance combined with the cloud availability of the Azure File Service.

Common use cases for Azure File Sync include disaster recovery scenarios, synchronization between multiple on-premises locations, and transitioning to a cloud-only implementation. It serves as a step to automatically synchronize both an on-premises file server location and Azure Files, easing the transition to a cloud-only Azure Files implementation.

With this, we conclude our exploration of three use-case scenarios for occasional data transfers. The next lesson will focus on two other data transfer scenarios of a larger scale. See you then!

Additional Migration Options

Two More Migration Solutions

Azure Data Box
Azure Migrate

Azure Data Box

Scenario: Transfer LOTS of data and/or limited bandwidth

Lots = Too much to transfer over the internet
Relative to available network bandwidth

Offline data transfer to/from Azure

Copy data to physical data storage device (Data Box)

• Encrypted

• Rugged

Ship Data Box to/from Azure

• To Azure: Data Box data transferred to storage account

• From Azure: Data Box delivered to on-premises location for on-site transfer

Data Box Use Cases

Initial bulk data migration

Disaster recovery

• Restore Azure backup to on-premises location

Security requirements

• Sensitive data that cannot be sent over the internet

The ideal scenario for using the Data Box solution is when you have a significant amount of data to transfer into or out of Azure Storage, and you may have limited network bandwidth available. In this case, "a lot" is a relative term, usually meaning too much data to transfer over the internet efficiently. Azure Data Box provides an offline data transfer method, acting as a physical box that is sent to your location. It is an encrypted and rugged hard drive, and for transferring data into Azure, you would load your data onto the Azure Data Box appliance, ship it to Azure, and they will transfer the data from your Data Box into a precreated storage account. This process also works in reverse, allowing you to take a large amount of data already in a storage account, copy it to a Data Box, and have it shipped to your on-premises location for an on-site transfer.

Typical use cases for Azure Data Box include initial bulk data migration, disaster recovery scenarios where large amounts of data need to be restored from Azure-side backups, and scenarios with security requirements where data is deemed too sensitive to transfer over the network.

Azure Migrate

Migrate non-Azure resources into Azure

• Servers

• Databases

• Applications

Includes, but not limited to, migration to storage accounts

Azure Migrate Scenario: Migrate Datacenter to Azure

Moving on, our final data migration scenario is the Azure Migrate service. Azure Migrate lets you plan out and implement various migration scenarios for moving non-Azure resources into Azure itself, such as servers, databases, and different applications. Azure Migrate focuses on a variety of migration scenarios, not limited to working with storage accounts.

One typical Azure Migrate scenario is migrating an on-premises data center into the Azure service. This suite of tools performs actions like discovering dependent resources to migrate, assisting in migrating virtual machines, helping migrate and convert on-premises databases into Azure-managed database services, and facilitating bulk data transfers with the Data Box service as part of a broader Azure Migrate implementation plan.

At this point, we have covered 2 more scenarios for transferring a significant amount of data and different data types, which can greatly ease the process of a full-scale cloud migration. Let's go ahead and move on.

Premium Performance Options

Premium Performance Options for Low-Latency Requirements

Stored on SSDs: