Azure Data Lake Store vs Azure Blob Storage containers

 

Capability	Azure Data Lake Store	Azure Blob Storage containers
Purpose	Optimized storage for big data analytics workloads	General purpose object store for a wide variety of storage scenarios
Use cases	Batch, streaming analytics, and machine learning data such as log files, IoT data, click streams, large datasets	Any type of text or binary data, such as application back end, backup data, media storage for streaming, and general purpose data
Structure	Hierarchical file system	Object store with flat namespace
Authentication	Based on Azure Active Directory Identities	Based on shared secrets Account Access Keys and Shared Access Signature Keys, and Azure role-based access control (Azure RBAC)
Authentication protocol	OAuth 2.0. Calls must contain a valid JWT (JSON web token) issued by Azure Active Directory	Hash-based message authentication code (HMAC). Calls must contain a Base64-encoded SHA-256 hash over a part of the HTTP request.
Authorization	POSIX access control lists (ACLs). ACLs based on Azure Active Directory identities can be set file and folder level.	For account-level authorization use Account Access Keys. For account, container, or blob authorization use Shared Access Signature Keys.
Auditing	Available.	Available
Encryption at rest	Transparent, server side	Transparent, server side; Client-side encryption
Developer SDKs	.NET, Java, Python, Node.js	.NET, Java, Python, Node.js, C++, Ruby
Analytics workload performance	Optimized performance for parallel analytics workloads, High Throughput and IOPS	Not optimized for analytics workloads
Size limits	No limits on account sizes, file sizes or number of files	Specific limits documented here
Geo-redundancy	Locally-redundant (LRS), globally redundant (GRS), read-access globally redundant (RA-GRS), zone-redundant (ZRS).	Locally redundant (LRS), globally redundant (GRS), read-access globally redundant (RA-GRS), zone-redundant (ZRS). See here for more information

Azure Storage: Azure Storage VS Azure Data Lake Storage (Gen-1 & Gen-II)

Azure Storage VS Azure Data Lake Storage (Gen-1 & Gen-II)

Azure Storage

· Object Storage

· Virtual folders -no real folders.

· No folder level security

· No folder specific performance optimizations

· Data is stored in Containers as blobs

· Storage cost is based on hot, cold and archive tiers

· Available in all regions globally

· Data replication and redundancy options

Azure Data Lake Storage (Gen-1)

· Hierarchical file system

o Supports nesting of files within folders.

· Folder level security can be implemented

o Fine-grained security visa access control lists

· Performance optimization at the folder level

· Parallel reads and writes

· Scaled out over multiple nodes

· Hadoop and big data optimizations

· Optimized for analytics workloads.

Azure Data Lake Storage (Gen-2)

· Multi-modal storage combining features from both Azure Storage and ADLS (Gen-1)

· Enable the Hierarchical Namespace to use for the file system.

· Data can be accessed using object storage endpoint or file system storage endpoint.

· Object Storage Endpoint – wasb://containername@accountname.blob.core.windows.net/

· File System Endpoint – abfs://filesystemname@accountname.dfs.core.windows.net/

· Optimized for analytics workloads.

Leverage partition scans & partition pruning to improve query performance.

Azure Data Analytics: How to access Azure Storage Blob data from Azure Data Lakes

1. Register Azure Storage Blob account in Azure Data Lakes

Ensure that your Windows Azure Blob Storage account is registered with your Azure Data Lake Analytics account. I have copied the steps below from Registering Your Windows Azure Blob Storage account.

1. Navigate to the Azure Portal and log in.
2. Navigate to your Azure Data Lake Analytics Account.
3. Select Data Sources under Settings.
4. Verify whether your WABS account is listed. If yes, stop here. If no, continue to next step.
5. Click Add Data Source.
6. Select Azure Storage from the Storage Type drop-down list.
7. Select Select Account from the Selection Method drop-down list.
8. Select your WABS account from the Azure Storage drop-down list.
9. Click Add