The Hadoop distribution that is running on Azure is developed and maintained by Hortonworks Data Platform.
New version releasing
It seems that the new versions of Hadoop reach Azure faster and faster. If in the past it use to take 4-6 months for a new version of Hadoop to reach Azure and HDInsight, now it is taking only a few weeks.
Hadoop Distributed File System(HDFS) exist also in HDInsight giving us the possibility to store data in a distributed file system.
The same programming model that we already know from Hadoop can be found into HDInsight. In general you should know that all the features that exist in Hadoop can be found in HDInsight, having the same properties and functionalities.
Can be used to monitor, manage and provision the cluster that is used for processing. The API of Ambari hide all the complexity that is behind Hadoop and let us to focus on the final result and less on the infrastructure.
The NoSQL Solution develop by Hadoop and used to store data in a semi-structured way. We can store billions of rows in this data storage. Behind HBase, Azure implementation is using Azure Storage.
Developed over MapReduce, it allows us to query using SQL like the Hadoop storage. Can be used with success when the stored data that is queried in a more structured.
Similar with Hive, but is recommended to use when data is not structured and we need to execute complex MapReduce operations over large data sets.
It allow us to define a machine learning that run over Hadoop. Based on past evens and data sets we can determine what will be the future behavior of different components.
On the core components of Hadoop that allow us to execute operations over large data sets in a distributed way (on multiple nodes – parallel). The data is organized in a (key, value) for later processing.
The next generation of MapReduce (MRv2) that will allow us to slit the jobs in two different entities:
- Scheduling and Monitor
- Resource Management
It is the framework used to coordinate the workflow that is behind this great system. Can be used for different tasks like shell script or scheduling.
Real time computing system for processing streams of data.
Contains small amount of metadata (like location, configuration, status) and is used to coordinate large distributed system using a simple hierarchy of data regions.
It used to import/export data from Hadoop to relational database like SQL.
Out of the box clusters
Hadoop clusters can be provision in minutes, without having to configure and manage them
Integration with Azure Services
HDInsight is integrated with different Azure Services like SQL Database or Websites. In this way we can implement different scenarios more easily.
There is full support for Excel using Power Query to access and view data from Hadoop.
We have the ability to specify the size of the Hadoop cluster that we want to use. Based on our needs we can set a custom cluster size.
Users have the ability to specify what kind of machines to use in their HDInsight cluster.
Low storage cost
Because Azure Storage Blobs are used to store Hadoop data, the cost of storage is low. In this way we can store large amount of data without thinking about costs.
Storing data in blobs remove the problems that can appear when we need to scale out the HDFS.
Migrate Hadoop from on-premises to HDInsight
With integration of HDP (Hortonworks Data Platform) we can migrate data from one storage to another automatically. Also using HDP we are allowed to query HDInsight and on-premises instance in the same time.
In this moment we don’t have the ability to stop a cluster. The current state of stop is equivalent with delete. Content from blobs is not lost.
Elastic size of cluster
In this moment we cannot dynamically change the size of a cluster. In this moment you need to tear down and rebuild the clusters if you need to change the size of it.
Applicable Use Cases
Below you can find some use cases when I would use HDInsight.
Analyze information from Smart Home solution
Data that are produced by a Smart Home solution can be very valuable, but the volume of data is very high. For this scenario HDInsight can be used with success to analyze the data and extract valuable information from it.
eCommerce application logs
An eCommerce application logs is like a gold mine. Based on this information you can know what the system bottleneck are, what products are more attractive for clients and many more. To analyze of all this data you can use with success HDInsight.
Image analyzing is a job that consume not only a lot of CPU power, but also a lot of storage. Because of this an image analyzing system can be constructed over HDInsight.
A sample that a realy enjoyed and is very usefull can be found here: http://azure.microsoft.com/en-us/documentation/articles/hdinsight-sample-pi-estimator/
Pros and Cons
- Easy to integrate
- Out of the box solution
- Integration with a lot of external libraries and systems
- Start/Stop features is not yet supported
- Elastic-scale is not yet supported
If you are taking into account HDInsight, you should keep an eye on:
- Cluster Size
- Instances type
- Outbound traffic
- Storage size