logo
logo
Sign in

New Features of Hadoop 3.0

avatar
lavanya s

https://prwatech.in/hadoop-admin-training-institute-bangalore/

Supports Java 8.

Supports HDFS Erasure Code.

Supports more than 2 NameNodes:- This architecture is able to tolerate the failure of any one node in the system.

Support for Microsoft Azure Data Lake:- Hadoop now supports integration with Microsoft Azure Data Lake.

Scalability:- YARN Timeline Service v.2 chooses Apache HBase as the primary backing storage.

Shaded client jars:- New Hadoop-client-apiand Hadoop-client-runtime artifacts that shade Hadoop’s dependencies into a single jar. This avoids leaking Hadoop’s dependencies onto the application’s classpath.

MapReduce task:- level native optimization:- MapReduce has added support for a native implementation of the map output collector. For shuffle-intensive jobs, this can lead to a performance improvement of 30% or more.

Support for Erasure Coding in HDFS:- Considering the rapid growth trends in data and datacentre hardware, support for erasure coding in Hadoop 3.0 is an important feature in years to come. Erasure Coding is a 50 years old technique that lets any random piece of data be recovered based on other pieces of data i.e. metadata stored around it. Erasure Coding is more like an advanced RAID technique that recovers data automatically when a hard disk fails.

Support for Opportunistic Containers and Distributed Scheduling:- A notion of ExecutionType has been introduced, whereby Applications can now request for containers with an execution type of Opportunistic. Containers of this type can be dispatched for execution at an NM even if there are no resources available at the moment of scheduling.

Opportunistic containers are by default allocated by the central RM, but support has also been added to allow opportunistic containers to be allocated by a distributed scheduler which is implemented as an AMRMProtocol interceptor.

 

collect
0
avatar
lavanya s
guide
Zupyak is the world’s largest content marketing community, with over 400 000 members and 3 million articles. Explore and get your content discovered.
Read more