Page tree
Skip to end of metadata
Go to start of metadata


This document tracks on-going efforts to upgrade from Hadoop 2.x to Hadoop 3.x  - Refer Umbrella Jira HADOOP-15501 for current status on this.

Upgrade Tests for HDFS/YARN

The following scenarios were tested while upgrading from Hadoop 2.8.4 to Hadoop 3.1.0


TypeComponentScenarioIssues FoundResolutionOverall Status
EXPRESS/ROLLING UPGRADEHDFSStarting 3.1.0 NameNode/DataNode with custom MetricsPlugin configured in hadoop2-metrics.properties

HADOOP-15502 - Getting issue details... STATUS

Workaround is applicable only for EXPRESS UPGRADE -

Replace MetricsPlugin implementation jars( eg: HadoopTimelineMetricsSink) with recompiled jars which use package "org.apache.commons.configuration2"

(tick)
EXPRESS UPGRADEYARNStarting Hadoop 3.1.0 YARN daemons

(tick)
ROLLING UPGRADEHDFS3.1.0 NN is started with rollingUpgrade with default policy configured for Erasure coding

HDFS-13596 - Getting issue details... STATUS

Workaround Not known(error)
ROLLING UPGRADEYARNStart 3.1.0 NM in batches after starting RM.

YARN-8346 - Getting issue details... STATUS

Fixed(tick)

EXPRESS/ROLLING

UPGRADE

YARNRM started with recovery enabled

YARN-8068 - Getting issue details... STATUS

Fixed(tick)


Workloads


Application TypeUpgrade TypeIssues FoundStatusOverall Status
MREXPRESS/ROLLING UPGRADE

YARN-8346 - Getting issue details... STATUS

Fixed(tick)
HIVE on TEZ
Hive with older versions of Tez (0.7, 0.8.x) with Hadoop 2 client ran into UT failures

Tez 0.10.0 will support Hadoop 3

  • TEZ-3923 Move master to Hadoop 3+ and create separate 0.9.x line

  • TEZ-3252 - [Umbrella] Enable support for Hadoop-3.x



(tick)

Spark 2.2/2.3

Spark 2.2/2.3 has a fork of older version of Hive (1.2) which does not work with Hadoop 3


Ongoing efforts in community to build/validate Spark with Hadoop 3 Libraries

    • SPARK-23534  Umbrella jira to Build/test with Hadoop 3

    • SPARK-23710  Upgrade to Hive 2.x ( which builds with Hadoop 3)







IN-PROGRESS
PIG


Support for Hadoop 3 In-Progress in the community - targeted for PIG 0.18.0

PIG-5253    Pig Hadoop 3 support


IN-PROGRESS
OOZIE
Dependent on PIG support for Hadoop 3

Support for Hadoop 3 In-Progress in the community - Targeted for OOZIE-5.1.0

OOZIE-2973  Make sure Oozie works with Hadoop 3


IN-PROGRESS

MR with Native Task Optimization

Validation Pending
MR with Shared Cache Manager

Validation Pending
  • No labels