2018 HDFS Bug Bash: Build/Test Improvement Edition

Scratch space to brainstorm ideas for improving HDFS test stability

Unit tests

• Update tests using build/test/data, /tmp, etc. to use correct temp
• Flaky tests
  • TestDFSStripedOutputStreamWithFailure ( HDFS-11600 - Getting issue details... STATUS )
  • TestWebHdfsTimeouts
  • TestErasureCodingPoliciesWithRandomECPolicy
  • TestErasureCodingPolicies
  • TestDFSStripedOutputStream
  • TestDFSStripedInputStream
  • TestDecommission
  • TestUnsetAndChangeDirectoryEcPolicy
  • TestMaintenanceState
  • TestDataNodeErasureCodingMetrics
  • TestReconstructStripedFile
  • TestReadStripedFileWithDNFailure
• MiniDFSCluster improvements
  • Reduce the number of threads that get started by MiniDFSCluster
    • HDFS-13265 - Getting issue details... STATUS
      
    • HDFS-13262 - Getting issue details... STATUS
      
• Over-reliance on MiniDFSCluster
• Ignored or skipped tests
  •  TestGlobPaths

  • TestHdfsNativeCodeLoader

  • TestSymlinkHdfsFileSystem

  • TestAppendDifferentChecksum

  • TestDecommission

  • TestDFSUpgrade

  • TestFileConcurrentReader

  • TestParallelReadUtil

  • TestWebHDFSAcl

Branch-2 nightly DNF https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/

Build

• HDFS-12711 - Getting issue details... STATUS
• HADOOP-14696 - Getting issue details... STATUS ,  HDFS-12220 - Getting issue details... STATUS (Windows parallel builds)
• Windows build (stopped running?)
• Upgrade surefire, unless there's a good reason not to in branch-2
  • HADOOP-15251 - Getting issue details... STATUS
  • HADOOP-15153 - Getting issue details... STATUS
• branch-2 + JDK7 qbt build fails due to hanging hdfs tests (see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/158/console)
• JDK9: qbt build weekly to run hdfs unit tests on JDK9 (now compilation fails due to the two issues)
  • HADOOP-12760 - Getting issue details... STATUS
  • HDFS-11610 - Getting issue details... STATUS