Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.



Ozone documentation is missing information about HDFS site dependencies

  • Saurabh pointed out that for Ozone to effectively use tmp, the temp directory needs to be setup in HDFS site xml file. This information is missing in Ozone documentation.
2Data node missing UISaurabh

The Data node UI is missing – so unlike HDFS it is hard to understand what is happening.

We discussed the Recon and the UI. We discussed the Namenode Analytics and how the design of Recon is based on that.

3Recon and SecuritySaurabhSaurabh pointed out that he is not able to run NNA in financial institution due to legal and compliance requirements. He said that if Recon as the same level of Kerberos Support as OzoneManager and SCM, this should not be an issue.
4Running Hive TPC-DS tests with OzoneSaurabh

Saurabh was trying to run Hive based TPC-DS tests using Ozone. Cloudera has done this internally and (Pifta, not on call, shared this info with me). This needs some fixes in the current master branch. We also discussed the internal results that we have observed while testing Hive TPC-DS with Ozone.

From Pifta:

  • Critical issues we have ran into with TPC-DS runs were:
    • HDDS-2359 fixed is just in master targeted to 0.5
    • HDDS-2347 fixed is just in master targeted to 0.5
    • HDDS-2203 which was fixed in HDDS-2233 fixed is just in master targeted to 0.5
5Ozone ClusterSaurabhHe also mentioned that he has been running Ozone latest(0.4.1) for over 22 days without any issues.
6Apache ReleaseMartonMarton had proposed that we should do a follow up Apache release as soon as we do a Cloudera based release. Since there will be lot of testing cycles spend on Ozone during that time. Since TPC-DS runs need these fixes in the Master, we have decided to start a Discuss thread on Ozone mailing list for the 0.4.2-Alpha(Bryce) release. This will also be useful from for anyone who is trying to run Ozone.
7Basic Ozone File System FeaturesSaurabh/Xiaoyu/MartonHe mentioned Saurabh mentioned that his company has not plans to upgrade to Hadoop 3.0 any time soon. The Ozone File system proper has dependencies on Hadoop 3.x. We also have a "basic" version of Ozone File system that can be used with Hadoop 2.7x. He  Saurabh wanted to get security and TDE support in this basic version of the Ozone FS driver. @Xiaoyu Yao, Seemed to think this can be done with Basic version of File system.
8HBase Support in OzoneKarthik

What is the status of running HBase on Ozone? or in other words, are we ready to go beyond Hive and Spark in our testing?

Currently, we have very minimal support for HFlush. Some of the Hbase committers also expressed the interest in running HBase over Ozone. The plan of record is to finish the HFlush support and get Ozone working on it. Anu Engineer To follow up with Karthik and Josh Elser to see how we can make progress on this work item.

9Du and other Metadata operations against OzoneKarthikThe question was, what happens if we run a "Du" like operation against Ozone. Does it slow down the Ozone Manager?, the answer was that all such operations like Du and other management queries will be send to Recon. Recon looks and acts like SCM and OM and we will have near-real time answers for all the management queries. Since these requests are handled by an independent server, Ozone Manager should not be impacted by any query run by the admins of the system.
10Tour of SonarMarton/Attilla/Vivek/Anu

Vivek, Marton and Attilla has enabled the SonarCloud support for Ozone. We have also fixed quite a lot of bugs in the system. Anu demoed the Sonar Cloud UI and explain explained how to use it spot bugs and fix things. The code coverage is 40% since we have enabled code coverage collection only on Unit tests. That is, integration and robot based tests are not used in code coverage numbers yet.

Marton suggested that perhaps this is a good opportunity for us to write more unit tests. So we have proper coverage just by running unit tests. We often commit patches when unit tests pass, but if we have only 40% code coverage with unit tests, it is certainly a great idea to improve the code coverage.