Scientific Data Text Representation
Tika now parses scientific data files such as HDF, NetCDF, and MatLab. Currently, the parsed output for these files is metadata. While there is a 'text/ -t' output for NetCDF files, it includes variable names and dimensions, which is still in its essence, metadata. Enabling the data themselves to be parsed would enable a new paradigm in search.
NetCDF Example
A netCDF classic or 64-bit offset dataset is stored as a single file comprising two parts:
- a header, containing all the information about dimensions, attributes, and variables except for the variable data;
- a data part, comprising fixed-size data, containing the data for variables that don't have an unlimited dimension; and variable-size data, containing the data for variables that have an unlimited dimension.
The NetCDFparser currently extracts the "header part".
- -text extracts file Dimensions and Variables
- -metadata extracts Global Attributes
We want the option to extract the "data part" of NetCDF files.
JIRA Issue: TIKA-1577
Lets use the NetCDF test file for our dev testing: tika/tika-parsers/src/test/resources/test-documents/sresa1b_ncar_ccsm3_0_run1_200001.nc
Suggestions include:
Use nested tables in HTML5:
http://stackoverflow.com/questions/10297874/html5-validity-of-nested-tables