Given the following scenario:
- FileManager is running on machineA.
- Workflow/PGE tasks should run on machineB.
There are 2 strategies to operate on remote files:
- Use NFS to simulate a local filesystem on machineB
- Use fmprod to download the product to machineB from a web URL
When your Workflow/PGE task is done, you can ingest the output files into the remote FileManager using org.apache.oodt.cas.filemgr.datatransfer.RemoteDataTransferFactory as the clientTransferer.
The idea here is to use NFS to mount the data archive onto a common root file path. Then all file paths will appear to be on the local file system.
Edit /etc/exports to allow machineB to see the data archive on machineA (for more information, type "man exports")
Restart the NFS service on machineA
Mount the remote file system on machineB (for more information, type "man mount")
Create a symbolic link on machineA
Edit product-types.xml to use the symbolic link
- The FileManager should now return file paths that are reachable by machineB.
If your products were already previously ingested, you must update the
[FileLocation] metadata element for your ingested products. There are several ways to update the catalog with the new NFS file location:
- Re-ingest, making sure the repository path in product-types.xml uses the NFS mount.
- Re-ingest, using a different Versioner, which gives the NFS mount as final file location.
MetadataBasedProductMover. Run the following locally on machineA:
The idea here is to deploy a Product Server to allow anyone to download products with an HTTP request.
Build fmprod (assumes you have OODT sources checked out)
- Deploy fmprod (cas-product-VERSION.war)
- Download the product with an HTTP request (GET or POST)
To get a file product, use the "productID" query parameter.
To get a hierarchical product: use additional 'refIndex' parameter or format parameter. Note 'refIndex=0' is the directory name.
To get a dataset (products of a certain type) as a zip file, use the "typeID" query parameter.
The filename is sent to to requesting client in the server header. It's possible to force wget and curl to use this information to name the file. To download a product and to figure out the product name from the server header information use one of the following commands:
- The previous download step can be wrapped in a "FileStager" task. Subsequent tasks can now operate on the downloaded local file.
The disadvantage of this approach is that products and metadata must be retrieved separately, since products are downloaded via the fmprod web service and metadata are obtained via direct connection to FileManager. Also, you cannot use PGE sql-like queries to obtain products.