Dear FIWARE Staff,

I would like to report a potential issue on the Cosmos Global Instance.

We are using Cosmos to store input files for Hadoop jobs and their related
output. We are transferring them from/to a remote back-end through the
WebHDFS API.

During outbound transfers of large files from Cosmos, the need to split the
file content in parts has risen. If we use a simple OPEN operation like

*http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN *

the transfer is interrupted after 2 minutes even if the response has not
transferred completely to the client. This should due to the timeout
configuration of the HTTP Server deployed on Cosmos.

As suggested by the WebHDFS API Doc
<https://hadoop.apache.org/docs/r1.0.4/webhdfs.html#OPEN>, one should
consider to transfer single parts of the requested file by setting offset
and length parameters to complete the request successfully.

The issue we found is that the length parameter seems to be ignored by
the HTTP Server.
For example, if we execute the following

http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN&offset=100&length=10

on a 200 bytes file, we would get the last 100 bytes of the file itself,
and not bytes from 101 to 110.

We checked the current version of Hadoop deployed on Cosmos and we found
out that release *0.20.2-cdh3u6 *is currently running. By giving a look to
the HDFS changelogs
<https://hadoop.apache.org/docs/r0.23.11/hadoop-project-dist/hadoop-hdfs/CHANGES.txt>,
there is an improvement (HDFS-3794) committed in release 0.23.3 that could
be potentially related to the reported occurrence.

Best Regards,
Andrea Sassi

Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost.
Please, send your messages using the new domain (Fiware-tech-help@lists.fiware.org) instead of the old one.
_______________________________________________
Fiware-tech-help mailing list
Fiware-tech-help@lists.fiware.org
https://lists.fiware.org/listinfo/fiware-tech-help
[Created via e-mail received from: Andrea Sassi <brainswitch@gmail.com>]