Created question in FIWARE Q/A platform on 19-01-2015 at 13:01
Please, ANSWER this question AT https://stackoverflow.com/questions/28025152/cygnus-dosent-forward-the-data-to-hdfs

Question:
Cygnus dosen't forward the data to HDFS

Description:
After configuring Cygnus in my filab vm, I'm getting this error:

[ERROR - es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink.process(OrionSink.java:140‌​)] Persistence error (The default_org/XYZ directory could not be created in HDFS. HttpFS response: 400 Bad Request)

A concrete example will this one:

(SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink.process(OrionSink.java:140)] Persistence error (The default_org/<XX:YY:ZZ:AA:BB>-<entity-type> directory could not be created in HDFS. HttpFS response: 400 Bad Request)

I am following the next manual to run cygnus:
https://github.com/telefonicaid/fiware-connectors/tree/develop/flume but the problem persist.

My cygnus.conf is the next one:

#=============================================

1. To be put in APACHE_FLUME_HOME/conf/cygnus.conf
   #
2. General configuration template explaining how to setup a sink of each of the available types (HDFS, CKAN, MySQL).

#=============================================

1. The next tree fields set the sources, sinks and channels used by Cygnus. You could use different names than the
2. ones suggested below, but in that case make sure you keep coherence in properties names along the configuration file.
3. Regarding sinks, you can use multiple types at the same time; the only requirement is to provide a channel for each
4. one of them (this example shows how to configure 3 sink types at the same time). Even, you can define more than one
5. sink of the same type and sharing the channel in order to improve the performance (this is like having
6. multi-threading).
   cygnusagent.sources = http-source
   cygnusagent.sinks = hdfs-sink
   cygnusagent.channels = hdfs-channel

#=============================================

1. source configuration
2. channel name where to write the notification events
   cygnusagent.sources.http-source.channels = hdfs-channel
3. source class, must not be changed
   cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
4. listening port the Flume source will use for receiving incoming notifications
   cygnusagent.sources.http-source.port = 5050
5. Flume handler that will parse the notifications, must not be changed
   cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
6. URL target
   cygnusagent.sources.http-source.handler.notification_target = /notify
7. Default service (service semantic depends on the persistence sink)
   cygnusagent.sources.http-source.handler.default_service = def_serv
8. Default service path (service path semantic depends on the persistence sink)
   cygnusagent.sources.http-source.handler.default_service_path = def_servpath
9. Number of channel re-injection retries before a Flume event is definitely discarded
   cygnusagent.sources.http-source.handler.events_ttl = 10
10. Management interface port (temporal location for this parameter)
    cygnusagent.sources.http-source.handler.management_port = 8081
11. Source interceptors, do not change
    cygnusagent.sources.http-source.interceptors = ts de
12. Interceptor type, do not change
    cygnusagent.sources.http-source.interceptors.ts.type = timestamp
13. Destination extractor interceptor, do not change
    cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor$Builder
14. Matching table for the destination extractor interceptor, put the right absolute path to the file if necessary
15. See the doc/design/interceptors document for more details
    cygnusagent.sources.http-source.interceptors.de.matching_table = /usr/cygnus/conf/matching_table.conf

1. ============================================
2. OrionHDFSSink configuration
3. channel name from where to read notification events
   cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
4. sink class, must not be changed
   cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
5. Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
   cygnusagent.sinks.hdfs-sink.cosmos_host = cosmos.lab.fi-ware.org
6. port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
   cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
7. default username allowed to write in HDFS
   cygnusagent.sinks.hdfs-sink.cosmos_default_username = <fiware_user>
8. default password for the default username
   cygnusagent.sinks.hdfs-sink.cosmos_default_password = <fiware_password>
9. HDFS backend type (webhdfs, httpfs or infinity)
   cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
10. how the attributes are stored, either per row either per column (row, column)
    cygnusagent.sinks.hdfs-sink.attr_persistence = column
11. Hive FQDN/IP address of the Hive server
    cygnusagent.sinks.hdfs-sink.hive_host = <fiware_user>@cosmos.lab.fi-ware.org
12. Hive port for Hive external table provisioning
    cygnusagent.sinks.hdfs-sink.hive_port = 10000

#=============================================

1. hdfs-channel configuration
2. channel type (must not be changed)
   cygnusagent.channels.hdfs-channel.type = memory
3. capacity of the channel
   cygnusagent.channels.hdfs-channel.capacity = 1000
4. amount of bytes that can be sent per transaction
   cygnusagent.channels.hdfs-channel.transactionCapacity = 100