Created question in FIWARE Q/A platform on 29-06-2015 at 13:06
Please, ANSWER this question AT https://stackoverflow.com/questions/31114205/persisting-data-cosmos-not-working

Question:
Persisting data cosmos not working

Description:
I'm trying to persist the information received in Orion in the public instance of Cosmos, but I get an http 500 error. This is the contextBroker log:

function=sendHttpSocket | comp=Orion | msg=clientSocketHttp.cpp[154]: Starting transaction to localhost:5060/notify
function=sendHttpSocket | comp=Orion | msg=clientSocketHttp.cpp[364]: Notification Successfully Sent to localhost:5060/notify
function=sendHttpSocket | comp=Orion | msg=clientSocketHttp.cpp[375]: Transaction ended
function=connectionTreat | comp=Orion | msg=rest.cpp[767]: Starting transaction from 192.168.201.95:49160/v1/updateContext
function=processContextElement | comp=Orion | msg=MongoCommonUpdate.cpp[1647]: Database Operation Successful (

{ _id.id: "613cb7464a7950ba", _id.type: "UMANICK_TECHNOLOGIES", _id.servicePath: /^\/$|^\/\/.*/ }

)
function=addTriggeredSubscriptions | comp=Orion | msg=MongoCommonUpdate.cpp[922]: Database Operation Successful ({ $or: [ { entities.id: "613cb7464a7950ba", $or: [

{ entities.type: "UMANICK_TECHNOLOGIES" }

, { entities.type:

{ $exists: false }

} ], entities.isPattern: "false", conditions.type: "ONCHANGE", conditions.value: "X_VALUE", expiration:

{ $gt: 1435574837 }

, servicePath:

{ $in: [ /^$|^/#$|^/$/, null ] }

}, { entities.isPattern: "true", conditions.type: "ONCHANGE", conditions.value: "X_VALUE", expiration:

{ $gt: 1435574837 }

, servicePath:

{ $in: [ /^$|^/#$|^/$/, null ] }

, $where: function(){for (var i=0; i < this.entities.length; i++)

{if (this.enti... } ] })
function=addTriggeredSubscriptions | comp=Orion | msg=MongoCommonUpdate.cpp[922]: Database Operation Successful ({ $or: [ { entities.id: "613cb7464a7950ba", $or: [ { entities.type: "UMANICK_TECHNOLOGIES" }, { entities.type: { $exists: false } } ], entities.isPattern: "false", conditions.type: "ONCHANGE", conditions.value: "Z_VALUE", expiration: { $gt: 1435574837 }, servicePath: { $in: [ /^$|^/#$|^/$/, null ] } }, { entities.isPattern: "true", conditions.type: "ONCHANGE", conditions.value: "Z_VALUE", expiration: { $gt: 1435574837 }, servicePath: { $in: [ /^$|^/#$|^/$/, null ] }, $where: function(){for (var i=0; i < this.entities.length; i++) {if (this.enti... }

] })
function=processContextElement | comp=Orion | msg=MongoCommonUpdate.cpp[1834]: Database Operation Successful (update

{ _id.id: "613cb7464a7950ba", _id.type: "UMANICK_TECHNOLOGIES", _id.servicePath: /\// }

)
function=entitiesQuery | comp=Orion | msg=MongoGlobal.cpp[1090]: Database Operation Successful ({ query: { $or: [

{ _id.id: "613cb7464a7950ba", _id.type: "UMANICK_TECHNOLOGIES" }

], _id.servicePath:

{ $in: [ null, /^/$/ ] }

, attrs.name:

{ $in: [ "X_VALUE", "Y_VALUE", "Z_VALUE" ] }

}, orderby:

{ creDate: 1 }

})
function=processSubscriptions | comp=Orion | msg=MongoCommonUpdate.cpp[1046]: Database Operation Successful (update: { $set:

{ lastNotification: 1435574837 }

, $inc:

{ count: 1 }

}, query:

{ _id: ObjectId('55912126a85d63ed38229b6d') }

)
function=requestCompleted | comp=Orion | msg=rest.cpp[382]: Transaction ended

The subscription Orion is working properly, the Cygnus agent receives the notification, but when try to persist data in Cosmos gives an http 500 error.

[SinkRunner-PollingRunner-DefaultSinkProcessor] (es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.persist:212) - Persisting data. File: 613cb7464a7950ba-UMANICK_TECHNOLOGIES-X_VALUE-float.txt, Data: 2015-06-29T12:50:18.763|1435575018|613cb7464a7950ba|UMANICK_TECHNOLOGIES|X_VALUE|float|0.30645782)
[SinkRunner-PollingRunner-DefaultSinkProcessor] (es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.exists:158) - HttpFS operation: GET http://130.206.80.46:14000/webhdfs/v1/user/ilko.garcia/mydataset/613cb7464a7950ba-UMANICK_TECHNOLOGIES-X_VALUE-float.txt?op=getfilestatus&amp;user.name=ilko.garcia HTTP/1.1
[SinkRunner-PollingRunner-DefaultSinkProcessor] (es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.exists:161) - HttpFS response: HTTP/1.1 200 OK
[SinkRunner-PollingRunner-DefaultSinkProcessor] (es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.append:123) - HttpFS operation: POST http://130.206.80.46:14000/webhdfs/v1/user/ilko.garcia/mydataset/613cb7464a7950ba-UMANICK_TECHNOLOGIES-X_VALUE-float.txt?op=append&amp;user.name=ilko.garcia HTTP/1.1
[SinkRunner-PollingRunner-DefaultSinkProcessor] (es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.append:126) - HttpFS response: HTTP/1.1 307 Temporary Redirect
[SinkRunner-PollingRunner-DefaultSinkProcessor] (es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.append:138) - HttpFS operation: POST http://130.206.80.46:14000/webhdfs/v1/user/ilko.garcia/mydataset/613cb7464a7950ba-UMANICK_TECHNOLOGIES-X_VALUE-float.txt?op=append&amp;user.name=ilko.garcia&amp;data=true HTTP/1.1
[SinkRunner-PollingRunner-DefaultSinkProcessor] (es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.append:141) - HttpFS response: HTTP/1.1 500 Internal Server Error

My Cygnus agent configuration file is as follows:

1. APACHE_FLUME_HOME/conf/cygnus.conf
   orionagent.sources = http-source
   orionagent.sinks = hdfs-sink
   orionagent.channels = notifications

1. Flume source, must not be changed
   orionagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
2. channel name where to write the notification events
   orionagent.sources.http-source.channels = notifications
3. listening port the Flume source will use for receiving incoming notifications
   orionagent.sources.http-source.port = 5060
4. Flume handler that will parse the notifications, must not be changed
   orionagent.sources.http-source.handler = es.tid.fiware.orionconnectors.cosmosinjector.OrionRestHandler
5. regular expression for the orion version the notifications will have in their headers
   orionagent.sources.http-source.handler.orion_version = 0\.19\.*
6. URL target
   orionagent.sources.http-source.handler.notification_target = /notify

1. channel name from where to read notification events
   orionagent.sinks.hdfs-sink.channel = notifications
2. Flume sink that will process and persist in HDFS the notification events, must not be changed
   orionagent.sinks.hdfs-sink.type = es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink
3. IP address of the Cosmos deployment where the notification events will be persisted
   orionagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
4. port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
   orionagent.sinks.hdfs-sink.cosmos_port = 14000
5. username allowed to write in HDFS (/user/myusername)
   orionagent.sinks.hdfs-sink.cosmos_username = ilko.garcia
   orionagent.sinks.hdfs-sink.cosmos_default_password = *************
6. dataset where to persist the data (/user/myusername/mydataset)
   orionagent.sinks.hdfs-sink.cosmos_dataset = mydataset
7. HDFS backend type (webhdfs, httpfs or infinity)
   orionagent.sinks.hdfs-sink.hdfs_api = httpfs
8. how the attributes are stored, either per row either per column (row, column)
   orionagent.sinks.hdfs-sink.attr_persistence = column

1. channel name
   orionagent.channels.notifications.type = memory
2. capacity of the channel
   orionagent.channels.notifications.capacity = 1000
3. amount of bytes that can be sent per transaction
   orionagent.channels.notifications.transactionCapacity = 100

My Orion instance it's on spain2 region.

Thanks in advance.

======================================================================
EDIT 1

I installed the new version of Cygnus (0.8.1), but does not catch on boot agent settings created.

Here you can see the log flume and configuration files Cygnus.

flume.log

msg=org.mortbay.log.Slf4jLog[67] : Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
msg=com.telefonica.iot.cygnus.nodes.CygnusApplication[238] : Starting Cygnus application
msg=org.mortbay.log.Slf4jLog[67] : jetty-6.1.26
msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider[61] : Configuration provider starting
msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable[133] : Reloading configuration file:/usr/cygnus/conf/agent_umanick.conf
msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:hdfs-sink
msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:hdfs-sink
msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:hdfs-sink
msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:hdfs-sink
msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:hdfs-sink
msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:hdfs-sink
msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[930] : Added sinks: hdfs-sink Agent: cygnusagent
msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:hdfs-sink
msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[336] : Agent configuration for 'cygnusagent' does not contain any valid channels. Marking it as invalid.
msg=org.apache.flume.conf.FlumeConfiguration[127] : Agent configuration invalid for agent 'cygnusagent'. It will be removed.
msg=org.apache.flume.conf.FlumeConfiguration[140] : Post-validation flume configuration contains configuration for agents: []
msg=org.apache.flume.node.AbstractConfigurationProvider[138] : No configuration found for this host:cygnusagent
msg=org.apache.flume.node.Application[138] : Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
msg=org.mortbay.log.Slf4jLog[67] : Started SocketConnector@0.0.0.0:8081
msg=org.apache.flume.node.Application[101] : Shutting down configuration: { sourceRunners:{} sinkRunners:{} channels:{} }
msg=org.apache.flume.node.Application[138] : Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }

agent_umanick.conf

#=============================================

1. To be put in APACHE_FLUME_HOME/conf/cygnus.conf
   #
2. General configuration template explaining how to setup a sink of each of the available types (HDFS, CKAN, MySQL).

#=============================================

1. The next tree fields set the sources, sinks and channels used by Cygnus. You could use different names than the
2. ones suggested below, but in that case make sure you keep coherence in properties names along the configuration file.
3. Regarding sinks, you can use multiple types at the same time; the only requirement is to provide a channel for each
4. one of them (this example shows how to configure 3 sink types at the same time). Even, you can define more than one
5. sink of the same type and sharing the channel in order to improve the performance (this is like having
6. multi-threading).
   cygnusagent.sources = http-source
   cygnusagent.sinks = hdfs-sink
   cygnusagent.channels = hdfs-channel

#=============================================

1. source configuration
2. channel name where to write the notification events
   cygnusagent.sources.http-source.channels = hdfs-channel
3. source class, must not be changed
   cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
4. listening port the Flume source will use for receiving incoming notifications
   cygnusagent.sources.http-source.port = 5050
5. Flume handler that will parse the notifications, must not be changed
   cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
6. URL target
   cygnusagent.sources.http-source.handler.notification_target = /notify
7. Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
   cygnusagent.sources.http-source.handler.events_ttl = 10

1. ============================================
2. OrionHDFSSink configuration
3. channel name from where to read notification events
   cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
4. sink class, must not be changed
   cygnusagent.sinks.hdfs-sink.type = com.telefonica.iot.cygnus.sinks.OrionHDFSSink
5. Comma-separated list of FQDN/IP address regarding the HDFS Namenode endpoints
6. If you are using Kerberos authentication, then the usage of FQDNs instead of IP addresses is mandatory
   cygnusagent.sinks.hdfs-sink.hdfs_host = 130.206.80.46
7. port of the HDFS service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs
   cygnusagent.sinks.hdfs-sink.hdfs_port = 14000
8. username allowed to write in HDFS
   cygnusagent.sinks.hdfs-sink.hdfs_username = ilko.garcia
9. password for the username
   cygnusagent.sinks.hdfs-sink.hdfs_password = *************
10. how the attributes are stored, either per row either per column (row, column)
    cygnusagent.sinks.hdfs-sink.attr_persistence = column

cygnus_instance_umanick.conf

1. The OS user that will be running Cygnus. Note this must be `root` if you want to run cygnus in a privileged port (<1024), either the admin port or the port in which Cygnus receives Orion notifications
   CYGNUS_USER=cygnus
2. Which is the config folder
   CONFIG_FOLDER=/usr/cygnus/conf
3. Which is the config file
   CONFIG_FILE=/usr/cygnus/conf/agent_umanick.conf
4. Name of the agent. The name of the agent is not trivial, since it is the base for the Flume parameters naming conventions, e.g. it appears in <AGENT_NAME>.sources.http-source.channels=...
   AGENT_NAME=orion
5. Name of the logfile located at /var/log/cygnus. It is important to put the extension '.log' in order to the log rotation works properly
   LOGFILE_NAME=cygnus.log
6. Administration port. Must be unique per instance
   ADMIN_PORT=8081
7. Polling interval (seconds) for the configuration reloading
   POLLING_INTERVAL=30

======================================================================
EDIT 2

I added the missing properties and the agent starts properly. But the data is still not persist in cosmos. Agent configuration is the same, with added properties missing

flume.log

msg=org.apache.flume.node.Application[145] : Starting Channel hdfs-channel
msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component type: CHANNEL, name: hdfs-channel started
msg=org.apache.flume.node.Application[173] : Starting Sink hdfs-sink
msg=org.apache.flume.node.Application[184] : Starting Source http-source
msg=org.mortbay.log.Slf4jLog[67] : jetty-6.1.26
msg=org.mortbay.log.Slf4jLog[67] : Started SocketConnector@0.0.0.0:5050
msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component type: SOURCE, name: http-source started
msg=com.telefonica.iot.cygnus.backends.http.HttpClientFactory[79] : Setting max total connections (500)
msg=com.telefonica.iot.cygnus.backends.http.HttpClientFactory[80] : Settubg default max connections per route (100)
msg=com.telefonica.iot.cygnus.sinks.OrionHDFSSink[257] : [hdfs-sink] Startup completed
msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[150] : Starting transaction (1435685260-766-0000000000)
msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[236] : Received data ({ "subscriptionId" : "5592bbe2a85d63e738d33bac", "originator" : "localhost", "contextResponses" : [ { "contextElement" : { "type" : "UMANICK_TECHNOLOGIES_S_L", "isPattern" : "false", "id" : "613cb7464a7950ba", "attributes" : [

{ "name" : "X_VALUE", "type" : "float", "value" : "0.15322891" }

,

{ "name" : "Y_VALUE", "type" : "float", "value" : "-0.076614454" }

,

{ "name" : "Z_VALUE", "type" : "float", "value" : "9.844957" }

] }, "statusCode" :

{ "code" : "200", "reasonPhrase" : "OK" }

} ]})
msg=com.telefonica.iot.cygnus.handlers.OrionRestHandler[258] : Event put in the channel (id=220543398, ttl=10)
msg=com.telefonica.iot.cygnus.sinks.OrionSink[128] : Event got from the channel (id=220543398, headers=

{content-type=application/json, transactionId=1435685260-766-0000000000, fiware-service=def_serv, fiware-servicepath=def_serv_path, ttl=10}

, bodyLength=722)
msg=com.telefonica.iot.cygnus.sinks.OrionSink[184] : null
msg=com.telefonica.iot.cygnus.sinks.OrionSink[193] : Finishing transaction (1435685260-766-0000000000)

======================================================================
EDIT 3

flume.log

Starting transaction (1435739504-848-0000000005)
Received data ({ "subscriptionId" : "5592bbe2a85d63e738d33bac", "originator" : "localhost", "contextResponses" : [ { "contextElement" : { "type" : "UMANICK_TECHNOLOGIES_S_L", "isPattern" : "false", "id" : "613cb7464a7950ba", "attributes" : [

{ "name" : "X_VALUE", "type" : "float", "value" : "0.22984336" }

,

{ "name" : "Y_VALUE", "type" : "float", "value" : "-0.11492168" }

,

{ "name" : "Z_VALUE", "type" : "float", "value" : "9.921572" }

] }, "statusCode" :

{ "code" : "200", "reasonPhrase" : "OK" }

} ]})
Event put in the channel (id=1449000407, ttl=10)
Event got from the channel (id=1449000407, headers=

{timestamp=1435739601153, content-type=application/json, transactionId=1435739504-848-0000000005, fiware-service=def_serv, fiware-servicepath=def_servpath, ttl=10, destination=613cb7464a7950ba_umanick_technologies_s_l}

, bodyLength=721)
Persisting data at OrionHDFSSink. HDFS file (def_serv/def_servpath/613cb7464a7950ba_umanick_technologies_s_l/613cb7464a7950ba_umanick_technologies_s_l.txt), Data (

{"recvTime":"2015-07-01T08:33:21.153Z","X_VALUE":"0.22984336", "X_VALUE_md":[],"Y_VALUE":"-0.11492168", "Y_VALUE_md":[],"Z_VALUE":"9.921572", "Z_VALUE_md":[]}

)
Finishing transaction (1435739504-848-0000000005)