Details
-
Type: Monitor
-
Status: Closed
-
Priority: Major
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: 2021
-
Component/s: FIWARE-TECH-HELP
-
Labels:
-
HD-Chapter:Data
-
HD-Enabler:Cosmos
Description
Created question in FIWARE Q/A platform on 08-07-2015 at 13:07
Please, ANSWER this question AT https://stackoverflow.com/questions/31291365/null-name-is-given-to-both-folder-and-file-when-sinking-data-from-orion-to-cos
Question:
"null" name is given to both folder and file when sinking data from Orion to Cosmos using Cygnus
Description:
I have an issue related to ngsi2cosmos data flow. Everything works fine when persisting the information received in Orion into the public instance of Cosmos, but the destination folder and file name are both "null".
Simple test as follows:
I create of a brand new NGSIEntity with these headers added: Fiware-Service: myservice & Fiware-ServicePath: /my
I add a new subscription with Cygnus as the reference endpoint.
I send an update to previously created NGSIEntity
When I check my user space in Cosmos I check that the following route has been created: /user/myuser/myservice/null/null.txt
File content is OK, every updated info in Orion has been correctly sinked into it. The problem is with folder and file names.
I can't make it work properly. Isn't it supposed to get entityId and entityType for folder and file naming?
Component versions:
Orion version: contextBroker-0.19.0-1.x86_64
Cygnus version: cygnus-0.5-91.g3eb100e.x86_64
Cosmos: global instance
Cygnus conf file:
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink
cygnusagent.channels = hdfs-channel
#=============================================
- source configuration
- channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel - source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource - listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050 - Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler - URL target
cygnusagent.sources.http-source.handler.notification_target = /notify - Default organization (organization semantic depend on the persistence sink)
cygnusagent.sources.http-source.handler.default_organization = org42 - Number of channel re-injection retries before a Flume event is definitely discarded
cygnusagent.sources.http-source.handler.events_ttl = 10 - Management interface port (FIXME: temporal location for this parameter)
cygnusagent.sources.http-source.handler.management_port = 8081 - Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts - Timestamp interceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp - Destination extractor interceptor, do not change
cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwreconnectors.cygnus.interceptors.DestinationExtractor$Builder - Matching table for the destination extractor interceptor, do not change
cygnusagent.sources.http-source.interceptors.de.matching_table = matching_table.conf
- ============================================
- OrionHDFSSink configuration
- channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel - sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink - Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46 - port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000 - default username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = myuser - default password for the default username
cygnusagent.sinks.hdfs-sink.cosmos_default_password = mypassword - HDFS backend type (webhdfs, httpfs or infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs - how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.hdfs-sink.attr_persistence = column - prefix for the database and table names, empty if no prefix is desired
cygnusagent.sinks.hdfs-sink.naming_prefix = - Hive FQDN/IP address of the Hive server
cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46 - Hive port for Hive external table provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000
#=============================================
- hdfs-channel configuration
- channel type (must not be changed)
cygnusagent.channels.hdfs-channel.type = memory - capacity of the channel
cygnusagent.channels.hdfs-channel.capacity = 1000 - amount of bytes that can be sent per transaction
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
Activity
Field | Original Value | New Value |
---|---|---|
Component/s | FIWARE-TECH-HELP [ 10278 ] |
Status | Open [ 1 ] | In Progress [ 3 ] |
Status | In Progress [ 3 ] | Answered [ 10104 ] |
HD-Enabler | Cosmos [ 10872 ] | |
Description |
Created question in FIWARE Q/A platform on 08-07-2015 at 13:07 {color: red}Please, ANSWER this question AT{color} https://stackoverflow.com/questions/31291365/null-name-is-given-to-both-folder-and-file-when-sinking-data-from-orion-to-cos +Question:+ "null" name is given to both folder and file when sinking data from Orion to Cosmos using Cygnus +Description:+ I have an issue related to ngsi2cosmos data flow. Everything works fine when persisting the information received in Orion into the public instance of Cosmos, but the destination folder and file name are both "null". Simple test as follows: I create of a brand new NGSIEntity with these headers added: Fiware-Service: myservice & Fiware-ServicePath: /my I add a new subscription with Cygnus as the reference endpoint. I send an update to previously created NGSIEntity When I check my user space in Cosmos I check that the following route has been created: /user/myuser/myservice/null/null.txt File content is OK, every updated info in Orion has been correctly sinked into it. The problem is with folder and file names. I can't make it work properly. Isn't it supposed to get entityId and entityType for folder and file naming? Component versions: Orion version: contextBroker-0.19.0-1.x86_64 Cygnus version: cygnus-0.5-91.g3eb100e.x86_64 Cosmos: global instance Cygnus conf file: cygnusagent.sources = http-source cygnusagent.sinks = hdfs-sink cygnusagent.channels = hdfs-channel #============================================= # source configuration # channel name where to write the notification events cygnusagent.sources.http-source.channels = hdfs-channel # source class, must not be changed cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource # listening port the Flume source will use for receiving incoming notifications cygnusagent.sources.http-source.port = 5050 # Flume handler that will parse the notifications, must not be changed cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler # URL target cygnusagent.sources.http-source.handler.notification_target = /notify # Default organization (organization semantic depend on the persistence sink) cygnusagent.sources.http-source.handler.default_organization = org42 # Number of channel re-injection retries before a Flume event is definitely discarded cygnusagent.sources.http-source.handler.events_ttl = 10 # Management interface port (FIXME: temporal location for this parameter) cygnusagent.sources.http-source.handler.management_port = 8081 # Source interceptors, do not change cygnusagent.sources.http-source.interceptors = ts # Timestamp interceptor, do not change cygnusagent.sources.http-source.interceptors.ts.type = timestamp # Destination extractor interceptor, do not change cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwreconnectors.cygnus.interceptors.DestinationExtractor$Builder # Matching table for the destination extractor interceptor, do not change cygnusagent.sources.http-source.interceptors.de.matching_table = matching_table.conf # ============================================ # OrionHDFSSink configuration # channel name from where to read notification events cygnusagent.sinks.hdfs-sink.channel = hdfs-channel # sink class, must not be changed cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink # Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46 # port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty cygnusagent.sinks.hdfs-sink.cosmos_port = 14000 # default username allowed to write in HDFS cygnusagent.sinks.hdfs-sink.cosmos_default_username = myuser # default password for the default username cygnusagent.sinks.hdfs-sink.cosmos_default_password = mypassword # HDFS backend type (webhdfs, httpfs or infinity) cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs # how the attributes are stored, either per row either per column (row, column) cygnusagent.sinks.hdfs-sink.attr_persistence = column # prefix for the database and table names, empty if no prefix is desired cygnusagent.sinks.hdfs-sink.naming_prefix = # Hive FQDN/IP address of the Hive server cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46 # Hive port for Hive external table provisioning cygnusagent.sinks.hdfs-sink.hive_port = 10000 #============================================= # hdfs-channel configuration # channel type (must not be changed) cygnusagent.channels.hdfs-channel.type = memory # capacity of the channel cygnusagent.channels.hdfs-channel.capacity = 1000 # amount of bytes that can be sent per transaction cygnusagent.channels.hdfs-channel.transactionCapacity = 100 |
Created question in FIWARE Q/A platform on 08-07-2015 at 13:07
{color: red}Please, ANSWER this question AT{color} https://stackoverflow.com/questions/31291365/null-name-is-given-to-both-folder-and-file-when-sinking-data-from-orion-to-cos +Question:+ "null" name is given to both folder and file when sinking data from Orion to Cosmos using Cygnus +Description:+ I have an issue related to ngsi2cosmos data flow. Everything works fine when persisting the information received in Orion into the public instance of Cosmos, but the destination folder and file name are both "null". Simple test as follows: I create of a brand new NGSIEntity with these headers added: Fiware-Service: myservice & Fiware-ServicePath: /my I add a new subscription with Cygnus as the reference endpoint. I send an update to previously created NGSIEntity When I check my user space in Cosmos I check that the following route has been created: /user/myuser/myservice/null/null.txt File content is OK, every updated info in Orion has been correctly sinked into it. The problem is with folder and file names. I can't make it work properly. Isn't it supposed to get entityId and entityType for folder and file naming? Component versions: Orion version: contextBroker-0.19.0-1.x86_64 Cygnus version: cygnus-0.5-91.g3eb100e.x86_64 Cosmos: global instance Cygnus conf file: cygnusagent.sources = http-source cygnusagent.sinks = hdfs-sink cygnusagent.channels = hdfs-channel #============================================= # source configuration # channel name where to write the notification events cygnusagent.sources.http-source.channels = hdfs-channel # source class, must not be changed cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource # listening port the Flume source will use for receiving incoming notifications cygnusagent.sources.http-source.port = 5050 # Flume handler that will parse the notifications, must not be changed cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler # URL target cygnusagent.sources.http-source.handler.notification_target = /notify # Default organization (organization semantic depend on the persistence sink) cygnusagent.sources.http-source.handler.default_organization = org42 # Number of channel re-injection retries before a Flume event is definitely discarded cygnusagent.sources.http-source.handler.events_ttl = 10 # Management interface port (FIXME: temporal location for this parameter) cygnusagent.sources.http-source.handler.management_port = 8081 # Source interceptors, do not change cygnusagent.sources.http-source.interceptors = ts # Timestamp interceptor, do not change cygnusagent.sources.http-source.interceptors.ts.type = timestamp # Destination extractor interceptor, do not change cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwreconnectors.cygnus.interceptors.DestinationExtractor$Builder # Matching table for the destination extractor interceptor, do not change cygnusagent.sources.http-source.interceptors.de.matching_table = matching_table.conf # ============================================ # OrionHDFSSink configuration # channel name from where to read notification events cygnusagent.sinks.hdfs-sink.channel = hdfs-channel # sink class, must not be changed cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink # Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46 # port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty cygnusagent.sinks.hdfs-sink.cosmos_port = 14000 # default username allowed to write in HDFS cygnusagent.sinks.hdfs-sink.cosmos_default_username = myuser # default password for the default username cygnusagent.sinks.hdfs-sink.cosmos_default_password = mypassword # HDFS backend type (webhdfs, httpfs or infinity) cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs # how the attributes are stored, either per row either per column (row, column) cygnusagent.sinks.hdfs-sink.attr_persistence = column # prefix for the database and table names, empty if no prefix is desired cygnusagent.sinks.hdfs-sink.naming_prefix = # Hive FQDN/IP address of the Hive server cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46 # Hive port for Hive external table provisioning cygnusagent.sinks.hdfs-sink.hive_port = 10000 #============================================= # hdfs-channel configuration # channel type (must not be changed) cygnusagent.channels.hdfs-channel.type = memory # capacity of the channel cygnusagent.channels.hdfs-channel.capacity = 1000 # amount of bytes that can be sent per transaction cygnusagent.channels.hdfs-channel.transactionCapacity = 100 |
HD-Chapter | Data [ 10838 ] |
Assignee | Francisco Romero [ frb ] |
Resolution | Done [ 10000 ] | |
Status | Answered [ 10104 ] | Closed [ 6 ] |
Fix Version/s | 2021 [ 12600 ] |
Transition | Time In Source Status | Execution Times | Last Executer | Last Execution Date | |||||
---|---|---|---|---|---|---|---|---|---|
|
2h 59m | 1 | Backlog Manager | 23/May/17 9:04 AM | |||||
|
3h | 1 | Backlog Manager | 23/May/17 12:04 PM | |||||
|
5d 19h 34m | 1 | Fernando Lopez | 29/May/17 7:39 AM |
2017-05-23 06:05|CREATED monitor | # answers= 1, accepted answer= False