Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-9159

[fiware-stackoverflow] "null" name is given to both folder and file when sinking data from Orion to Cosmos using Cygnus

    Details

      Description

      Created question in FIWARE Q/A platform on 08-07-2015 at 13:07
      Please, ANSWER this question AT https://stackoverflow.com/questions/31291365/null-name-is-given-to-both-folder-and-file-when-sinking-data-from-orion-to-cos

      Question:
      "null" name is given to both folder and file when sinking data from Orion to Cosmos using Cygnus

      Description:
      I have an issue related to ngsi2cosmos data flow. Everything works fine when persisting the information received in Orion into the public instance of Cosmos, but the destination folder and file name are both "null".

      Simple test as follows:

      I create of a brand new NGSIEntity with these headers added: Fiware-Service: myservice & Fiware-ServicePath: /my
      I add a new subscription with Cygnus as the reference endpoint.
      I send an update to previously created NGSIEntity

      When I check my user space in Cosmos I check that the following route has been created: /user/myuser/myservice/null/null.txt

      File content is OK, every updated info in Orion has been correctly sinked into it. The problem is with folder and file names.
      I can't make it work properly. Isn't it supposed to get entityId and entityType for folder and file naming?

      Component versions:

      Orion version: contextBroker-0.19.0-1.x86_64
      Cygnus version: cygnus-0.5-91.g3eb100e.x86_64
      Cosmos: global instance

      Cygnus conf file:

      cygnusagent.sources = http-source
      cygnusagent.sinks = hdfs-sink
      cygnusagent.channels = hdfs-channel

      #=============================================

      1. source configuration
      2. channel name where to write the notification events
        cygnusagent.sources.http-source.channels = hdfs-channel
      3. source class, must not be changed
        cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
      4. listening port the Flume source will use for receiving incoming notifications
        cygnusagent.sources.http-source.port = 5050
      5. Flume handler that will parse the notifications, must not be changed
        cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
      6. URL target
        cygnusagent.sources.http-source.handler.notification_target = /notify
      7. Default organization (organization semantic depend on the persistence sink)
        cygnusagent.sources.http-source.handler.default_organization = org42
      8. Number of channel re-injection retries before a Flume event is definitely discarded
        cygnusagent.sources.http-source.handler.events_ttl = 10
      9. Management interface port (FIXME: temporal location for this parameter)
        cygnusagent.sources.http-source.handler.management_port = 8081
      10. Source interceptors, do not change
        cygnusagent.sources.http-source.interceptors = ts
      11. Timestamp interceptor, do not change
        cygnusagent.sources.http-source.interceptors.ts.type = timestamp
      12. Destination extractor interceptor, do not change
        cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwreconnectors.cygnus.interceptors.DestinationExtractor$Builder
      13. Matching table for the destination extractor interceptor, do not change
        cygnusagent.sources.http-source.interceptors.de.matching_table = matching_table.conf
      1. ============================================
      2. OrionHDFSSink configuration
      3. channel name from where to read notification events
        cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
      4. sink class, must not be changed
        cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
      5. Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
        cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
      6. port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
        cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
      7. default username allowed to write in HDFS
        cygnusagent.sinks.hdfs-sink.cosmos_default_username = myuser
      8. default password for the default username
        cygnusagent.sinks.hdfs-sink.cosmos_default_password = mypassword
      9. HDFS backend type (webhdfs, httpfs or infinity)
        cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
      10. how the attributes are stored, either per row either per column (row, column)
        cygnusagent.sinks.hdfs-sink.attr_persistence = column
      11. prefix for the database and table names, empty if no prefix is desired
        cygnusagent.sinks.hdfs-sink.naming_prefix =
      12. Hive FQDN/IP address of the Hive server
        cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
      13. Hive port for Hive external table provisioning
        cygnusagent.sinks.hdfs-sink.hive_port = 10000

      #=============================================

      1. hdfs-channel configuration
      2. channel type (must not be changed)
        cygnusagent.channels.hdfs-channel.type = memory
      3. capacity of the channel
        cygnusagent.channels.hdfs-channel.capacity = 1000
      4. amount of bytes that can be sent per transaction
        cygnusagent.channels.hdfs-channel.transactionCapacity = 100

        Activity

        Hide
        backlogmanager Backlog Manager added a comment -

        2017-05-23 06:05|CREATED monitor | # answers= 1, accepted answer= False

        Show
        backlogmanager Backlog Manager added a comment - 2017-05-23 06:05|CREATED monitor | # answers= 1, accepted answer= False
        Hide
        backlogmanager Backlog Manager added a comment -

        2017-05-23 09:05|UPDATED status: transition Answer| # answers= 1, accepted answer= False

        Show
        backlogmanager Backlog Manager added a comment - 2017-05-23 09:05|UPDATED status: transition Answer| # answers= 1, accepted answer= False
        Hide
        backlogmanager Backlog Manager added a comment -

        2017-05-23 12:05|UPDATED status: transition Answered| # answers= 1, accepted answer= False

        Show
        backlogmanager Backlog Manager added a comment - 2017-05-23 12:05|UPDATED status: transition Answered| # answers= 1, accepted answer= False

          People

          • Assignee:
            frb Francisco Romero
            Reporter:
            backlogmanager Backlog Manager
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: