Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-4987

FIWARE.Question.Tech.Data.BigData-Analysis.Fiware - Cygnus to Cosmos, I can not upload data to HDFS

    Details

      Description

      Created question in FIWARE Q/A platform on 04-02-2015 at 00:02
      Please, ANSWER this question AT http://stackoverflow.com/questions/28310863/fiware-cygnus-to-cosmos-i-can-not-upload-data-to-hdfs

      Question:
      Fiware - Cygnus to Cosmos, I can not upload data to HDFS

      Description:
      I've got an Orion instance with Cygnus, subcription and notify work fine but I can not send files to cosmos.lab.fi-ware.org from my instance.

      [ERROR - es.tid.fiware.orionco nnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)] Connection to http://130.206.80.46:14000 refused

      My cygnus.conf :

      1. APACHE_FLUME_HOME/conf/cygnus.conf
      2. The next tree fields set the sources, sinks and channels used by Cygnus. You could use different names than the
      3. ones suggested below, but in that case make sure you keep coherence in properties names along the configuration file.
      4. Regarding sinks, you can use multiple ones at the same time; the only requirement is to provide a channel for each
      5. one of them (this example shows how to configure 3 sinks at the same time).
        cygnusagent.sources = http-source
        cygnusagent.sinks = hdfs-sink
        cygnusagent.channels = hdfs-channel
        #=============================================
      6. source configuration
      7. channel name where to write the notification events
        cygnusagent.sources.http-source.channels = hdfs-channel
      8. source class, must not be changed
        cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
      9. listening port the Flume source will use for receiving incoming notifications
        cygnusagent.sources.http-source.port = 5050
      10. Flume handler that will parse the notifications, must not be changed
        cygnusagent.sources.http-source.handler =
        es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
      11. URL target
        cygnusagent.sources.http-source.handler.notification_target = /notify
      12. Default organization (organization semantic depend on the persistence sink)
        cygnusagent.sources.http-source.handler.default_organization = org42
      13. ============================================
      14. OrionHDFSSink configuration
      15. channel name from where to read notification events
        cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
      16. sink class, must not be changed
        cygnusagent.sinks.hdfs-sink.type =
        es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
      17. The FQDN/IP address of the Cosmos deployment where the notification events will be persisted
        cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
      18. port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
        cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
      19. default username allowed to write in HDFS
        cygnusagent.sinks.hdfs-sink.cosmos_default_username = myUsername
      20. default password for the default username
        cygnusagent.sinks.hdfs-sink.cosmos_default_password = **********
      21. HDFS backend type (webhdfs, httpfs or infinity)
        cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
      22. how the attributes are stored, either per row either per column (row, column)
        cygnusagent.sinks.hdfs-sink.attr_persistence = column
      23. prefix for the database and table names, empty if no prefix is desired
        cygnusagent.sinks.hdfs-sink.naming_prefix =
      24. Hive port for Hive external table provisioning
        cygnusagent.sinks.hdfs-sink.hive_port = 10000
        #=============================================
      25. hdfs-channel configuration
      26. channel type (must not be changed)
        cygnusagent.channels.hdfs-channel.type = memory
      27. capacity of the channel
        cygnusagent.channels.hdfs-channel.capacity = 1000
      28. amount of bytes that can be sent per transaction
        cygnusagent.channels.hdfs-channel.transactionCapacity = 100

      Error log :

      2015-02-04 22:52:28,627 (lifecycleSupervisor-1-1)
      [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:68)]
      HttpFS operation: PUT 130.206.80.46:14000/webhdfs/v1/user/maxime.mularz/4planet/?op=mkdirs&user.name=maxime.mularz HTTP/1.1
      2015-02-04 22:53:31,690 (lifecycleSupervisor-1-1)
      [ERROR -.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)]
      Connection to http://130.206.80.46:14000 refused
      2015-02-04 22:56:02,182 (SinkRunner-PollingRunner-DefaultSinkProcessor)
      [INFO - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.persist(OrionHDFSSink.java:212)]
      Persisting data. File: Room1-Room-temperature-float.txt, Data: 2015- 02-04T22:56:02.182|1423086962|Room1|Room|temperature|float|90)
      2015-02-04 22:56:02,183 (SinkRunner-PollingRunner-DefaultSinkProcessor)
      [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.exists(HttpFSBackend.java:158)]
      HttpFS operation: GET 130.206.80.46:14000/webhdfs/v1/user/maxime.mularz/4planet/Room1-Room-temperature-float.txt?op=getfilestatus&user.name=maxime.mularz HTTP/1.1

      Thanks in advance.

        Activity

        Hide
        backlogmanager Backlog Manager added a comment -

        2015-10-27 03:05|CREATED monitor | # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2015-10-27 03:05|CREATED monitor | # answers= 1, accepted answer= True
        Hide
        backlogmanager Backlog Manager added a comment -

        2015-10-27 06:05|UPDATED status: transition Answer| # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2015-10-27 06:05|UPDATED status: transition Answer| # answers= 1, accepted answer= True
        Hide
        backlogmanager Backlog Manager added a comment -

        2015-10-27 09:05|UPDATED status: transition Answered| # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2015-10-27 09:05|UPDATED status: transition Answered| # answers= 1, accepted answer= True
        Hide
        backlogmanager Backlog Manager added a comment -

        2015-11-23 12:05|UPDATED status: transition Finish| # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2015-11-23 12:05|UPDATED status: transition Finish| # answers= 1, accepted answer= True

          People

          • Assignee:
            frb Francisco Romero
            Reporter:
            backlogmanager Backlog Manager
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: