Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-9256

[fiware-stackoverflow] Storing SQL in Cosmos

    Details

      Description

      Created question in FIWARE Q/A platform on 22-10-2014 at 19:10
      Please, ANSWER this question AT https://stackoverflow.com/questions/26513757/storing-sql-in-cosmos

      Question:
      Storing SQL in Cosmos

      Description:
      I need to persist data in Cosmos in SQL tables instead of HDFS files.

      I've deployed a VM in Cloud section of FI-Lab where I've installed the 0.14.0 Orion version and 0.3 of Cygnus. I've configured Cygnus to store data in HDFS and SQL... Problem is, persistence in HDFS files works fine but it is not possible in SQL tables despite of in the past I got it. That's why I'm confused

      I guess if HDFS persistence works, it should be problem of cygnus.config file. So I show it below:

      1. APACHE_FLUME_HOME/conf/cygnus.conf
      1. The next tree fields set the sources, sinks and channels used by Cygnus. You could use different names than the
      2. ones suggested below, but in that case make sure you keep coherence in properties names along the configuration file.
      3. Regarding sinks, you can use multiple ones at the same time; the only requirement is to provide a channel for each
      4. one of them (this example shows how to configure 3 sinks at the same time).
        cygnusagent.sources = http-source
        cygnusagent.sinks = hdfs-sink mysql-sink
        cygnusagent.channels = hdfs-channel mysql-channel

      #=============================================

      1. source configuration
      2. channel name where to write the notification events
        cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel
      3. source class, must not be changed
        cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
      4. listening port the Flume source will use for receiving incoming notifications
        cygnusagent.sources.http-source.port = 5050
      5. Flume handler that will parse the notifications, must not be changed
        cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
      6. URL target
        cygnusagent.sources.http-source.handler.notification_target = /notify
      7. Default organization (organization semantic depend on the persistence sink)
        cygnusagent.sources.http-source.handler.default_organization = org42
      1. ============================================
      2. OrionHDFSSink configuration
      3. channel name from where to read notification events
        cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
      4. sink class, must not be changed
        cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
      5. The FQDN/IP address of the Cosmos deployment where the notification events will be persisted
        cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
      6. port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
        cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
      7. default username allowed to write in HDFS
        cygnusagent.sinks.hdfs-sink.cosmos_default_username = quiquehz
      8. default password for the default username
        cygnusagent.sinks.hdfs-sink.cosmos_default_password = 'password'
      9. HDFS backend type (webhdfs, httpfs or infinity)
        cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
      10. how the attributes are stored, either per row either per column (row, column)
        cygnusagent.sinks.hdfs-sink.attr_persistence = column
      11. prefix for the database and table names, empty if no prefix is desired
        cygnusagent.sinks.hdfs-sink.naming_prefix =
      12. Hive port for Hive external table provisioning
        cygnusagent.sinks.hdfs-sink.hive_port = 10000
      1. ============================================
      2. OrionMySQLSink configuration
      3. channel name from where to read notification events
        cygnusagent.sinks.mysql-sink.channel = mysql-channel
      4. sink class, must not be changed
        cygnusagent.sinks.mysql-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink
      5. the FQDN/IP address where the MySQL server runs
        cygnusagent.sinks.mysql-sink.mysql_host = 130.206.80.46
      6. the port where the MySQL server listes for incomming connections
        cygnusagent.sinks.mysql-sink.mysql_port = 3306
      7. a valid user in the MySQL server
        cygnusagent.sinks.mysql-sink.mysql_username = quiquehz
      8. password for the user above
        cygnusagent.sinks.mysql-sink.mysql_password = 'password'
      9. how the attributes are stored, either per row either per column (row, column)
        cygnusagent.sinks.mysql-sink.attr_persistence = column
      10. prefix for the database and table names, empty if no prefix is desired
        cygnusagent.sinks.mysql-sink.naming_prefix =

      #=============================================

      1. hdfs-channel configuration
      2. channel type (must not be changed)
        cygnusagent.channels.hdfs-channel.type = memory
      3. capacity of the channel
        cygnusagent.channels.hdfs-channel.capacity = 1000
      4. amount of bytes that can be sent per transaction
        cygnusagent.channels.hdfs-channel.transactionCapacity = 100

      #=============================================

      1. mysql-channel configuration
      2. channel type (must not be changed)
        cygnusagent.channels.mysql-channel.type = memory
      3. capacity of the channel
        cygnusagent.channels.mysql-channel.capacity = 1000
      4. amount of bytes that can be sent per transaction
        cygnusagent.channels.mysql-channel.transactionCapacity = 100

        Activity

        Hide
        backlogmanager Backlog Manager added a comment -

        2017-05-27 00:05|CREATED monitor | # answers= 1, accepted answer= False

        Show
        backlogmanager Backlog Manager added a comment - 2017-05-27 00:05|CREATED monitor | # answers= 1, accepted answer= False
        Hide
        backlogmanager Backlog Manager added a comment -

        2017-05-27 03:05|UPDATED status: transition Answer| # answers= 1, accepted answer= False

        Show
        backlogmanager Backlog Manager added a comment - 2017-05-27 03:05|UPDATED status: transition Answer| # answers= 1, accepted answer= False
        Hide
        backlogmanager Backlog Manager added a comment -

        2017-05-27 06:05|UPDATED status: transition Answered| # answers= 1, accepted answer= False

        Show
        backlogmanager Backlog Manager added a comment - 2017-05-27 06:05|UPDATED status: transition Answered| # answers= 1, accepted answer= False

          People

          • Assignee:
            frb Francisco Romero
            Reporter:
            backlogmanager Backlog Manager
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: