Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-4785

FIWARE.Question.Tech.Data.BigData-Analysis.How to store data in MySql using cygnus?

    Details

      Description

      Created question in FIWARE Q/A platform on 21-04-2015 at 17:04
      Please, ANSWER this question AT http://stackoverflow.com/questions/29776890/how-to-store-data-in-mysql-using-cygnus

      Question:
      How to store data in MySql using cygnus?

      Description:
      I have read all the documentation about how cygnus works, I specifically tested this one successfully. I also finished reading this tutorial, but I am sure I haven't configured something correctly.

      in cygnus_instance_1.conf I created:

      CYGNUS_USER=root
      CONFIG_FOLDER=/usr/cygnus/conf
      CONFIG_FILE=/usr/cygnus/conf/agent_1.conf
      AGENT_NAME=cygnusagent
      LOGFILE_NAME=cygnus.log
      ADMIN_PORT=8081

      and in agent_1.conf I created:

      #=============================================

      1. To be put in APACHE_FLUME_HOME/conf/cygnus.conf
        #
      2. General configuration template explaining how to setup a sink of each of the available types (HDFS, CKAN, MySQL).

      #=============================================

      1. The next tree fields set the sources, sinks and channels used by Cygnus. You could use different names than the
      2. ones suggested below, but in that case make sure you keep coherence in properties names along the configuration file.
      3. Regarding sinks, you can use multiple types at the same time; the only requirement is to provide a channel for each
      4. one of them (this example shows how to configure 3 sink types at the same time). Even, you can define more than one
      5. sink of the same type and sharing the channel in order to improve the performance (this is like having
      6. multi-threading).
        cygnusagent.sources = http-source
        cygnusagent.sinks = hdfs-sink mysql-sink ckan-sink
        cygnusagent.channels = hdfs-channel mysql-channel ckan-channel

      #=============================================

      1. source configuration
      2. channel name where to write the notification events
        cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel ckan-channel
      3. source class, must not be changed
        cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
      4. listening port the Flume source will use for receiving incoming notifications
        cygnusagent.sources.http-source.port = 5050
      5. Flume handler that will parse the notifications, must not be changed
        cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
      6. URL target
        cygnusagent.sources.http-source.handler.notification_target = /notify
      7. Default service (service semantic depends on the persistence sink)
        cygnusagent.sources.http-source.handler.default_service = def_serv
      8. Default service path (service path semantic depends on the persistence sink)
        cygnusagent.sources.http-source.handler.default_service_path = def_servpath
      9. Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
        cygnusagent.sources.http-source.handler.events_ttl = 10
      10. Source interceptors, do not change
        cygnusagent.sources.http-source.interceptors = ts de
      11. Interceptor type, do not change
        cygnusagent.sources.http-source.interceptors.ts.type = timestamp
      12. Destination extractor interceptor, do not change
        cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor$Builder
      13. Matching table for the destination extractor interceptor, put the right absolute path to the file if necessary
      14. See the doc/design/interceptors document for more details
        cygnusagent.sources.http-source.interceptors.de.matching_table = /usr/cygnus/conf/matching_table.conf
      1. ============================================
      2. OrionHDFSSink configuration
      3. channel name from where to read notification events
        cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
      4. sink class, must not be changed
        cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
      5. Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
      6. If you are using Kerberos authentication, then the usage of FQDNs instead of IP addresses is mandatory
        cygnusagent.sinks.hdfs-sink.cosmos_host = x1.y1.z1.w1,x2.y2.z2.w2
      7. port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
        cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
      8. default username allowed to write in HDFS
        cygnusagent.sinks.hdfs-sink.cosmos_default_username = cosmos_username
      9. default password for the default username
        cygnusagent.sinks.hdfs-sink.cosmos_default_password = xxxxxxxxxxxxx
      10. HDFS backend type (webhdfs, httpfs or infinity)
        cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
      11. how the attributes are stored, either per row either per column (row, column)
        cygnusagent.sinks.hdfs-sink.attr_persistence = column
      12. Hive FQDN/IP address of the Hive server
        cygnusagent.sinks.hdfs-sink.hive_host = x.y.z.w
      13. Hive port for Hive external table provisioning
        cygnusagent.sinks.hdfs-sink.hive_port = 10000
      14. Kerberos-based authentication enabling
        cygnusagent.sinks.hdfs-sink.krb5_auth = false
      15. Kerberos username
        cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_user = krb5_username
      16. Kerberos password
        cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_password = xxxxxxxxxxxxx
      17. Kerberos login file
        cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_login_conf_file = /usr/cygnus/conf/krb5_login.conf
      18. Kerberos configuration file
        cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_conf_file = /usr/cygnus/conf/krb5.conf
      1. ============================================
      2. OrionCKANSink configuration
      3. channel name from where to read notification events
        cygnusagent.sinks.ckan-sink.channel = ckan-channel
      4. sink class, must not be changed
        cygnusagent.sinks.ckan-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionCKANSink
      5. the CKAN API key to use
        cygnusagent.sinks.ckan-sink.api_key = ckanapikey
      6. the FQDN/IP address for the CKAN API endpoint
        cygnusagent.sinks.ckan-sink.ckan_host = x.y.z.w
      7. the port for the CKAN API endpoint
        cygnusagent.sinks.ckan-sink.ckan_port = 80
      8. Orion URL used to compose the resource URL with the convenience operation URL to query it
        cygnusagent.sinks.ckan-sink.orion_url = http://localhost:1026
      9. how the attributes are stored, either per row either per column (row, column)
        cygnusagent.sinks.ckan-sink.attr_persistence = row
      10. enable SSL for secure Http transportation; 'true' or 'false'
        cygnusagent.sinks.ckan-sink.ssl = false
      1. ============================================
      2. OrionMySQLSink configuration
      3. channel name from where to read notification events
        cygnusagent.sinks.mysql-sink.channel = mysql-channel
      4. sink class, must not be changed
        cygnusagent.sinks.mysql-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink
      5. the FQDN/IP address where the MySQL server runs
        cygnusagent.sinks.mysql-sink.mysql_host = localhost
      6. the port where the MySQL server listes for incomming connections
        cygnusagent.sinks.mysql-sink.mysql_port = 3306
      7. a valid user in the MySQL server
        cygnusagent.sinks.mysql-sink.mysql_username = root
      8. password for the user above
        cygnusagent.sinks.mysql-sink.mysql_password = klasika
      9. how the attributes are stored, either per row either per column (row, column)
        cygnusagent.sinks.mysql-sink.attr_persistence = column

      #=============================================

      1. hdfs-channel configuration
      2. channel type (must not be changed)
        cygnusagent.channels.hdfs-channel.type = memory
      3. capacity of the channel
        cygnusagent.channels.hdfs-channel.capacity = 1000
      4. amount of bytes that can be sent per transaction
        cygnusagent.channels.hdfs-channel.transactionCapacity = 100

      #=============================================

      1. ckan-channel configuration
      2. channel type (must not be changed)
        cygnusagent.channels.ckan-channel.type = memory
      3. capacity of the channel
        cygnusagent.channels.ckan-channel.capacity = 1000
      4. amount of bytes that can be sent per transaction
        cygnusagent.channels.ckan-channel.transactionCapacity = 100

      #=============================================

      1. mysql-channel configuration
      2. channel type (must not be changed)
        cygnusagent.channels.mysql-channel.type = memory
      3. capacity of the channel
        cygnusagent.channels.mysql-channel.capacity = 1000
      4. amount of bytes that can be sent per transaction
        cygnusagent.channels.mysql-channel.transactionCapacity = 100

      Although I dont use OrionHDFSSink and OrionCKANSink, I didnt touch those configurations because I really am not sure weather I should.

      When I finally subscribeContext and target cygnus @ default port 5050, I get a normal response, but nothing is created in my database

      What am I doing wrong here?

        Activity

        Hide
        backlogmanager Backlog Manager added a comment -

        2015-10-14 09:05|CREATED monitor | # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2015-10-14 09:05|CREATED monitor | # answers= 1, accepted answer= True
        Hide
        backlogmanager Backlog Manager added a comment -

        2015-10-14 12:05|UPDATED status: transition Answer| # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2015-10-14 12:05|UPDATED status: transition Answer| # answers= 1, accepted answer= True
        Hide
        backlogmanager Backlog Manager added a comment -

        2015-10-14 15:05|UPDATED status: transition Answered| # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2015-10-14 15:05|UPDATED status: transition Answered| # answers= 1, accepted answer= True
        Hide
        backlogmanager Backlog Manager added a comment -

        2015-10-15 03:05|UPDATED status: transition Finish| # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2015-10-15 03:05|UPDATED status: transition Finish| # answers= 1, accepted answer= True

          People

          • Assignee:
            frb Francisco Romero
            Reporter:
            backlogmanager Backlog Manager
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: