Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-12709

[Fiware-tech-help] Fiware Cygnus does not store data in Ckan

    Details

    • Type: extRequest
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Fix Version/s: 2021
    • Component/s: FIWARE-TECH-HELP
    • Labels:
      None

      Description

      Hello !

      I asked this question on stackoverflow.com to maybe help people having same
      issue :
      https://stackoverflow.com/questions/46096676/fiware-cygnus-does-not-store-data-in-ckan

      I come back to you in this great Fiware's adventure because i have a little
      issue. Indeed data from the context broker are not stored in ckan
      correctly. A Ckan organization called "def_serv", a ckan dataset called
      "def_serv" too, and a ressource called "stade_philippe_roth_station_meteo"
      are created, but no any data are stored. However, data is perfectly stored
      in mysql. I check the 37 tables content of the postgresql database and not
      found any of my data. I taked care with my token, host, port etc..

      I notice the URL created is "http://none", surely not normal.

      Ckan Interface

      Configuration file `agent_default.conf`

      cygnusagent.sources = http-source
      cygnusagent.sinks = mysql-sink ckan-sink
      cygnusagent.channels = mysql-channel ckan-channel

      cygnusagent.sources.http-source.channels = mysql-channel ckan-channel
      cygnusagent.sources.http-source.type =
      org.apache.flume.source.http.HTTPSource
      cygnusagent.sources.http-source.port = 5050
      cygnusagent.sources.http-source.handler =
      com.telefonica.iot.cygnus.handlers.OrionRestHandler
      cygnusagent.sources.http-source.handler.events_ttl = 10
      cygnusagent.sources.http-source.interceptors = ts gi
      cygnusagent.sources.http-source.interceptors.ts.type = timestamp
      cygnusagent.sources.http-source.interceptors.gi.type =
      com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder

      cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file =
      /usr/cygnus/conf/grouping_rules.conf

      cygnusagent.sinks.mysql-sink.channel = mysql-channel
      cygnusagent.sinks.mysql-sink.type =
      com.telefonica.iot.cygnus.sinks.OrionMySQLSink
      cygnusagent.sinks.mysql-sink.enable_grouping = false
      cygnusagent.sinks.mysql-sink.enable_lowercase = false
      cygnusagent.sinks.mysql-sink.data_model = dm-by-entity
      cygnusagent.sinks.mysql-sink.mysql_host = localhost
      cygnusagent.sinks.mysql-sink.mysql_port = 3306
      cygnusagent.sinks.mysql-sink.mysql_username = root
      cygnusagent.sinks.mysql-sink.mysql_password = pass
      cygnusagent.sinks.mysql-sink.attr_persistence = row
      cygnusagent.sinks.mysql-sink.table_type = dm-by-entity
      cygnusagent.sinks.mysql-sink.batch_size = 1
      cygnusagent.sinks.mysql-sink.batch_timeout = 30
      cygnusagent.sinks.mysql-sink.batch_ttl = 10

      cygnusagent.sinks.ckan-sink.channel = ckan-channel
      cygnusagent.sinks.ckan-sink.type =
      com.telefonica.iot.cygnus.sinks.OrionCKANSink
      cygnusagent.sinks.ckan-sink.enable_grouping = false
      cygnusagent.sinks.ckan-sink.enable_lowercase = false
      cygnusagent.sinks.ckan-sink.data_model = dm-by-entity
      cygnusagent.sinks.ckan-sink.api_key = avalidtoken
      cygnusagent.sinks.ckan-sink.ckan_host = 104.154.59.218
      cygnusagent.sinks.ckan-sink.ckan_port = 80
      cygnusagent.sinks.ckan-sink.orion_url = localhost:1026
      cygnusagent.sinks.ckan-sink.attr_persistence = row
      cygnusagent.sinks.ckan-sink.ssl = false
      cygnusagent.sinks.ckan-sink.batch_size = 1
      cygnusagent.sinks.ckan-sink.batch_timeout = 30
      cygnusagent.sinks.ckan-sink.batch_ttl = 10

      cygnusagent.channels.mysql-channel.type = memory
      cygnusagent.channels.mysql-channel.capacity = 1000
      cygnusagent.channels.mysql-channel.transactionCapacity = 100

      cygnusagent.channels.ckan-channel.type = memory
      cygnusagent.channels.ckan-channel.capacity = 1000
      cygnusagent.channels.ckan-channel.transactionCapacity = 100

      Log file `cygnus.log` (too long for stackoverflow)

      https://pastebin.com/wW8EzCgQ

      Thank you !

      __________________________________________________________________________________________

      You can get more information about our cookies and privacy policies on the following links:

      Fiware-tech-help mailing list
      Fiware-tech-help@lists.fiware.org
      https://lists.fiware.org/listinfo/fiware-tech-help

      [Created via e-mail received from: Pierre Josselin <pierre.josselin@oasc.fr>]

        Activity

        Hide
        andres.munoza Andres Muñoz added a comment -

        Hi Pierre,

        I am reading your question and your log file, but for having a better view of the issue, I need you answer some questions:
        Are you working with Orion as context broker?
        Are you using Cygnus-common or Cygnus-NGSI?
        What version of Cygnus are you using?

        Moreover, When we want to work with data coming from Orion context broker usually we install and configure Cygnus-NGSI because this allows to work with these data in an easy way, I see that un your conf file you are using
        cygnusagent.sinks.ckan-sink.type =com.telefonica.iot.cygnus.sinks.OrionCKANSink but when you use Cygnus NGSI this fiel must be set to com.telefonica.iot.cygnus.sinks.NGSICKANSink. An example of a agent configuration for Cygnus-NGSI using Orion and CKAN sink is available in this link http://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/flume_extensions_catalogue/ngsi_ckan_sink/index.html .

        Please, try to follow the configuration example that is provided in the link give to you and send me the answers of the question that I asked on the top of this comment.

        Best Regards

        Show
        andres.munoza Andres Muñoz added a comment - Hi Pierre, I am reading your question and your log file, but for having a better view of the issue, I need you answer some questions: Are you working with Orion as context broker? Are you using Cygnus-common or Cygnus-NGSI? What version of Cygnus are you using? Moreover, When we want to work with data coming from Orion context broker usually we install and configure Cygnus-NGSI because this allows to work with these data in an easy way, I see that un your conf file you are using cygnusagent.sinks.ckan-sink.type =com.telefonica.iot.cygnus.sinks.OrionCKANSink but when you use Cygnus NGSI this fiel must be set to com.telefonica.iot.cygnus.sinks.NGSICKANSink. An example of a agent configuration for Cygnus-NGSI using Orion and CKAN sink is available in this link http://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/flume_extensions_catalogue/ngsi_ckan_sink/index.html . Please, try to follow the configuration example that is provided in the link give to you and send me the answers of the question that I asked on the top of this comment. Best Regards

          People

          • Assignee:
            andres.munoza Andres Muñoz
            Reporter:
            fw.ext.user FW External User
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: