Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-5444

FIWARE.Request.Tech.Data.CKAN.Fiware Cygnus/CKAN: no data have been persisted in CKAN

    Details

    • Type: extRequest
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Fix Version/s: 2021
    • Component/s: FIWARE-TECH-HELP
    • Labels:
      None

      Description

      I am trying to use cygnus with CKAN, but no data have been persisted in
      CKAN when the attribute is JSON type. First, I send information to Orion:

      Accept: application/json
      X-AUTH-TOKEN: <mytoken>
      Fiware-Service: PapelClubDemo
      Fiware-ServicePath: /events/leonliterario
      {
      "contextElements": [
      {
      "type": "Events",
      "isPattern": "false",
      "id": "thisweek",
      "attributes": [
      {
      "name": "schedule",
      "type": "json",
      "value": [

      {"title": "Presentación Viva Mi Gente","date": "2015-11-30","location": "Salón de actos del ICAL","url": "http://www.papel.club"}

      ]
      }
      ]
      }
      ],
      "updateAction": "APPEND"
      }

      I have a suscription in Cygnus for this entity and this is the information
      I receive in Cygnus log:

      01 Dec 2015 19:05:13,701 INFO [891993589@qtp-1988714671-0]
      (com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:232) -
      Received data ({"subscriptionId" : "565dd3497b72b7c7092d5a29",
      "originator" : "localhost", "contextResponses" : [ { "contextElement"
      : { "type" : "Events","isPattern" : "false", "id" : "thisweek004",
      "attributes" : [ { "name" : "schedule", "type" : "json", "value" : [

      { "title" : "Presentación Viva MiGente", "date" : "2015-11-30", "location" : "Salón de actos del ICAL", "url" : "http://www.papel.club" }

      ,

      { "title" : "Presentación Viva Mi Gente2","date" : "2015-11-30", "location" : "Salón de actos del ICAL", "url" : "http://www.papel.club" }

      ] } ] }, "statusCode" :

      { "code" : "200","reasonPhrase" : "OK" }

      } ]}) 01 Dec 2015 19:05:13,702 INFO
      [891993589@qtp-1988714671-0](com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:255)

      • Event put in the channel (id=2134043204, ttl=10) 01 Dec 2015
        19:05:16,842 INFO[SinkRunner-PollingRunner-DefaultSinkProcessor]
        (com.telefonica.iot.cygnus.sinks.OrionCKANSink.persistOne:207) -
        [ckan-sink] Persisting data atOrionCKANSink (orgName=papelclubdemo,
        pkgName=papelclubdemo_events_leonliterario,
        resName=thisweek004_events,data=1448989513702,2015-12-01T17:05:13.702Z,thisweek004,Events,schedule,json,[ {"title":"Presentación Viva MiGente","date":"2015-11-30","location":"Salón de actos del ICAL","url":"http://www.papel.club"}

        ,

        {"title":"Presentación Viva MiGente2","date":"2015-11-30","location":"Salón de actos del ICAL","url":"http://www.papel.club"}

        ],[])
        01 Dec 2015 19:05:19,479 ERROR
        [SinkRunner-PollingRunner-DefaultSinkProcessor]
        (com.telefonica.iot.cygnus.sinks.OrionSink.process:224) - Runtime
        error (Don't know how to treat response code 503)
        01 Dec 2015 19:05:19,480 INFO
        [SinkRunner-PollingRunner-DefaultSinkProcessor]
        (com.telefonica.iot.cygnus.sinks.OrionSink.process:233) - Finishing
        transaction (1448984542-601-0000000018)

      This is my cygnus agent config:

      1. Flume handler that will parse the notifications, must not be changed
        cygnusagent.sources.http-source.handler =
        com.telefonica.iot.cygnus.handlers.OrionRestHandler
      2. URL target
        cygnusagent.sources.http-source.handler.notification_target = /notify
      3. Default service (service semantic depends on the persistence sink)
        cygnusagent.sources.http-source.handler.default_service = def_serv
      4. Default service path (service path semantic depends on the persistence sink)
        cygnusagent.sources.http-source.handler.default_service_path = def_servpath
      5. Number of channel re-injection retries before a Flume event is
        definitely discarded (-1 means infinite retries)
        cygnusagent.sources.http-source.handler.events_ttl = 10
      6. Source interceptors, do not change
        cygnusagent.sources.http-source.interceptors = ts gi
      7. TimestampInterceptor, do not change
        cygnusagent.sources.http-source.interceptors.ts.type = timestamp
      8. GroupinInterceptor, do not change
        cygnusagent.sources.http-source.interceptors.gi.type =
        com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
      9. Grouping rules for the GroupingInterceptor, put the right absolute
        path to the file if necessary
      10. See the doc/design/interceptors document for more details
        cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file
        = /usr/cygnus/conf/grouping_rules.conf
      1. ============================================
      2. OrionCKANSink configuration
      3. channel name from where to read notification events
        cygnusagent.sinks.ckan-sink.channel = ckan-channel
      4. sink class, must not be changed
        cygnusagent.sinks.ckan-sink.type = com.telefonica.iot.cygnus.sinks.OrionCKANSink
      5. the CKAN API key to use
        #cygnusagent.sinks.ckan-sink.api_key = <mykey>
      6. the FQDN/IP address for the CKAN API endpoint
        cygnusagent.sinks.ckan-sink.ckan_host = demo.ckan.org
      7. the port for the CKAN API endpoint
        cygnusagent.sinks.ckan-sink.ckan_port = 80
      8. Orion URL used to compose the resource URL with the convenience
        operation URL to query it
        cygnusagent.sinks.ckan-sink.orion_url = http://127.0.0.1:1026
      9. how the attributes are stored, either per row either per column (row, column)
        cygnusagent.sinks.ckan-sink.attr_persistence = row
      10. enable SSL for secure Http transportation; 'true' or 'false'
        cygnusagent.sinks.ckan-sink.ssl = false

      When cygnus persists data into demo.ckan.org, organization, dataset and
      resource are created correctyl but data are not loaded.

      Thanks in advance,

      Alfonso García Fernández

      Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost.
      Please, send your messages using the new domain (Fiware-tech-help@lists.fiware.org) instead of the old one.
      _______________________________________________
      Fiware-tech-help mailing list
      Fiware-tech-help@lists.fiware.org
      https://lists.fiware.org/listinfo/fiware-tech-help
      [Created via e-mail received from: =?UTF-8?Q?Alfonso_Garc=C3=ADa_Fern=C3=A1ndez?= <alfongf@gmail.com>]

        Activity

        Hide
        ckan-fiware-okfn OKFN CKAN team added a comment -

        Hi Alfonso,

        I'm a CKAN developer so I can't help you on the Cygnus side of things.

        I'm assuming that the Dataset that you created is this one:

        http://demo.ckan.org/dataset/papelclubdemo_events_leonliterario

        And that the resource that you are referring to is this one:

        http://demo.ckan.org/dataset/papelclubdemo_events_leonliterario/resource/f82a9e79-c3ea-46fd-aced-6224c50e4b85

        That visualization is the expected behaviour for a resource of type JSON on CKAN (displaying the formatted contents of the file).

        If you want to display the contents as a structured table, you need to push the data to the DataStore, like I assume you did on this other one:

        http://demo.ckan.org/dataset/papelclubdemo_events_leonliterario/resource/dbd6bcbd-25ab-4cfa-9b58-be0e41b99183

        Hope this helps, let me know if I misunderstood the issue.

        Best,

        Adrià

        Show
        ckan-fiware-okfn OKFN CKAN team added a comment - Hi Alfonso, I'm a CKAN developer so I can't help you on the Cygnus side of things. I'm assuming that the Dataset that you created is this one: http://demo.ckan.org/dataset/papelclubdemo_events_leonliterario And that the resource that you are referring to is this one: http://demo.ckan.org/dataset/papelclubdemo_events_leonliterario/resource/f82a9e79-c3ea-46fd-aced-6224c50e4b85 That visualization is the expected behaviour for a resource of type JSON on CKAN (displaying the formatted contents of the file). If you want to display the contents as a structured table, you need to push the data to the DataStore, like I assume you did on this other one: http://demo.ckan.org/dataset/papelclubdemo_events_leonliterario/resource/dbd6bcbd-25ab-4cfa-9b58-be0e41b99183 Hope this helps, let me know if I misunderstood the issue. Best, Adrià
        Hide
        fw.ext.user FW External User added a comment -

        Hi Adrià,

        I create the resources "thisweek_events" and "nextweek_events" from
        demo.ckan.org portal uploading json files and in this case it's ok.
        However, the resources "testweeks_event" and "thisweek004_events" where
        created from cygnus. In case of "testweeks_event", I send an attribute of
        string type to cygnus and ckan creates correctly the dataset (if don't
        exist) and the resource and persists the data. However, in case
        "thisweek004_events" I send an attribute of json type to cygnus and ckan
        creates correctly the dataset (if don't exist) and the resource but doesn't
        persist the data, that's my problem.

        I don't know how to solve it, it's possible something was wrong but I'm not
        able to find it.

        Best,

        Alfonso

        Show
        fw.ext.user FW External User added a comment - Hi Adrià, I create the resources "thisweek_events" and "nextweek_events" from demo.ckan.org portal uploading json files and in this case it's ok. However, the resources "testweeks_event" and "thisweek004_events" where created from cygnus. In case of "testweeks_event", I send an attribute of string type to cygnus and ckan creates correctly the dataset (if don't exist) and the resource and persists the data. However, in case "thisweek004_events" I send an attribute of json type to cygnus and ckan creates correctly the dataset (if don't exist) and the resource but doesn't persist the data, that's my problem. I don't know how to solve it, it's possible something was wrong but I'm not able to find it. Best, Alfonso
        Hide
        ichulani ilknur chulani added a comment -

        Hi Francisco,

        Is this anything you are able to help with, as the question seems to involve Cygnus as well?

        Kind regards,

        ilknur

        Show
        ichulani ilknur chulani added a comment - Hi Francisco, Is this anything you are able to help with, as the question seems to involve Cygnus as well? Kind regards, ilknur
        Hide
        fw.ext.user FW External User added a comment -

        Hi Ilknur,

        I already answered this issue in SOF on December the 10th:
        http://stackoverflow.com/questions/34026522/fiware-cygnus-no-data-have-been
        -persisted-in-ckan/34199026#34199026

        Sadly, no more feedback was received by the user (in CC).

        Regards,
        Francisco

        El 14/1/16 12:22, "ilknur chulani (JIRA)" <jira-help-desk@fi-ware.org>

        ________________________________

        Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

        The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

        Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição

        Show
        fw.ext.user FW External User added a comment - Hi Ilknur, I already answered this issue in SOF on December the 10th: http://stackoverflow.com/questions/34026522/fiware-cygnus-no-data-have-been -persisted-in-ckan/34199026#34199026 Sadly, no more feedback was received by the user (in CC). Regards, Francisco El 14/1/16 12:22, "ilknur chulani (JIRA)" <jira-help-desk@fi-ware.org> ________________________________ Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción. The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it. Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição

          People

          • Assignee:
            frb Francisco Romero
            Reporter:
            fw.ext.user FW External User
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: