[HELP-5442] FIWARE.Request.Tech.Data.CKAN.Problema al intentar grabar datos en CKAN desde Cygnus - CKAN side - JIRA

Details

Type: extRequest
Status: Closed
Priority: Major
Resolution: Done
Fix Version/s: 2021
Component/s: FIWARE-TECH-HELP
Labels:
None

Sender Email:
alfongf@gmail.com
HD-Chapter:
Data
HD-Enabler:
CKAN

Description

Buenas tardes.

Estoy intentando grabar datos en CKAN a través de Cygnus, pero no consigo
que funcione correctamente cuando el tipo del atributo es JSON. Lo primero
que hago es la llamada a ContextBroker:

Accept: application/json
X-AUTH-TOKEN: <mytoken>
Fiware-Service: PapelClubDemo
Fiware-ServicePath: /events/leonliterario
{
"contextElements": [
{
"type": "Events",
"isPattern": "false",
"id": "thisweek",
"attributes": [
{
"name": "schedule",
"type": "json",
"value": [

{"title": "Presentación Viva Mi Gente","date": "2015-11-30","location": "Salón de actos del ICAL","url": "http://www.papel.club"}

]
}
]
}
],
"updateAction": "APPEND"
}

Con la correspondiente suscripción de cygnus para esta entidad, obtengo
esta entrada en el log de cygnus:

01 Dec 2015 19:05:13,701 INFO [891993589@qtp-1988714671-0]
(com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:232) -
Received data ({"subscriptionId" : "565dd3497b72b7c7092d5a29",
"originator" : "localhost", "contextResponses" : [ { "contextElement"
: { "type" : "Events","isPattern" : "false", "id" : "thisweek",
"attributes" : [ { "name" : "schedule", "type" : "json", "value" : [

{ "title" : "Presentación Viva Mi Gente", "date" : "2015-11-30", "location" : "Salón de actos del ICAL", "url" : "http://www.papel.club" }

{ "title" : "Presentación Viva Mi Gente2","date" : "2015-11-30", "location" : "Salón de actos del ICAL", "url" : "http://www.papel.club" }

] } ] }, "statusCode" :

{ "code" : "200","reasonPhrase" : "OK" }

} ]})

01 Dec 2015 19:05:13,702 INFO
[891993589@qtp-1988714671-0](com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents:255)

Event put in the channel (id=2134043204, ttl=10) 01 Dec 2015
19:05:16,842 INFO[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionCKANSink.persistOne:207) -
[ckan-sink] Persisting data atOrionCKANSink (orgName=papelclubdemo,
pkgName=papelclubdemo_events_leonliterario,
resName=thisweek004_events,data=1448989513702,2015-12-01T17:05:13.702Z,thisweek,Events,schedule,json,[ {"title":"Presentación Viva MiGente","date":"2015-11-30","location":"Salón de actos del ICAL","url":"http://www.papel.club"}
,
{"title":"Presentación Viva MiGente2","date":"2015-11-30","location":"Salón de actos del ICAL","url":"http://www.papel.club"}
],[])
01 Dec 2015 19:05:19,479 ERROR
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionSink.process:224) - Runtime
error (Don't know how to treat response code 503)
01 Dec 2015 19:05:19,480 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionSink.process:233) - Finishing
transaction (1448984542-601-0000000018)

Esta es la configuración de mi fichero agent en cygnus:

Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler =
com.telefonica.iot.cygnus.handlers.OrionRestHandler
URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = def_serv
Default service path (service path semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service_path = def_servpath
Number of channel re-injection retries before a Flume event is
definitely discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10
Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts gi
TimestampInterceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
GroupinInterceptor, do not change
cygnusagent.sources.http-source.interceptors.gi.type =
com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
Grouping rules for the GroupingInterceptor, put the right absolute
path to the file if necessary
See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file
= /usr/cygnus/conf/grouping_rules.conf

============================================
OrionCKANSink configuration
channel name from where to read notification events
cygnusagent.sinks.ckan-sink.channel = ckan-channel
sink class, must not be changed
cygnusagent.sinks.ckan-sink.type = com.telefonica.iot.cygnus.sinks.OrionCKANSink
the CKAN API key to use
#cygnusagent.sinks.ckan-sink.api_key = <mykey>
the FQDN/IP address for the CKAN API endpoint
cygnusagent.sinks.ckan-sink.ckan_host = demo.ckan.org
the port for the CKAN API endpoint
cygnusagent.sinks.ckan-sink.ckan_port = 80
Orion URL used to compose the resource URL with the convenience
operation URL to query it
cygnusagent.sinks.ckan-sink.orion_url = http://127.0.0.1:1026
how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.ckan-sink.attr_persistence = row
enable SSL for secure Http transportation; 'true' or 'false'
cygnusagent.sinks.ckan-sink.ssl = false

Cuando cygnus reciba la información la envía a ckan y este la recibe,
creándome la organización (si no existe), el dataset y el recurso, pero
cuando accedo a ver el contenido del recurso este está vacío. Estoy
utilizando demo.ckan.org.

Muchas gracias,

Alfonso García Fernández

Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost.
Please, send your messages using the new domain (Fiware-tech-help@lists.fiware.org) instead of the old one.
_______________________________________________
Fiware-tech-help mailing list
Fiware-tech-help@lists.fiware.org
https://lists.fiware.org/listinfo/fiware-tech-help
[Created via e-mail received from: =?UTF-8?Q?Alfonso_Garc=C3=ADa_Fern=C3=A1ndez?= <alfongf@gmail.com>]

Issue Links

relates to

HELP-5452 FIWARE.Request.Tech.Data.BigData-Analysis.Problema al intentar grabar datos en CKAN desde Cygnus - Cygnus side

Closed

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Manuel Escriche added a comment - 02/Dec/15 9:40 AM

Hi Alfonso,

The issue seems to happen in CKAN. So I'm assigning the issue to its GE Owner, who might require some clarifications in english to you.

Kind regards,
Manuel

Show

Manuel Escriche added a comment - 02/Dec/15 9:40 AM Hi Alfonso, The issue seems to happen in CKAN. So I'm assigning the issue to its GE Owner, who might require some clarifications in english to you. Kind regards, Manuel

Hide

Permalink

Manuel Escriche added a comment - 02/Dec/15 10:43 AM

~~HELP-5444~~

Show

Manuel Escriche added a comment - 02/Dec/15 10:43 AM HELP-5444

Hide

Permalink

OKFN CKAN team added a comment - 02/Dec/15 10:56 AM

Hi Alfonso,

I'm a CKAN developer so I can't help you on the Cygnus side of things.

I'm assuming that the Dataset that you created is this one:

http://demo.ckan.org/dataset/papelclubdemo_events_leonliterario

And that the resource that you are referring to is this one:

http://demo.ckan.org/dataset/papelclubdemo_events_leonliterario/resource/f82a9e79-c3ea-46fd-aced-6224c50e4b85

That visualization is the expected behaviour for a resource of type JSON on CKAN (displaying the formatted contents of the file).

If you want to display the contents as a structured table, you need to push the data to the DataStore, like I assume you did on this other one:

http://demo.ckan.org/dataset/papelclubdemo_events_leonliterario/resource/dbd6bcbd-25ab-4cfa-9b58-be0e41b99183

Hope this helps, let me know if I misunderstood the issue.

Best,

Adrià

Show

OKFN CKAN team added a comment - 02/Dec/15 10:56 AM Hi Alfonso, I'm a CKAN developer so I can't help you on the Cygnus side of things. I'm assuming that the Dataset that you created is this one: http://demo.ckan.org/dataset/papelclubdemo_events_leonliterario And that the resource that you are referring to is this one: http://demo.ckan.org/dataset/papelclubdemo_events_leonliterario/resource/f82a9e79-c3ea-46fd-aced-6224c50e4b85 That visualization is the expected behaviour for a resource of type JSON on CKAN (displaying the formatted contents of the file). If you want to display the contents as a structured table, you need to push the data to the DataStore, like I assume you did on this other one: http://demo.ckan.org/dataset/papelclubdemo_events_leonliterario/resource/dbd6bcbd-25ab-4cfa-9b58-be0e41b99183 Hope this helps, let me know if I misunderstood the issue. Best, Adrià

Hide

Permalink

FW External User added a comment - 02/Dec/15 11:07 AM

Hi Adrià,

I create the resources "thisweek_events" and "nextweek_events" from
demo.ckan.org portal uploading json files and in this case it's ok.
However, the resources "testweeks_event" and "thisweek004_events" where
created from cygnus. In case of "testweeks_event", I send an attribute of
string type to cygnus and ckan creates correctly the dataset (if don't
exist) and the resource and persists the data. However, in case
"thisweek004_events" I send an attribute of json type to cygnus and ckan
creates correctly the dataset (if don't exist) and the resource but doesn't
persist the data, that's my problem.

Thanks,

Alfonso

Show

FW External User added a comment - 02/Dec/15 11:07 AM Hi Adrià, I create the resources "thisweek_events" and "nextweek_events" from demo.ckan.org portal uploading json files and in this case it's ok. However, the resources "testweeks_event" and "thisweek004_events" where created from cygnus. In case of "testweeks_event", I send an attribute of string type to cygnus and ckan creates correctly the dataset (if don't exist) and the resource and persists the data. However, in case "thisweek004_events" I send an attribute of json type to cygnus and ckan creates correctly the dataset (if don't exist) and the resource but doesn't persist the data, that's my problem. Thanks, Alfonso

Hide

Permalink

OKFN CKAN team added a comment - 02/Dec/15 2:52 PM

Hi Alfonso,

> In case of "testweeks_event", I send an attribute of
string type to cygnus and ckan creates correctly the dataset (if don't
exist) and the resource and persists the data. However, in case
"thisweek004_events" I send an attribute of json type to cygnus and ckan
creates correctly the dataset (if don't exist) and the resource but doesn't
persist the data, that's my problem.

From the CKAN perspective both these two resources look exactly the same, so I'm assuming there is a difference in the actual API request that Cygnus is doing to create the dataset in CKAN.

Do you have a way of debugging this?

In any case this is caused by the way Cygnus sends the request to CKAN. If we can look at the actual JSON request being sent to CKAN from Cygnus we can help suggest the correct format, but this needs to be fixed or clarified on the Cygnus side.

@Manuel perhaps someone from the Cygnus team can help here?

Adrià

Show

OKFN CKAN team added a comment - 02/Dec/15 2:52 PM Hi Alfonso, > In case of "testweeks_event", I send an attribute of string type to cygnus and ckan creates correctly the dataset (if don't exist) and the resource and persists the data. However, in case "thisweek004_events" I send an attribute of json type to cygnus and ckan creates correctly the dataset (if don't exist) and the resource but doesn't persist the data, that's my problem. From the CKAN perspective both these two resources look exactly the same, so I'm assuming there is a difference in the actual API request that Cygnus is doing to create the dataset in CKAN. Do you have a way of debugging this? In any case this is caused by the way Cygnus sends the request to CKAN. If we can look at the actual JSON request being sent to CKAN from Cygnus we can help suggest the correct format, but this needs to be fixed or clarified on the Cygnus side. @Manuel perhaps someone from the Cygnus team can help here? Adrià

Hide

Permalink

Manuel Escriche added a comment - 03/Dec/15 8:48 AM

Hi Alfonso,

A second ticket to get support from Cygnus GE owner has been created and assigned. It's key is ~~HELP-5452~~.

@Adrià,
if help requested in ~~HELP-5442~~ and ~~HELP-5444~~ are over, please, evolve them.

Thanks for cooperation!
Kind regards,
Manuel

Show

Manuel Escriche added a comment - 03/Dec/15 8:48 AM Hi Alfonso, A second ticket to get support from Cygnus GE owner has been created and assigned. It's key is HELP-5452 . @Adrià, if help requested in HELP-5442 and HELP-5444 are over, please, evolve them. Thanks for cooperation! Kind regards, Manuel

People

Assignee:

OKFN CKAN team

Reporter:

FW External User

Votes:

0 Vote for this issue

Watchers:

3 Start watching this issue

Dates

Created:

01/Dec/15 7:43 PM

Updated:

27/May/21 10:58 AM

Resolved:

07/Dec/15 9:45 AM

Agile

View on Board