[HELP-12977] [Fiware-tech-help] IMPACT: Delivering CartoDb events through Cygnus - JIRA

Details

Type: extRequest
Status: Closed
Priority: Major
Resolution: Done
Fix Version/s: 2021
Component/s: FIWARE-TECH-HELP
Labels:
None

Sender Email:
peter@familiohq.com,jsalvachua@dit.upm.es,jira-help-desk@jira.fiware.org
External Participants:
jsalvachua@dit.upm.es,fiware-tech-help@lists.fiware.org
HD-Chapter:
Data
HD-Enabler:
Cygnus

Description

I've setup Orion and Cygnus so whenever a new entity is created in Orion
it's transferring its data to Carto using the Cygnus CartoDbSink.

However I'm having some issues with this implementation. I'm getting the
data correctly from Orion to Cygnus but when the sink tries to forward
those data to Carto I'm getting this error message (also attached for
better formatting) when trying to persist it in Carto

cygnus_1 | time=2017-11-23T13:40:11.176Z | lvl=INFO |
corr=d3be117c-d053-11e7-ad11-0242ac120009 |
trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
comp=cygnus-ngsi | op=getEvents |
msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[312] :
[NGSIRestHandler] Received data
({"subscriptionId":"5a16cd57fde41d458740a0b4","originator":"localhost","contextResponses":[{"contextElement":{"type":"Event","isPattern":"false","id":"getif","attributes":[

{"name":"location","type":"geo:point","value":"79.00555, -148.99808"}

]},"statusCode":{"code":"200","reasonPhrase":"OK"}}]})

cygnus_1 | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
corr=d3be117c-d053-11e7-ad11-0242ac120009 |
trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
comp=cygnus-ngsi | op=getEvents |
msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[320] :
[NGSIRestHandler] Parsed NotifyContextRequest:
{"subscriptionId":"5a16cd57fde41d458740a0b4","originator":"localhost","contextResponses":[{"contextElement":{"id":"getif","type":"Event","isPattern":"false","attributes":[

{"name":"location","type":"geo:point","value":"79.00555, -148.99808","metadatas":[]}

]},"statusCode":{"code":"200","reasonPhrase":"OK"}}]}

cygnus_1 | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
corr=d3be117c-d053-11e7-ad11-0242ac120009 |
trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
comp=cygnus-ngsi | op=getEvents |
msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[345] :
[NGSIRestHandler] NGSI event created for ContextElementResponse:
{"contextElement":{"id":"getif","type":"Event","isPattern":"false","attributes":[

{"name":"location","type":"geo:point","value":"79.00555, -148.99808","metadatas":[]}

]},"statusCode":{"code":"200","reasonPhrase":"OK"}}

cygnus_1 | time=2017-11-23T13:40:11.254Z | lvl=INFO |
corr=d3be117c-d053-11e7-ad11-0242ac120009 |
trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
comp=cygnus-ngsi | op=persistRawAggregation |
msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink[616] : [cartodb-sink]
Persisting data at NGSICartoDBSink. Schema (default), Table
(x002fxffffgetifxffffx0045vent), Data
(('2017-11-23T13:40:11.241Z','/','getif','Event',ST_SetSRID(ST_MakePoint(79.00555,-148.99808),
4326)))

cygnus_1 | time=2017-11-23T13:40:11.255Z | lvl=ERROR |
corr=d3be117c-d053-11e7-ad11-0242ac120009 |
trans=efab5d17-12c3-4716-8b1b-*15352ec6ae18

srv=default

subsrv=/

comp=cygnus-ngsi

op=run

msg=org.apache.flume.SinkRunner$PollingRunner[160] : Unable to deliver
event. Exception follows.*

cygnus_1 | java.lang.NullPointerException

*cygnus_1 | at
com.telefonica.iot.cygnus.sinks.NGSICartoDBSink.persistRawAggregation(NGSICartoDBSink.java:625)*

*cygnus_1 | at
com.telefonica.iot.cygnus.sinks.NGSICartoDBSink.persistBatch(NGSICartoDBSink.java:417)*

*cygnus_1 | at
com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:558)*

*cygnus_1 | at
com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)*

*cygnus_1 | at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)*

*cygnus_1 | at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*

cygnus_1 | at java.lang.Thread.run(Thread.java:748)

cygnus_1 | java.lang.IllegalStateException: begin() called when
transaction is OPEN!

cygnus_1 | at
com.google.common.base.Preconditions.checkState(Preconditions.java:145)

cygnus_1 | at
org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)

cygnus_1 | at
com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:482)

cygnus_1 | at
com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)

cygnus_1 | at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

cygnus_1 | at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

cygnus_1 | at java.lang.Thread.run(Thread.java:748)

cygnus_1 | time=2017-11-23T13:40:18.648Z | lvl=DEBUG | corr=N/A

trans=N/A

srv=N/A

subsrv=N/A

comp=cygnus-ngsi

op=run

msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable[126]
: Checking file:/opt/apache-flume/conf/agent.conf for changes

cygnus_1 | time=2017-11-23T13:40:18.654Z | lvl=WARN | corr=N/A

trans=N/A

srv=N/A

subsrv=N/A

comp=cygnus-ngsi

op=<init>

msg=com.telefonica.iot.cygnus.interceptors.CygnusGroupingRules[49] : No
grouping rules have been read. Details:
/usr/cygnus/conf/grouping_rules.conf (No such file or directory)

cygnus_1 | java.lang.IllegalStateException: begin() called when
transaction is OPEN!

cygnus_1 | at
com.google.common.base.Preconditions.checkState(Preconditions.java:145)

cygnus_1 | at
org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)

cygnus_1 | at
com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:482)

cygnus_1 | at
com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)

cygnus_1 | at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

cygnus_1 | at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

cygnus_1 | at java.lang.Thread.run(Thread.java:748)

cygnus_1 | java.lang.IllegalStateException: begin() called when
transaction is OPEN!

cygnus_1 | at
com.google.common.base.Preconditions.checkState(Preconditions.java:145)

cygnus_1 | at
org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)

cygnus_1 | at
com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:482)

cygnus_1 | at
com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)

cygnus_1 | at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

cygnus_1 | at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

cygnus_1 | at java.lang.Thread.run(Thread.java:748)

Are you able to help me figure out why I can't send this information to
carto?

–
Peter Theill
CTO
+45 6171 5096 <61%2071%2050%2096>
peter@familiohq.com

Familio
Artillerivej 86
DK-2300 Copenhagen S
https://familiohq.com/

__________________________________________________________________________________________

You can get more information about our cookies and privacy policies on the following links:

Fiware-tech-help mailing list
Fiware-tech-help@lists.fiware.org
https://lists.fiware.org/listinfo/fiware-tech-help

[Created via e-mail received from: Peter Theill <peter@familiohq.com>]

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Download All

Attachments

carto-cygnus-log.txt

23/Nov/17 11:02 AM

13 kB

FW External User
cygnus-log.txt

20/Dec/17 5:27 PM

15 kB

FW External User
notification-json-geom.sh

12/Dec/17 12:04 PM

2 kB

Andres Muñoz

Activity

Descending order - Click to sort in ascending order

Fernando Lopez made changes - 27/May/21 10:57 AM

Fix Version/s

2021 [ 12600 ]

Andres Muñoz made changes - 06/Feb/18 10:10 AM

Resolution		Done [ 10000 ]
Status	Answered [ 10104 ]	Closed [ 6 ]

Andres Muñoz made changes - 06/Feb/18 10:10 AM

Status

In Progress [ 3 ]

Answered [ 10104 ]

Andres Muñoz made changes - 06/Feb/18 10:10 AM

Status

Impeded [ 10004 ]

In Progress [ 3 ]

Andres Muñoz made changes - 23/Jan/18 10:19 AM

Status

In Progress [ 3 ]

Impeded [ 10004 ]

Andres Muñoz made changes - 23/Jan/18 10:19 AM

Resolution	Done [ 10000 ]
Status	Closed [ 6 ]	In Progress [ 3 ]

Andres Muñoz made changes - 23/Jan/18 10:19 AM

Resolution		Done [ 10000 ]
Status	Answered [ 10104 ]	Closed [ 6 ]

Fernando Lopez made changes - 19/Jan/18 6:12 PM

HD-Enabler		Cygnus [ 11304 ]
HD-Chapter		Data [ 10838 ]

Fernando Lopez made changes - 31/Dec/17 10:47 AM

Comment

Fernando Lopez made changes - 31/Dec/17 10:47 AM

Comment

[ Comment by jira-help-desk@jira.fiware.org :

Spam detection software, running on the system "fiware-forge-01", has
identified this incoming email as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email. If you have any questions, see
the administrator of that system for details.

Content preview: [ https://jira.fiware.org/browse/HELP-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
   ] FW External User updated ~~HELP-12977~~: > [Fiware-general-help] [SPAM] [FIWARE-JIRA]
   (HELP-13082) [SPAM] Website Job Application > > > Key: ~~HELP-12977~~ > URL:
  https://jira.fiware.org/browse/HELP-12977 > Project: Help-Desk > Issue Type:
   extRequest > Reporter: FW External User > Attachments: [FIWARE-JIRA] (~~HELP-12977~~)
   [Fiware-general-help] [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM] Website Job
   Application, [FIWARE-JIRA] (HELP-13082) [Fiware-general-help] [SPAM] Website
   Job Application > > > Spam detection software, running on the system "fiware-forge-01",
   has > identified this incoming email as possible spam. The original message
   > has been attached to this so you can view it (if it isn't spam) or label
   > similar future email. If you have any questions, see > the administrator
   of that system for details. > Content preview: [ https://jira.fiware.org/browse/HELP-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
   > ] FW External User updated HELP-13082: > [Fiware-general-help] [SPAM] Website
   > Job Application > > > Key: HELP-13082 > URL: https://jira.fiware.org/browse/HELP-13082
   > > Project: Help-Desk > Issue Type: extRequest > Components: SPAM > Reporter:
   > FW External User > Attachments: [FIWARE-JIRA] (HELP-12876) [Fiware-general-help]
   > [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM] Website Job Application, [FIWARE-JIRA]
   > (HELP-12877) [Fiware-general-help] [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM]
   > Website Job Application, [FIWARE-JIRA] (HELP-12878) [Fiware-general-help]
   > [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM] Website Job Application, [FIWARE-JIRA]
   > (HELP-12879) [Fiware-general-help] [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM]
   > Website Job Application, [FIWARE-JIRA] (HELP-12880) [Fiware-general-help]
   > [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM] Website Job Application, [FIWARE-JIRA]
   > (HELP-12881) [Fiware-general-help] [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM]
   > Website Job Application, [FIWARE-JIRA] (HELP-12882) [Fiware-general-help]
   > [SPAM] [FIWARE-JIRA] [...]

Content analysis details: (6.9 points, 4.5 required)

pts rule name description
---- ---------------------- --------------------------------------------------
1.7 URIBL_BLACK Contains an URL listed in the URIBL blacklist
                            [URIs: latedeliver.com]
2.5 URIBL_DBL_SPAM Contains a spam URL listed in the DBL blocklist
                            [URIs: latedeliver.com]
0.0 FSL_HELO_NON_FQDN_1 FSL_HELO_NON_FQDN_1
3.8 HELO_LOCALHOST HELO_LOCALHOST
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
                            [score: 0.0000]
0.8 RDNS_NONE Delivered to internal network by a host with no rDNS

]

Fernando Lopez made changes - 31/Dec/17 10:47 AM

Comment

[ Comment by jira-help-desk@jira.fiware.org :

Spam detection software, running on the system "fiware-forge-01", has
identified this incoming email as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email. If you have any questions, see
the administrator of that system for details.

Content preview: [ https://jira.fiware.org/browse/HELP-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
   ] FW External User updated ~~HELP-12977~~: > [Fiware-general-help] [SPAM] [FIWARE-JIRA]
   (HELP-13082) [SPAM] Website Job Application > > > Key: ~~HELP-12977~~ > URL:
  https://jira.fiware.org/browse/HELP-12977 > Project: Help-Desk > Issue Type:
   extRequest > Reporter: FW External User > Attachments: [FIWARE-JIRA] (HELP-13082)
   [Fiware-general-help] [SPAM] Website Job Application > > > Spam detection
   software, running on the system "fiware-forge-01", has > identified this
  incoming email as possible spam. The original message > has been attached
  to this so you can view it (if it isn't spam) or label > similar future email.
   If you have any questions, see > the administrator of that system for details.
   > Content preview: [ https://jira.fiware.org/browse/HELP-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
   > ] FW External User updated HELP-13082: > [Fiware-general-help] [SPAM] Website
   > Job Application > > > Key: HELP-13082 > URL: https://jira.fiware.org/browse/HELP-13082
   > > Project: Help-Desk > Issue Type: extRequest > Components: SPAM > Reporter:
   > FW External User > Attachments: [FIWARE-JIRA] (HELP-12876) [Fiware-general-help]
   > [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM] Website Job Application, [FIWARE-JIRA]
   > (HELP-12877) [Fiware-general-help] [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM]
   > Website Job Application, [FIWARE-JIRA] (HELP-12878) [Fiware-general-help]
   > [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM] Website Job Application, [FIWARE-JIRA]
   > (HELP-12879) [Fiware-general-help] [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM]
   > Website Job Application, [FIWARE-JIRA] (HELP-12880) [Fiware-general-help]
   > [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM] Website Job Application, [FIWARE-JIRA]
   > (HELP-12881) [Fiware-general-help] [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM]
   > Website Job Application, [FIWARE-JIRA] (HELP-12882) [Fiware-general-help]
   > [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM] Website Job Application, [FIWARE-JIRA]
   > (HELP-12883) [Fiware-general-help] [SPAM] [...]

Content analysis details: (6.9 points, 4.5 required)

pts rule name description
---- ---------------------- --------------------------------------------------
1.7 URIBL_BLACK Contains an URL listed in the URIBL blacklist
                            [URIs: latedeliver.com]
2.5 URIBL_DBL_SPAM Contains a spam URL listed in the DBL blocklist
                            [URIs: latedeliver.com]
0.0 FSL_HELO_NON_FQDN_1 FSL_HELO_NON_FQDN_1
3.8 HELO_LOCALHOST HELO_LOCALHOST
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
                            [score: 0.0000]
0.8 RDNS_NONE Delivered to internal network by a host with no rDNS

]

Fernando Lopez made changes - 31/Dec/17 10:46 AM

Attachment

[FIWARE-JIRA] (~~HELP-12977~~) [Fiware-general-help] [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM] Website Job Application [ 61087 ]

Fernando Lopez made changes - 31/Dec/17 10:46 AM

Attachment

[FIWARE-JIRA] (~~HELP-12977~~) [Fiware-general-help] [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM] Website Job Application [ 62597 ]

Fernando Lopez made changes - 31/Dec/17 10:46 AM

Attachment

[FIWARE-JIRA] (~~HELP-12977~~) [Fiware-general-help] [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM] Website Job Application [ 62598 ]

FW External User made changes - 27/Dec/17 4:32 AM

Attachment

[FIWARE-JIRA] (HELP-12977) [Fiware-general-help] [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM] Website Job Application [ 62598 ]

FW External User made changes - 27/Dec/17 4:32 AM

Attachment

[FIWARE-JIRA] (HELP-12977) [Fiware-general-help] [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM] Website Job Application [ 62597 ]

FW External User made changes - 26/Dec/17 11:25 PM

Attachment

[FIWARE-JIRA] (HELP-12977) [Fiware-general-help] [SPAM] [FIWARE-JIRA] (HELP-13082) [SPAM] Website Job Application [ 61087 ]

FW External User made changes - 26/Dec/17 11:25 PM

Sender Email

peter@familiohq.com,jsalvachua@dit.upm.es

peter@familiohq.com,jsalvachua@dit.upm.es,jira-help-desk@jira.fiware.org

FW External User made changes - 20/Dec/17 5:27 PM

Attachment

cygnus-log.txt [ 60128 ]

FW External User made changes - 20/Dec/17 1:49 PM

External Participants

fiware-tech-help@lists.fiware.org

jsalvachua@dit.upm.es,fiware-tech-help@lists.fiware.org

Joaquín Salvachúa made changes - 14/Dec/17 1:27 PM

Assignee

Joaquín Salvachúa [ joaquin.salvachua ]

Andres Muñoz [ andres.munoza ]

Joaquín Salvachúa made changes - 12/Dec/17 1:18 PM

Status

In Progress [ 3 ]

Answered [ 10104 ]

Joaquín Salvachúa made changes - 12/Dec/17 1:18 PM

Assignee

Fernando Lopez [ fla ]

Joaquín Salvachúa [ joaquin.salvachua ]

Joaquín Salvachúa made changes - 12/Dec/17 1:17 PM

Assignee

Joaquín Salvachúa [ joaquin.salvachua ]

Fernando Lopez [ fla ]

Andres Muñoz made changes - 12/Dec/17 12:05 PM

Attachment

notification-json-geom.sh [ 60108 ]

Joaquín Salvachúa made changes - 23/Nov/17 5:34 PM

Status

Open [ 1 ]

In Progress [ 3 ]

Veronika Vlnkova made changes - 23/Nov/17 2:36 PM

Assignee

Joaquín Salvachúa [ joaquin.salvachua ]

FW External User made changes - 23/Nov/17 11:09 AM

External Participants		fiware-tech-help@lists.fiware.org
Sender Email	peter@familiohq.com	peter@familiohq.com,jsalvachua@dit.upm.es

Backlog Manager made changes - 23/Nov/17 11:04 AM

Field	Original Value	New Value
Component/s		FIWARE-TECH-HELP [ 10278 ]

FW External User created issue - 23/Nov/17 11:02 AM

People

Assignee:

Andres Muñoz

Reporter:

FW External User

Votes:

0 Vote for this issue

Watchers:

4 Start watching this issue

Dates

Created:

23/Nov/17 11:02 AM

Updated:

27/May/21 10:57 AM

Resolved:

06/Feb/18 10:10 AM

Agile

View on Board