Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-12977

[Fiware-tech-help] IMPACT: Delivering CartoDb events through Cygnus

        Details

        • Type: extRequest
        • Status: Closed
        • Priority: Major
        • Resolution: Done
        • Fix Version/s: 2021
        • Component/s: FIWARE-TECH-HELP
        • Labels:
          None

          Description

          I've setup Orion and Cygnus so whenever a new entity is created in Orion
          it's transferring its data to Carto using the Cygnus CartoDbSink.

          However I'm having some issues with this implementation. I'm getting the
          data correctly from Orion to Cygnus but when the sink tries to forward
          those data to Carto I'm getting this error message (also attached for
          better formatting) when trying to persist it in Carto

          cygnus_1 | time=2017-11-23T13:40:11.175Z | lvl=INFO |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[294] :
          [NGSIRestHandler] Starting internal transaction
          (efab5d17-12c3-4716-8b1b-15352ec6ae18)

          cygnus_1 | time=2017-11-23T13:40:11.176Z | lvl=INFO |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[312] :
          [NGSIRestHandler] Received data
          ({"subscriptionId":"5a16cd57fde41d458740a0b4","originator":"localhost","contextResponses":[{"contextElement":{"type":"Event","isPattern":"false","id":"getif","attributes":[

          {"name":"location","type":"geo:point","value":"79.00555, -148.99808"}

          ]},"statusCode":{"code":"200","reasonPhrase":"OK"}}]})

          cygnus_1 | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[320] :
          [NGSIRestHandler] Parsed NotifyContextRequest:
          {"subscriptionId":"5a16cd57fde41d458740a0b4","originator":"localhost","contextResponses":[{"contextElement":{"id":"getif","type":"Event","isPattern":"false","attributes":[

          {"name":"location","type":"geo:point","value":"79.00555, -148.99808","metadatas":[]}

          ]},"statusCode":{"code":"200","reasonPhrase":"OK"}}]}

          cygnus_1 | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[345] :
          [NGSIRestHandler] NGSI event created for ContextElementResponse:
          {"contextElement":{"id":"getif","type":"Event","isPattern":"false","attributes":[

          {"name":"location","type":"geo:point","value":"79.00555, -148.99808","metadatas":[]}

          ]},"statusCode":{"code":"200","reasonPhrase":"OK"}}

          cygnus_1 | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[350] :
          [NGSIRestHandler] Header added to NGSI event (fiware-service: default)

          cygnus_1 | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[353] :
          [NGSIRestHandler] Header added to NGSI event (fiware-servicepath: /)

          cygnus_1 | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[356] :
          [NGSIRestHandler] Header added to NGSI event (fiware-correlator:
          d3be117c-d053-11e7-ad11-0242ac120009)

          cygnus_1 | time=2017-11-23T13:40:11.240Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[359] :
          [NGSIRestHandler] Header added to NGSI event (transaction-id:
          efab5d17-12c3-4716-8b1b-15352ec6ae18)

          cygnus_1 | time=2017-11-23T13:40:11.240Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[385] :
          [NGSIRestHandler] NGSI events put in the channel, ids=1516398619

          cygnus_1 | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=intercept |
          msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[84] :
          [gi] Event intercepted, id=1516398619

          cygnus_1 | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=intercept |
          msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[117] :
          [gi] Adding flume event header (notified-entity: getif_Event)

          cygnus_1 | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=intercept |
          msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[120] :
          [gi] Adding flume event header (grouped-entity: getif_Event)

          cygnus_1 | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=intercept |
          msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[123] :
          [gi] Adding flume event header (grouped-servicepath: /)

          cygnus_1 | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=intercept |
          msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[127] :
          [gi] Event put in the channel, id=1516398619

          orion_1 | time=Thursday 23 Nov 13:40:11 2017.243Z | lvl=INFO |
          corr=N/A | trans=1511444367-204-00000000002 | from=pending | srv=pending |
          subsrv=pending | comp=Orion |
          op=httpRequestSend.cpp[620]:httpRequestSendWithCurl | msg=Notification
          Successfully Sent to http://staging.familiohq.com:5050/notify

          orion_1 | time=Thursday 23 Nov 13:40:11 2017.243Z | lvl=INFO |
          corr=N/A | trans=1511444367-204-00000000002 | from=pending | srv=pending |
          subsrv=pending | comp=Orion | op=logMsg.h[1916]:lmTransactionEnd |
          msg=Transaction ended

          cygnus_1 | time=2017-11-23T13:40:11.244Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=processNewBatches |
          msg=com.telefonica.iot.cygnus.sinks.NGSISink[554] : Batch completed

          cygnus_1 | time=2017-11-23T13:40:11.244Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=persistBatch |
          msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink[381] : [cartodb-sink]
          Processing sub-batch regarding the default_/_getif_Event destination

          cygnus_1 | time=2017-11-23T13:40:11.254Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=aggregate |
          msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink$CartoDBAggregator[570]
          : [cartodb-sink] Processing context element (id=getif, type=Event)

          cygnus_1 | time=2017-11-23T13:40:11.254Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=aggregate |
          msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink$CartoDBAggregator[592]
          : [cartodb-sink] Processing context attribute (name=location,
          type=geo:point)

          cygnus_1 | time=2017-11-23T13:40:11.254Z | lvl=INFO |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=persistRawAggregation |
          msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink[616] : [cartodb-sink]
          Persisting data at NGSICartoDBSink. Schema (default), Table
          (x002fxffffgetifxffffx0045vent), Data
          (('2017-11-23T13:40:11.241Z','/','getif','Event',ST_SetSRID(ST_MakePoint(79.00555,-148.99808),
          4326)))

          cygnus_1 | time=2017-11-23T13:40:11.255Z | lvl=ERROR |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-*15352ec6ae18

          srv=default subsrv=/ comp=cygnus-ngsi op=run

          msg=org.apache.flume.SinkRunner$PollingRunner[160] : Unable to deliver
          event. Exception follows.*

          cygnus_1 | java.lang.NullPointerException

          *cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSICartoDBSink.persistRawAggregation(NGSICartoDBSink.java:625)*

          *cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSICartoDBSink.persistBatch(NGSICartoDBSink.java:417)*

          *cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:558)*

          *cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)*

          *cygnus_1 | at
          org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)*

          *cygnus_1 | at
          org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*

          cygnus_1 | at java.lang.Thread.run(Thread.java:748)

          cygnus_1 | time=2017-11-23T13:40:16.256Z | lvl=ERROR |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=run |
          msg=org.apache.flume.SinkRunner$PollingRunner[160] : Unable to deliver
          event. Exception follows.

          cygnus_1 | java.lang.IllegalStateException: begin() called when
          transaction is OPEN!

          cygnus_1 | at
          com.google.common.base.Preconditions.checkState(Preconditions.java:145)

          cygnus_1 | at
          org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)

          cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:482)

          cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)

          cygnus_1 | at
          org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

          cygnus_1 | at
          org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

          cygnus_1 | at java.lang.Thread.run(Thread.java:748)

          cygnus_1 | time=2017-11-23T13:40:18.648Z | lvl=DEBUG | corr=N/A

          trans=N/A srv=N/A subsrv=N/A comp=cygnus-ngsi op=run

          msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable[126]
          : Checking file:/opt/apache-flume/conf/agent.conf for changes

          cygnus_1 | time=2017-11-23T13:40:18.654Z | lvl=WARN | corr=N/A

          trans=N/A srv=N/A subsrv=N/A comp=cygnus-ngsi op=<init>

          msg=com.telefonica.iot.cygnus.interceptors.CygnusGroupingRules[49] : No
          grouping rules have been read. Details:
          /usr/cygnus/conf/grouping_rules.conf (No such file or directory)

          cygnus_1 | time=2017-11-23T13:40:21.257Z | lvl=ERROR |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=run |
          msg=org.apache.flume.SinkRunner$PollingRunner[160] : Unable to deliver
          event. Exception follows.

          cygnus_1 | java.lang.IllegalStateException: begin() called when
          transaction is OPEN!

          cygnus_1 | at
          com.google.common.base.Preconditions.checkState(Preconditions.java:145)

          cygnus_1 | at
          org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)

          cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:482)

          cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)

          cygnus_1 | at
          org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

          cygnus_1 | at
          org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

          cygnus_1 | at java.lang.Thread.run(Thread.java:748)

          cygnus_1 | time=2017-11-23T13:40:26.258Z | lvl=ERROR |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=run |
          msg=org.apache.flume.SinkRunner$PollingRunner[160] : Unable to deliver
          event. Exception follows.

          cygnus_1 | java.lang.IllegalStateException: begin() called when
          transaction is OPEN!

          cygnus_1 | at
          com.google.common.base.Preconditions.checkState(Preconditions.java:145)

          cygnus_1 | at
          org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)

          cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:482)

          cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)

          cygnus_1 | at
          org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

          cygnus_1 | at
          org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

          cygnus_1 | at java.lang.Thread.run(Thread.java:748)

          Are you able to help me figure out why I can't send this information to
          carto?

          –
          Peter Theill
          CTO
          +45 6171 5096 <61%2071%2050%2096>
          peter@familiohq.com

          Familio
          Artillerivej 86
          DK-2300 Copenhagen S
          https://familiohq.com/

          __________________________________________________________________________________________

          You can get more information about our cookies and privacy policies on the following links:

          Fiware-tech-help mailing list
          Fiware-tech-help@lists.fiware.org
          https://lists.fiware.org/listinfo/fiware-tech-help

          [Created via e-mail received from: Peter Theill <peter@familiohq.com>]

          1. Text File
            cygnus-log.txt
            15 kB
            FW External User
          2. File
            notification-json-geom.sh
            2 kB
            Andres Muñoz
          3. Text File
            carto-cygnus-log.txt
            13 kB
            FW External User

            Activity

              People

              • Assignee:
                andres.munoza Andres Muñoz
                Reporter:
                fw.ext.user FW External User
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: