Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-12977

[Fiware-tech-help] IMPACT: Delivering CartoDb events through Cygnus

        Details

        • Type: extRequest
        • Status: Closed
        • Priority: Major
        • Resolution: Done
        • Fix Version/s: 2021
        • Component/s: FIWARE-TECH-HELP
        • Labels:
          None

          Description

          I've setup Orion and Cygnus so whenever a new entity is created in Orion
          it's transferring its data to Carto using the Cygnus CartoDbSink.

          However I'm having some issues with this implementation. I'm getting the
          data correctly from Orion to Cygnus but when the sink tries to forward
          those data to Carto I'm getting this error message (also attached for
          better formatting) when trying to persist it in Carto

          cygnus_1 | time=2017-11-23T13:40:11.175Z | lvl=INFO |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[294] :
          [NGSIRestHandler] Starting internal transaction
          (efab5d17-12c3-4716-8b1b-15352ec6ae18)

          cygnus_1 | time=2017-11-23T13:40:11.176Z | lvl=INFO |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[312] :
          [NGSIRestHandler] Received data
          ({"subscriptionId":"5a16cd57fde41d458740a0b4","originator":"localhost","contextResponses":[{"contextElement":{"type":"Event","isPattern":"false","id":"getif","attributes":[

          {"name":"location","type":"geo:point","value":"79.00555, -148.99808"}

          ]},"statusCode":{"code":"200","reasonPhrase":"OK"}}]})

          cygnus_1 | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[320] :
          [NGSIRestHandler] Parsed NotifyContextRequest:
          {"subscriptionId":"5a16cd57fde41d458740a0b4","originator":"localhost","contextResponses":[{"contextElement":{"id":"getif","type":"Event","isPattern":"false","attributes":[

          {"name":"location","type":"geo:point","value":"79.00555, -148.99808","metadatas":[]}

          ]},"statusCode":{"code":"200","reasonPhrase":"OK"}}]}

          cygnus_1 | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[345] :
          [NGSIRestHandler] NGSI event created for ContextElementResponse:
          {"contextElement":{"id":"getif","type":"Event","isPattern":"false","attributes":[

          {"name":"location","type":"geo:point","value":"79.00555, -148.99808","metadatas":[]}

          ]},"statusCode":{"code":"200","reasonPhrase":"OK"}}

          cygnus_1 | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[350] :
          [NGSIRestHandler] Header added to NGSI event (fiware-service: default)

          cygnus_1 | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[353] :
          [NGSIRestHandler] Header added to NGSI event (fiware-servicepath: /)

          cygnus_1 | time=2017-11-23T13:40:11.239Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[356] :
          [NGSIRestHandler] Header added to NGSI event (fiware-correlator:
          d3be117c-d053-11e7-ad11-0242ac120009)

          cygnus_1 | time=2017-11-23T13:40:11.240Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[359] :
          [NGSIRestHandler] Header added to NGSI event (transaction-id:
          efab5d17-12c3-4716-8b1b-15352ec6ae18)

          cygnus_1 | time=2017-11-23T13:40:11.240Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=getEvents |
          msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[385] :
          [NGSIRestHandler] NGSI events put in the channel, ids=1516398619

          cygnus_1 | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=intercept |
          msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[84] :
          [gi] Event intercepted, id=1516398619

          cygnus_1 | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=intercept |
          msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[117] :
          [gi] Adding flume event header (notified-entity: getif_Event)

          cygnus_1 | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=intercept |
          msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[120] :
          [gi] Adding flume event header (grouped-entity: getif_Event)

          cygnus_1 | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=intercept |
          msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[123] :
          [gi] Adding flume event header (grouped-servicepath: /)

          cygnus_1 | time=2017-11-23T13:40:11.241Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=intercept |
          msg=com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor[127] :
          [gi] Event put in the channel, id=1516398619

          orion_1 | time=Thursday 23 Nov 13:40:11 2017.243Z | lvl=INFO |
          corr=N/A | trans=1511444367-204-00000000002 | from=pending | srv=pending |
          subsrv=pending | comp=Orion |
          op=httpRequestSend.cpp[620]:httpRequestSendWithCurl | msg=Notification
          Successfully Sent to http://staging.familiohq.com:5050/notify

          orion_1 | time=Thursday 23 Nov 13:40:11 2017.243Z | lvl=INFO |
          corr=N/A | trans=1511444367-204-00000000002 | from=pending | srv=pending |
          subsrv=pending | comp=Orion | op=logMsg.h[1916]:lmTransactionEnd |
          msg=Transaction ended

          cygnus_1 | time=2017-11-23T13:40:11.244Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=processNewBatches |
          msg=com.telefonica.iot.cygnus.sinks.NGSISink[554] : Batch completed

          cygnus_1 | time=2017-11-23T13:40:11.244Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=persistBatch |
          msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink[381] : [cartodb-sink]
          Processing sub-batch regarding the default_/_getif_Event destination

          cygnus_1 | time=2017-11-23T13:40:11.254Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=aggregate |
          msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink$CartoDBAggregator[570]
          : [cartodb-sink] Processing context element (id=getif, type=Event)

          cygnus_1 | time=2017-11-23T13:40:11.254Z | lvl=DEBUG |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=aggregate |
          msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink$CartoDBAggregator[592]
          : [cartodb-sink] Processing context attribute (name=location,
          type=geo:point)

          cygnus_1 | time=2017-11-23T13:40:11.254Z | lvl=INFO |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=persistRawAggregation |
          msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink[616] : [cartodb-sink]
          Persisting data at NGSICartoDBSink. Schema (default), Table
          (x002fxffffgetifxffffx0045vent), Data
          (('2017-11-23T13:40:11.241Z','/','getif','Event',ST_SetSRID(ST_MakePoint(79.00555,-148.99808),
          4326)))

          cygnus_1 | time=2017-11-23T13:40:11.255Z | lvl=ERROR |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-*15352ec6ae18

          srv=default subsrv=/ comp=cygnus-ngsi op=run

          msg=org.apache.flume.SinkRunner$PollingRunner[160] : Unable to deliver
          event. Exception follows.*

          cygnus_1 | java.lang.NullPointerException

          *cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSICartoDBSink.persistRawAggregation(NGSICartoDBSink.java:625)*

          *cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSICartoDBSink.persistBatch(NGSICartoDBSink.java:417)*

          *cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:558)*

          *cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)*

          *cygnus_1 | at
          org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)*

          *cygnus_1 | at
          org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*

          cygnus_1 | at java.lang.Thread.run(Thread.java:748)

          cygnus_1 | time=2017-11-23T13:40:16.256Z | lvl=ERROR |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=run |
          msg=org.apache.flume.SinkRunner$PollingRunner[160] : Unable to deliver
          event. Exception follows.

          cygnus_1 | java.lang.IllegalStateException: begin() called when
          transaction is OPEN!

          cygnus_1 | at
          com.google.common.base.Preconditions.checkState(Preconditions.java:145)

          cygnus_1 | at
          org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)

          cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:482)

          cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)

          cygnus_1 | at
          org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

          cygnus_1 | at
          org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

          cygnus_1 | at java.lang.Thread.run(Thread.java:748)

          cygnus_1 | time=2017-11-23T13:40:18.648Z | lvl=DEBUG | corr=N/A

          trans=N/A srv=N/A subsrv=N/A comp=cygnus-ngsi op=run

          msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable[126]
          : Checking file:/opt/apache-flume/conf/agent.conf for changes

          cygnus_1 | time=2017-11-23T13:40:18.654Z | lvl=WARN | corr=N/A

          trans=N/A srv=N/A subsrv=N/A comp=cygnus-ngsi op=<init>

          msg=com.telefonica.iot.cygnus.interceptors.CygnusGroupingRules[49] : No
          grouping rules have been read. Details:
          /usr/cygnus/conf/grouping_rules.conf (No such file or directory)

          cygnus_1 | time=2017-11-23T13:40:21.257Z | lvl=ERROR |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=run |
          msg=org.apache.flume.SinkRunner$PollingRunner[160] : Unable to deliver
          event. Exception follows.

          cygnus_1 | java.lang.IllegalStateException: begin() called when
          transaction is OPEN!

          cygnus_1 | at
          com.google.common.base.Preconditions.checkState(Preconditions.java:145)

          cygnus_1 | at
          org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)

          cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:482)

          cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)

          cygnus_1 | at
          org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

          cygnus_1 | at
          org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

          cygnus_1 | at java.lang.Thread.run(Thread.java:748)

          cygnus_1 | time=2017-11-23T13:40:26.258Z | lvl=ERROR |
          corr=d3be117c-d053-11e7-ad11-0242ac120009 |
          trans=efab5d17-12c3-4716-8b1b-15352ec6ae18 | srv=default | subsrv=/ |
          comp=cygnus-ngsi | op=run |
          msg=org.apache.flume.SinkRunner$PollingRunner[160] : Unable to deliver
          event. Exception follows.

          cygnus_1 | java.lang.IllegalStateException: begin() called when
          transaction is OPEN!

          cygnus_1 | at
          com.google.common.base.Preconditions.checkState(Preconditions.java:145)

          cygnus_1 | at
          org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)

          cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:482)

          cygnus_1 | at
          com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:370)

          cygnus_1 | at
          org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

          cygnus_1 | at
          org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

          cygnus_1 | at java.lang.Thread.run(Thread.java:748)

          Are you able to help me figure out why I can't send this information to
          carto?

          –
          Peter Theill
          CTO
          +45 6171 5096 <61%2071%2050%2096>
          peter@familiohq.com

          Familio
          Artillerivej 86
          DK-2300 Copenhagen S
          https://familiohq.com/

          __________________________________________________________________________________________

          You can get more information about our cookies and privacy policies on the following links:

          Fiware-tech-help mailing list
          Fiware-tech-help@lists.fiware.org
          https://lists.fiware.org/listinfo/fiware-tech-help

          [Created via e-mail received from: Peter Theill <peter@familiohq.com>]

          1. Text File
            carto-cygnus-log.txt
            13 kB
            FW External User
          2. Text File
            cygnus-log.txt
            15 kB
            FW External User
          3. File
            notification-json-geom.sh
            2 kB
            Andres Muñoz

            Activity

            Ascending order - Click to sort in descending order
            15 older comments
            Hide
            Permalink
            andres.munoza Andres Muñoz added a comment -

            Hello Peter,

            I am still checking your issue but I need that you perform some other tests, please once again run the script that I provide you but this time with this 2 more test values :

            ./notification.sh http://localhost:5050/notify myservice /traffic 40.361 -3.4099

            remember that you must execute this script pointing to your cygnus host and port

            then tell me if it was successful persisted in the cartodb otherwise give me the output log.

            If and only if it fails please try to execute the next command for testing if the cartodb entities are correctly created and available

            for inserting data to cartodb

            curl -G "https://<my_user>.cartodb.com/api/v2/sql?api_key=<api_key>" --data-urlencode "q=INSERT INTO <table_name> (recvTime, fiwareServicePath, entityId, entityType, <attr_1>, <attr_1>_md, ..., <attr_n>, <attr_n>_md, the_geom) VALUES ('2016-04-19T07:09:53.116Z', '<service_path>', '<entity_id>', '<entity_type>', '<attr_1_value>', '<attr_1_metadata>', ..., '<attr_n_value>', '<attr_n_metadata>', 'ST_SetSRID(ST_MakePoint(<lat>, <lon>), 4326))"

            for query this data

            curl -G "https://<my_user>.cartodb.com/api/v2/sql?api_key=<api_key>" --data-urlencode "q=SELECT * FROM <table_name>"

            I will be waiting for your comments.

            Best Regards

            Show
            andres.munoza Andres Muñoz added a comment - Hello Peter, I am still checking your issue but I need that you perform some other tests, please once again run the script that I provide you but this time with this 2 more test values : ./notification.sh http://localhost:5050/notify myservice /traffic 40.361 -3.4099 remember that you must execute this script pointing to your cygnus host and port then tell me if it was successful persisted in the cartodb otherwise give me the output log. If and only if it fails please try to execute the next command for testing if the cartodb entities are correctly created and available for inserting data to cartodb curl -G "https://<my_user>.cartodb.com/api/v2/sql?api_key=<api_key>" --data-urlencode "q=INSERT INTO <table_name> (recvTime, fiwareServicePath, entityId, entityType, <attr_1>, <attr_1>_md, ..., <attr_n>, <attr_n>_md, the_geom) VALUES ('2016-04-19T07:09:53.116Z', '<service_path>', '<entity_id>', '<entity_type>', '<attr_1_value>', '<attr_1_metadata>', ..., '<attr_n_value>', '<attr_n_metadata>', 'ST_SetSRID(ST_MakePoint(<lat>, <lon>), 4326))" for query this data curl -G "https://<my_user>.cartodb.com/api/v2/sql?api_key=<api_key>" --data-urlencode "q=SELECT * FROM <table_name>" I will be waiting for your comments. Best Regards
            Hide
            Permalink
            fw.ext.user FW External User added a comment -

            Comment by peter@familiohq.com :

            Thanks! I'll run the scripts and get back to you.

            On Mon, Jan 8, 2018 at 4:22 PM Help-Desk <jira-help-desk@jira.fiware.org>
            wrote:

            >
            >

            Show
            fw.ext.user FW External User added a comment - Comment by peter@familiohq.com : Thanks! I'll run the scripts and get back to you. On Mon, Jan 8, 2018 at 4:22 PM Help-Desk <jira-help-desk@jira.fiware.org> wrote: > >
            Hide
            Permalink
            andres.munoza Andres Muñoz added a comment -

            Hello Peter,

            I am waiting for the results of the test, as soon as you have any feedback please tell me.

            Best Regards

            Show
            andres.munoza Andres Muñoz added a comment - Hello Peter, I am waiting for the results of the test, as soon as you have any feedback please tell me. Best Regards
            Hide
            Permalink
            andres.munoza Andres Muñoz added a comment -

            Hello Peter,

            Please, Could you give me some feedback about the tests that I told you to do?

            Best Regards,

            Show
            andres.munoza Andres Muñoz added a comment - Hello Peter, Please, Could you give me some feedback about the tests that I told you to do? Best Regards,
            Hide
            Permalink
            andres.munoza Andres Muñoz added a comment -

            Hi,

            I am still waiting for the user feedback.

            Best Regards

            Show
            andres.munoza Andres Muñoz added a comment - Hi, I am still waiting for the user feedback. Best Regards

              People

              • Assignee:
                andres.munoza Andres Muñoz
                Reporter:
                fw.ext.user FW External User
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: