Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-19318

FIWARE.Question.Tech.Cygnus does not persist data into PostSQL.

    Details

      Description

      Created question in FIWARE Q/A platform on 10-03-2022 at 17:03
      Please, ANSWER this question AT https://stackoverflow.com/questions/71428255/cygnus-does-not-persist-data-into-postsql

      Question:
      Cygnus does not persist data into PostSQL

      Description:
      I am using the following tutorial to persist data in a PostGresQL database via Cygnus.
      https://documenter.getpostman.com/view/513743/RWEcR2DC
      When using a MySQL database with Cygnus everything works fine as you can see below:
      mysql> SHOW tables FROM openiot;
      -------------------------------------------------------

      Tables_in_openiot

      -------------------------------------------------------

      urn_ngsi-ld_TemperatureSensor_11401_TemperatureSensor
      urn_ngsi-ld_TemperatureSensor_12435_TemperatureSensor
      urn_ngsi-ld_TemperatureSensor_13477_TemperatureSensor
      urn_ngsi-ld_TemperatureSensor_13828_TemperatureSensor
      urn_ngsi-ld_TemperatureSensor_15205_TemperatureSensor
      urn_ngsi-ld_TemperatureSensor_1546_TemperatureSensor

      -------------------------------------------------------
      109 rows in set (0.00 sec)

      But when using PostGres as backend, the NGSIPostgreSQLSink does not create tables.
      Here is my setup on docker:
      postgres:
      image: postgres:latest
      hostname: postgres
      expose:

      • "5432"
        ports:
      • "5432:5432"
        environment:
      • "POSTGRES_PASSWORD=password"
      • "POSTGRES_USER=postgres"
      • "POSTGRES_DB=postgres"
        volumes:
      • data-postgres:/var/lib/postgresql/data

      cygnus:
      image: fiware/cygnus-ngsi:latest
      hostname: cygnus
      depends_on:

      • postgres
        expose:
      • "5080"
        ports:
      • "5055:5055"
      • "5080:5080"
        environment:
      • "CYGNUS_POSTGRESQL_HOST=postgres"
      • "CYGNUS_POSTGRESQL_PORT=5432"
      • "CYGNUS_POSTGRESQL_USER=postgres"
      • "CYGNUS_POSTGRESQL_PASS=password"
      • "CYGNUS_POSTGRESQL_ENABLE_CACHE=true"
      • "CYGNUS_POSTGRESQL_SERVICE_PORT=5055"
      • "CYGNUS_LOG_LEVEL=DEBUG"
      • "CYGNUS_API_PORT=5080"
      • "CYGNUS_SERVICE_PORT=5055"

      If I list the schemas with a PostGres client there is no openiot schema:
      postgres=# \dn
      List of schemas
      Name | Owner
      -------+---------
      public | postgres
      (1 row)

      I also tried to mount the following agent.conf and cygnus_instance.conf files in docker:
      docker
      volumes:

      • ./conf/cygnus:/opt/apache-flume/conf/agent:ro

      agent.conf
      cygnus-ngsi.sources = http-source
      cygnus-ngsi.sinks = postgresql-sink
      cygnus-ngsi.channels = postgresql-channel

      cygnus-ngsi.sources.http-source.channels = postgresql-channel
      cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
      cygnus-ngsi.sources.http-source.port = 5050
      cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
      cygnus-ngsi.sources.http-source.handler.notification_target = /notify
      cygnus-ngsi.sources.http-source.handler.default_service = default
      cygnus-ngsi.sources.http-source.handler.default_service_path = /
      cygnus-ngsi.sources.http-source.interceptors = ts gi
      cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
      cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
      cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
      cygnus-ngsi.sources.http-source.interceptors.nmi.type = com.telefonica.iot.cygnus.interceptors.NGSINameMappingsInterceptor$Builder
      cygnus-ngsi.sources.http-source.interceptors.nmi.name_mappings_conf_file = /usr/cygnus/conf/name_mappings.conf

      cygnus-ngsi.sinks.postgresql-sink.channel = postgresql-channel
      cygnus-ngsi.sinks.postgresql-sink.type = com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink
      cygnus-ngsi.sinks.postgresql-sink.enable_encoding = false
      cygnus-ngsi.sinks.postgresql-sink.enable_name_mappings = false
      cygnus-ngsi.sinks.postgresql-sink.enable_grouping = false
      cygnus-ngsi.sinks.postgresql-sink.enable_lowercase = false
      cygnus-ngsi.sinks.postgresql-sink.postgresql_host = postgres
      cygnus-ngsi.sinks.postgresql-sink.postgresql_port = 5432
      cygnus-ngsi.sinks.postgresql-sink.postgresql_database = postgres
      cygnus-ngsi.sinks.postgresql-sink.postgresql_username = postgres
      cygnus-ngsi.sinks.postgresql-sink.postgresql_password = password
      cygnus-ngsi.sinks.postgresql-sink.attr_persistence = column
      cygnus-ngsi.sinks.postgresql-sink.data_model = by-service-path
      cygnus-ngsi.sinks.postgresql-sink.batch_size = 100
      cygnus-ngsi.sinks.postgresql-sink.batch_timeout = 30
      cygnus-ngsi.sinks.postgresql-sink.batch_ttl = 10
      cygnus-ngsi.sinks.postgresql-sink.persist_errors = true

      cygnus-ngsi.channels.postgresql-channel.type = memory
      cygnus-ngsi.channels.postgresql-channel.capacity = 100000
      cygnus-ngsi.channels.postgresql-channel.transactionCapacity = 10000

      cygnus_instance.conf
      CYGNUS_USER=cygnus
      CONFIG_FOLDER=/usr/cygnus/conf
      CONFIG_FILE=/usr/cygnus/conf/agent.conf
      AGENT_NAME=cygnusagent
      LOGFILE_NAME=cygnus.log
      ADMIN_PORT=8081
      POLLING_INTERVAL=30

      But to no avail Cygnus is still not persisting data in PostGres.
      Here are some logs from the Cygnus service:
      time=2022-03-10T17:02:12.429Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation start false:ACTION:Starts the instance

      time=2022-03-10T17:02:12.429Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: start()

      time=2022-03-10T17:02:12.429Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[318] : Method Annotation found for: stop

      time=2022-03-10T17:02:12.429Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation stop false:ACTION:Stops the instance

      time=2022-03-10T17:02:12.430Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: stop()

      time=2022-03-10T17:02:12.431Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[286] : Influenced by: org.eclipse.jetty.util.component.Container

      time=2022-03-10T17:02:12.431Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[291] : Annotations not found for: org.eclipse.jetty.util.component.Container

      time=2022-03-10T17:02:12.431Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[286] : Influenced by: org.eclipse.jetty.util.component.Destroyable

      time=2022-03-10T17:02:12.432Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[318] : Method Annotation found for: destroy

      time=2022-03-10T17:02:12.432Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation destroy false:ACTION:Destroys this component

      time=2022-03-10T17:02:12.432Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: destroy()

      time=2022-03-10T17:02:12.432Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[286] : Influenced by: org.eclipse.jetty.util.component.Dumpable

      time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[318] : Method Annotation found for: dump

      time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation dump false:INFO:Dump the nested Object state as a String

      time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: dump()

      time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[286] : Influenced by: org.eclipse.jetty.server.Handler

      time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[305] : Attribute Annotation found for: getServer

      time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineAttribute | msg=org.eclipse.jetty.jmx.ObjectMBean[657] : defineAttribute server false:true:class org.eclipse.jetty.server.handler.ErrorHandler:the jetty server for this handler

      time=2022-03-10T17:02:12.433Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineAttribute | msg=org.eclipse.jetty.jmx.ObjectMBean[729] : passed convert checks server for type class org.eclipse.jetty.server.Server

      time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=getMBeanInfo | msg=org.eclipse.jetty.jmx.ObjectMBean[318] : Method Annotation found for: destroy

      time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[809] : defineOperation destroy false:ACTION:destroy associated resources

      time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=defineOperation | msg=org.eclipse.jetty.jmx.ObjectMBean[863] : Method Cache: destroy()

      time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=beanAdded | msg=org.eclipse.jetty.jmx.MBeanContainer[208] : Registered org.eclipse.jetty.server.handler:type=errorhandler,id=0

      time=2022-03-10T17:02:12.434Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=addBean | msg=org.eclipse.jetty.util.component.ContainerLifeCycle[322] : org.eclipse.jetty.server.Server@52e18a5d added

      {org.eclipse.jetty.server.handler.ErrorHandler@4a257334,AUTO}

      time=2022-03-10T17:02:12.436Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=doStart | msg=org.eclipse.jetty.server.Server[372] : jetty-9.4.6.v20170531

      time=2022-03-10T17:02:12.471Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=doStart | msg=org.eclipse.jetty.server.handler.AbstractHandler[110] : starting org.eclipse.jetty.server.Server@52e18a5d

      time=2022-03-10T17:02:12.471Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=setStarting | msg=org.eclipse.jetty.util.component.AbstractLifeCycle[185] : starting qtp260686092

      {STOPPED,8<=0<=200,i=0,q=0}

      time=2022-03-10T17:02:12.488Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=setStarted | msg=org.eclipse.jetty.util.component.AbstractLifeCycle[177] : STARTED @1893ms qtp260686092

      {STARTED,8<=8<=200,i=7,q=0}

      time=2022-03-10T17:02:12.488Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=beanAdded | msg=org.eclipse.jetty.jmx.MBeanContainer[131] : beanAdded ServerConnector@cc1c486

      {HTTP/1.1,[http/1.1]}{0.0.0.0:5055}->org.eclipse.jetty.server.Server@52e18a5d

      time=2022-03-10T17:02:12.489Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=beanAdded | msg=org.eclipse.jetty.jmx.MBeanContainer[131] : beanAdded ServerConnector@cc1c486{HTTP/1.1,[http/1.1]} {0.0.0.0:5055}

      ->qtp260686092

      {STARTED,8<=8<=200,i=8,q=0}

      time=2022-03-10T17:02:12.489Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=beanAdded | msg=org.eclipse.jetty.jmx.MBeanContainer[131] : beanAdded ServerConnector@cc1c486

      {HTTP/1.1,[http/1.1]} {0.0.0.0:5055}

      ->org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@36176a5b

      ing | msg=org.eclipse.jetty.util.component.AbstractLifeCycle[185] : starting o.e.j.s.ServletContextHandler@30833221

      {/,null,UNAVAILABLE}

      time=2022-03-10T17:02:12.551Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=<clinit> | msg=org.eclipse.jetty.http.PreEncodedHttpField[64] : HttpField encoders loaded: [org.eclipse.jetty.http.Http1FieldPreEncoder@5b3a334b]

      Thanks in advance.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open In Progress In Progress
        23h 59m 1 Backlog Manager 12/Mar/22 6:31 AM
        In Progress In Progress Answered Answered
        1d 1 Backlog Manager 13/Mar/22 6:31 AM
        Answered Answered Closed Closed
        9d 2h 57m 1 Fernando Lopez 22/Mar/22 9:28 AM
        backlogmanager Backlog Manager made changes -
        Summary [fiware-stackoverflow] Cygnus does not persist data into PostSQL FIWARE.Question.Tech.Cygnus does not persist data into PostSQL.
        HD-Enabler Unknown [ 10910 ]
        HD-Chapter Unknown [ 10845 ]
        HD-Node Unknown [ 10852 ]
        fla Fernando Lopez made changes -
        Resolution Done [ 10000 ]
        Status Answered [ 10104 ] Closed [ 6 ]
        fla Fernando Lopez made changes -
        Assignee Backlog Manager [ backlogmanager ]
        Hide
        backlogmanager Backlog Manager added a comment -

        2022-03-13 05:31|UPDATED status: transition Answered| # answers= 1, accepted answer= False

        Show
        backlogmanager Backlog Manager added a comment - 2022-03-13 05:31|UPDATED status: transition Answered| # answers= 1, accepted answer= False
        backlogmanager Backlog Manager made changes -
        Status In Progress [ 3 ] Answered [ 10104 ]
        Hide
        backlogmanager Backlog Manager added a comment -

        2022-03-12 05:31|UPDATED status: transition Answer| # answers= 1, accepted answer= False

        Show
        backlogmanager Backlog Manager added a comment - 2022-03-12 05:31|UPDATED status: transition Answer| # answers= 1, accepted answer= False
        backlogmanager Backlog Manager made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        backlogmanager Backlog Manager made changes -
        HD-Enabler Unknown [ 10910 ]
        HD-Chapter Unknown [ 10845 ]
        HD-Node Unknown [ 10852 ]
        backlogmanager Backlog Manager made changes -
        Field Original Value New Value
        Component/s FIWARE-TECH-HELP [ 10278 ]
        Hide
        backlogmanager Backlog Manager added a comment -

        2022-03-11 05:31|CREATED monitor | # answers= 0, accepted answer= False

        Show
        backlogmanager Backlog Manager added a comment - 2022-03-11 05:31|CREATED monitor | # answers= 0, accepted answer= False
        backlogmanager Backlog Manager created issue -

          People

          • Assignee:
            backlogmanager Backlog Manager
            Reporter:
            backlogmanager Backlog Manager
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: