Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-20975

[fiware-stackoverflow] Data loss when using Fiware Orion Broker, QuantumLeap and CrateDB

    Details

      Description

      Created question in FIWARE Q/A platform on 15-03-2023 at 18:03
      Please, ANSWER this question AT https://stackoverflow.com/questions/75748890/data-loss-when-using-fiware-orion-broker-quantumleap-and-cratedb

      Question:
      Data loss when using Fiware Orion Broker, QuantumLeap and CrateDB

      Description:
      I'm using Fiware Orion Broker, QuantumLeap and CrateDB, with the aim of recording all temporal data in cratedb.
      My docker-compose configuration is this:
      orion:
      image: fiware/orion:$

      {ORION_VERSION}

      hostname: orion
      container_name: fiware-orion
      depends_on:

      • mongo-db
        networks:
      • fiware
        expose:
      • "$ {ORION_PORT}"
        ports:
        - "${ORION_PORT}

        :$

        {ORION_PORT}"
        command: -dbhost mongo-db
        healthcheck:
        test: curl --fail -s http://orion:${ORION_PORT}

        /version || exit 1
        interval: 5s

      mongo-db:
      image: mongo:latest
      hostname: mongo-db
      container_name: db-mongo
      expose:

      • "$ {MONGO_DB_PORT}"
        ports:
        - "${MONGO_DB_PORT}

        :$

        {MONGO_DB_PORT}

        "
        networks:

      • fiware
        volumes:
      • ./volumes/mongo-db:/data
        healthcheck:
        test: |
        host=`hostname --ip-address || echo '127.0.0.1'`;
        mongo --quiet $host/test --eval 'quit(db.runCommand( { ping: 1 }

        ).ok ? 0 : 2)' && echo 0 || echo 1
        interval: 5s

      quantumleap:
      image: orchestracities/quantumleap:latest
      hostname: quantumleap
      container_name: fiware-quantumleap
      ports:

      • "$ {QUANTUMLEAP_PORT}:${QUANTUMLEAP_PORT}

        "
        depends_on:

      • crate-db
      • redis-db
        environment:
      • CRATE_HOST=crate-db
      • LOGLEVEL=WARNING
        healthcheck:
        test: curl --fail -s http://quantumleap:$ {QUANTUMLEAP_PORT}

        /version || exit 1
        networks:

      • fiware

      crate-db:
      image: crate:latest
      hostname: crate-db
      container_name: db-crate
      ports:

      • "4200:4200"
      • "4300:4300"
        command: crate -Cauth.host_based.enabled=false -Ccluster.name=democluster -Chttp.cors.enabled=true -Chttp.cors.allow-origin="*" -Cdiscovery.type=single-node
        environment:
      • CRATE_HEAP_SIZE=2g
        volumes:
      • ./volumes/crate-db:/data
        networks:
      • fiware

      I'm running performance tests through Apache JMeter, and consecutive requests are sent for 1 minutes to evaluate their performance, but I'm having a problem where some data is not being registered in the CrateDB, that is, in the last test I did, it was done about 18000 requests, and in CrateDB only about 10000 are registered.
      I also tried using the TimescaleDB database in QuantumLeap, but the same problem happens, so I assume that the problem is not with the database.
      Does anyone know what the problem could be?

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open In Progress In Progress
        41d 3h 44m 1 Ken Zangelin 01/May/23 11:15 AM
        In Progress In Progress Answered Answered
        18s 1 Ken Zangelin 01/May/23 11:16 AM

          People

          • Assignee:
            kzangeli Ken Zangelin
            Reporter:
            backlogmanager Backlog Manager
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated: