Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-5820

FIWARE.Request.Tech.Data.CEP.Proton load issue

        Details

        • Type: extRequest
        • Status: Closed
        • Priority: Major
        • Resolution: Done
        • Fix Version/s: 2021
        • Component/s: FIWARE-TECH-HELP
        • Labels:
          None

          Description

          Hi,

          I am forwarding this message (as suggested) to report a Proton issue.

          Thanks,

          MarcoF

          ---------- Forwarded message ----------
          From: Fabio Lanari (Kiunsys) <fabio.lanari@kiunsys.com>
          Date: 2 February 2016 at 08:51
          Subject: Fwd: RE: Opportunities for dissemination in FIWARE events
          To: Marco Fagiani <m.fagiani85@gmail.com>
          Cc: Fabio Formosa <f.formosa@liberologico.com>

          La risposta di Telefonica.

          -------- Messaggio Inoltrato --------
          Oggetto: RE: Opportunities for dissemination in FIWARE events
          Data: Mon, 1 Feb 2016 22:54:32 +0000
          Mittente: SERGIO GARCIA GOMEZ <sergio.garciagomez@telefonica.com>
          <sergio.garciagomez@telefonica.com>
          A: Fabio Lanari (Kiunsys) <fabio.lanari@kiunsys.com>
          <fabio.lanari@kiunsys.com>, JOSE MANUEL CANTERA FONSECA
          <josemanuel.canterafonseca@telefonica.com>
          <josemanuel.canterafonseca@telefonica.com>, Paola Ponticelli
          <paola.ponticelli@kiunsys.com> <paola.ponticelli@kiunsys.com>
          CC: SANTIAGO MARTINEZ GARCIA <santiago.martinezgarcia@telefonica.com>
          <santiago.martinezgarcia@telefonica.com>

          Dear Fabio,

          The best way to handle this is to forward this message to
          fiware-tech-help@lists.fiware.org

          It’s better if you do it because you will get the email notifications on
          changes and answers. We will notify the Proton team anyway.

          Best Regards,

          Sergio.

          De: Fabio Lanari (Kiunsys) [mailto:fabio.lanari@kiunsys.com
          <fabio.lanari@kiunsys.com>]
          Enviado el: 01 February 2016 10:24
          Para: JOSE MANUEL CANTERA FONSECA
          <josemanuel.canterafonseca@telefonica.com>
          <josemanuel.canterafonseca@telefonica.com>; Paola Ponticelli
          <paola.ponticelli@kiunsys.com> <paola.ponticelli@kiunsys.com>
          CC: SERGIO GARCIA GOMEZ <sergio.garciagomez@telefonica.com>
          <sergio.garciagomez@telefonica.com>
          Asunto: Re: Opportunities for dissemination in FIWARE events

          Hi Sergio,
          we have used these few days for performing one more test (VM scaling up).

          The main problem, as I said, is that Proton is consuming too much HW
          resources while managing only a bunch of alerting rules, which produces or
          the app crash or requires the VM restart.

          We have tested the entire system in 2 different IaaS installations.

          a) GEs used: IDAS IoT Agent 1.3.1. + ORION 0.26.1 + PROTON CEP 3.3.1
          b) Installation type: local installation on a IaaS provider
          c) Installation method: Docker + Docker Compose

          Installations specs:

          • 1st installation: 1 VM (14 GB RAM, 4 core) containing the 3
            GEs
            <-- in this case we haven' seen any problems. It should be said that this
            was an initial installation where we implemented only 2 or 3 Proton rules
          • 2nd installation - small size (due to IaaS provider migration): 1 VM (4
            GB RAM, 2 core) containig the 3 GEs <-- in this case Proton eats up to 1.2
            GB of RAM and constantly uses 98% of 2 cores (with 8 Proton rules)
          • 2nd installation - mid-size (scaling up of the previous VM): 1 VM (8 GB
            RAM, 4 core) containing the 3 GEs <-- Proton still eats up to 1.2 GB of
            RAM and constantly uses 3 cores at 100% (with 8 Proton rules)

          We don't know where the problem is.
          Our dev team thought there was too much messages between Orion and Proton,
          that Proton is unable to manage, so they are looking for a workaround, a
          method to reduce them. Here is the link to our bug/request:

          http://stackoverflow.com/questions/34491828/fiware-orion-context-broker-metadata-updates-trigger-notifications

          Thank you for your help

          Fabio L.

          -------- Messaggio originale --------
          Oggetto: Re: Opportunities for dissemination in FIWARE events
          Mittente: JOSE MANUEL CANTERA FONSECA
          <josemanuel.canterafonseca@telefonica.com>
          <josemanuel.canterafonseca@telefonica.com>
          <josemanuel.canterafonseca@telefonica.com>
          A: Paola Ponticelli <paola.ponticelli@kiunsys.com>
          <paola.ponticelli@kiunsys.com>
          Data: 29/1/2016 10:13:56

          Fabio, Paola,

          Please don’t forget to send us the ticket and some description about the
          problems found on your integration.

          Thanks, best

          Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost.
          Please, send your messages using the new domain (Fiware-tech-help@lists.fiware.org) instead of the old one.
          _______________________________________________
          Fiware-tech-help mailing list
          Fiware-tech-help@lists.fiware.org
          https://lists.fiware.org/listinfo/fiware-tech-help
          [Created via e-mail received from: Marco Fagiani <m.fagiani85@gmail.com>]

          1. File
            INES.json
            21 kB
            FW External User

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            TALI Fabiana Fournier added a comment -

            The CEP load is determined by several factors:
            1. The incoming events rate
            2. The output event rate
            3. The number of rules
            4. The number of open processing contexts that are open in parallel during a run
            5. The complexity of the rules evaluation

            I suspect that the issue here relates to #4, but in order to analyse it I need more data:
            1. The CEP definition file (can be exported from the Authoring Tool)
            2. The input events that were processed by the CEP and their approximate volume per second

            Thanks

            Show
            TALI Fabiana Fournier added a comment - The CEP load is determined by several factors: 1. The incoming events rate 2. The output event rate 3. The number of rules 4. The number of open processing contexts that are open in parallel during a run 5. The complexity of the rules evaluation I suspect that the issue here relates to #4, but in order to analyse it I need more data: 1. The CEP definition file (can be exported from the Authoring Tool) 2. The input events that were processed by the CEP and their approximate volume per second Thanks
            Hide
            Permalink
            TALI Fabiana Fournier added a comment -

            The CEP load is determined by several factors:
            1. The incoming events rate
            2. The output event rate
            3. The number of rules
            4. The number of open processing contexts that are open in parallel during a run
            5. The complexity of the rules evaluation
            I suspect that the issue here relates to #4, but in order to analyse it I need more data:
            1. The CEP definition file (can be exported from the Authoring Tool)
            2. The input events that were processed by the CEP and their approximate volume per second
            Thanks,
            Tali

            Show
            TALI Fabiana Fournier added a comment - The CEP load is determined by several factors: 1. The incoming events rate 2. The output event rate 3. The number of rules 4. The number of open processing contexts that are open in parallel during a run 5. The complexity of the rules evaluation I suspect that the issue here relates to #4, but in order to analyse it I need more data: 1. The CEP definition file (can be exported from the Authoring Tool) 2. The input events that were processed by the CEP and their approximate volume per second Thanks, Tali
            Hide
            Permalink
            fw.ext.user FW External User added a comment -

            Hi,

            Two events (PSSContextUpdate and DCUContextUpdate) are defined, with the
            same 11 attributes, in addition to the entityId and entityType.
            Each minute, about 3 DCU events, and 73 PSS events are received.
            Please find attached the adopted definition file.

            Thanks,

            MarcoF

            Show
            fw.ext.user FW External User added a comment - Hi, Two events (PSSContextUpdate and DCUContextUpdate) are defined, with the same 11 attributes, in addition to the entityId and entityType. Each minute, about 3 DCU events, and 73 PSS events are received. Please find attached the adopted definition file. Thanks, MarcoF
            Hide
            Permalink
            TALI Fabiana Fournier added a comment -

            Thank you Marco for the details.

            An initial investigation of your application definition didn't reveal why the CEP uses a lot of resources (all contexts but one uses "ignore" type initiator, one uses "add" but it is reported to have low frequency of about 3 events per minute, and it has relative 30 minutes terminator).

            I open a bug ticket to have our developer investigate it:
            FIWARE.Bug.Data.CEP.HeavyLoad.Analyze
            https://jira.fiware.org/browse/DATA-1324

            Thanks,
            Tali

            Show
            TALI Fabiana Fournier added a comment - Thank you Marco for the details. An initial investigation of your application definition didn't reveal why the CEP uses a lot of resources (all contexts but one uses "ignore" type initiator, one uses "add" but it is reported to have low frequency of about 3 events per minute, and it has relative 30 minutes terminator). I open a bug ticket to have our developer investigate it: FIWARE.Bug.Data.CEP.HeavyLoad.Analyze https://jira.fiware.org/browse/DATA-1324 Thanks, Tali

              People

              • Assignee:
                TALI Fabiana Fournier
                Reporter:
                fw.ext.user FW External User
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: