Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-12626

FIWARE.Request.Tech.Unresolved question stackoverflow: IoT Agent LWM2M Memory exhaust and production environment.

    Details

    • Type: extRequest
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Fix Version/s: 2021
    • Component/s: FIWARE-TECH-HELP
    • Labels:
      None

      Description

      Hello. Can anyone give some insights to this question?

      IoT Agent LWM2M Memory exhaust and production environment <https://stackoverflow.com/questions/45172339/iot-agent-lwm2m-memory-exhaust-and-production-environment>

      I'm working on a project which aims to connect +1k devices to a machine, which may probably autoscale. The problem is that with only 2 connected devices, the IDAS IoTAgent ends in a couple of days with a "Memory exhaust" message. It is running in a docker instance in a machine in AWS m3.medium with memory swap.

      On the other hand, the git repository is not longer maintained as far as I can see, since there are not commits since Dec 16'. Is this a recomendable Generic Enabler for production? Has it been tested for a big amount of connected devices? Is this actually out of maintenance?

      Here it is the url to the git repo:

      https://github.com/telefonicaid/lightweightm2m-iotagent <https://github.com/telefonicaid/lightweightm2m-iotagent>
      Thanks in advance

      Confidentiality notice:
      This e-mail message, including any attachments, may contain legally privileged and/or confidential information. If you are not the intended recipient(s), or the employee or agent responsible for delivery of this message to the intended recipient(s), you are hereby notified that any dissemination, distribution, or copying of this e-mail message is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete this e-mail message from your computer.

      __________________________________________________________________________________________

      You can get more information about our cookies and privacy policies on the following links:

      Fiware-tech-help mailing list
      Fiware-tech-help@lists.fiware.org
      https://lists.fiware.org/listinfo/fiware-tech-help

      [Created via e-mail received from: Jose Benitez <jose@secmotic.com>]

      1. Jose3.png
        34 kB
      2. Jose3.png
        34 kB
      3. Jose3.png
        34 kB

        Activity

        Hide
        Stefano Stefano De Panfilis added a comment -

        dear jose,

        thank you for your interest in fiware and for having put this issue to our attention. let me answer to your questions one by one:
        1. stackoverflow.

        indeed the issue is related to a bug in a config.js IotAgent LWM2M.

        https://github.com/Fiware/iot.IoTagent-LWM2M/blob/master/docs/deviceProvisioning.mdhttps://github.com/Fiware/iot.IoTagent-LWM2M/blob/master/docs/deviceProvisioning.md

        The payload is malformed and it return the following error:

        op=IoTAgentNGSI.DbConn | msg=No host found for MongoDB driver.

        To solve it, you must create an attribute called: 'devicedRegitry', in the payload config.js file as follows:

        var config = {};

        config.lwm2m = {
        logLevel: 'DEBUG',
        port: 5684,
        defaultType: 'Device',
        ipProtocol: 'udp4',
        serverProtocol: 'udp4',
        /**

        • When a LWM2M client has active attributes, the IOTA sends an observe instruction for each one, just after the
        • client registers. This may cause cause an error when the client takes too long to start listening, as the
        • observe requests may not reach its destiny. This timeout (ms) is used to give the client the opportunity to
        • create the listener before the server sends the requests.
          */
          delayedObservationTimeout: 50,
          formats: [
          { name: 'application-vnd-oma-lwm2m/text', value: 1541 }

          ,

          { name: 'application-vnd-oma-lwm2m/tlv', value: 1542 }

          ,

          { name: 'application-vnd-oma-lwm2m/json', value: 1543 }

          ,

          { name: 'application-vnd-oma-lwm2m/opaque', value: 1544 }

          ],
          writeFormat: 'application-vnd-oma-lwm2m/text',
          types: [ ]
          };

        config.ngsi = {
        logLevel: 'DEBUG',
        contextBroker:

        { host: 'localhost', port: '1026' }

        ,
        server:

        { port: 4041, host: 'localhost' }

        ,
        deviceRegistry:

        { type: 'mongodb' }

        ,
        mongodb:

        { host:'172.17.0.2', db: 'iotagentLW2M2M' }

        ,
        types: { },
        service: 'smartGondor',
        subservice: '/gardens',
        providerUrl: 'http://localhost:4041',
        deviceRegistrationDuration: 'P1M'
        };

        module.exports = config;

        just let us know if this fixes your problem.

        2. "Is this a recommendable Generic Enabler for production?"
        absolutely yes (ref nex point)

        3. "Has it been tested for a big amount of connected devices?"
        yes and indeed we were a bit surprised by your situation indeed what we have through our quality assurance team is that In that they assessed that the IoT Agent was able to manage up to 140 updates per second, i.e., 140 requests from devices. We don’t have the number of devices as these updates may come from one or several devices. For 100 concurrent threads, it was able to manage in average 114 updates/sec. If we increment the number of concurrent threads, 140 updates/sec will be the cap. I hope this will give you a reference figure.
        just ket us know which are your requirements and we will come back to you with additional info.

        4. "Is this actually out of maintenance?"
        yes it is. the plan is to have a new version of the IoT Agent to be fully compliant with FIWARE NGSI v2. indeed, i guess you already spotted it the current version refers only to FIWARE NGSI v1. in addition new enhancements will appear soon. as you might know all the developments of fiware technologies are done by the fiware open source community and as any open source community we are committed to fix bugs, if any, and to provide enhancements, but this according to the priorities of the community members. as fiware foundation we have very limited resources but flexible enough to put them at disposal of things that we found critical (as this bug you found), but the large bunch of the work is done by the community.

        hope all of the above helps you in solving your issues and understanding the matter. serving people like you is our mission!

        ciao,
        stefano de panfilis
        chief operating officer
        fiware foundation

        Show
        Stefano Stefano De Panfilis added a comment - dear jose, thank you for your interest in fiware and for having put this issue to our attention. let me answer to your questions one by one: 1. stackoverflow. indeed the issue is related to a bug in a config.js IotAgent LWM2M. https://github.com/Fiware/iot.IoTagent-LWM2M/blob/master/docs/deviceProvisioning.mdhttps://github.com/Fiware/iot.IoTagent-LWM2M/blob/master/docs/deviceProvisioning.md The payload is malformed and it return the following error: op=IoTAgentNGSI.DbConn | msg=No host found for MongoDB driver. To solve it, you must create an attribute called: 'devicedRegitry', in the payload config.js file as follows: var config = {}; config.lwm2m = { logLevel: 'DEBUG', port: 5684, defaultType: 'Device', ipProtocol: 'udp4', serverProtocol: 'udp4', /** When a LWM2M client has active attributes, the IOTA sends an observe instruction for each one, just after the client registers. This may cause cause an error when the client takes too long to start listening, as the observe requests may not reach its destiny. This timeout (ms) is used to give the client the opportunity to create the listener before the server sends the requests. */ delayedObservationTimeout: 50, formats: [ { name: 'application-vnd-oma-lwm2m/text', value: 1541 } , { name: 'application-vnd-oma-lwm2m/tlv', value: 1542 } , { name: 'application-vnd-oma-lwm2m/json', value: 1543 } , { name: 'application-vnd-oma-lwm2m/opaque', value: 1544 } ], writeFormat: 'application-vnd-oma-lwm2m/text', types: [ ] }; config.ngsi = { logLevel: 'DEBUG', contextBroker: { host: 'localhost', port: '1026' } , server: { port: 4041, host: 'localhost' } , deviceRegistry: { type: 'mongodb' } , mongodb: { host:'172.17.0.2', db: 'iotagentLW2M2M' } , types: { }, service: 'smartGondor', subservice: '/gardens', providerUrl: 'http://localhost:4041', deviceRegistrationDuration: 'P1M' }; module.exports = config; just let us know if this fixes your problem. 2. "Is this a recommendable Generic Enabler for production?" absolutely yes (ref nex point) 3. "Has it been tested for a big amount of connected devices?" yes and indeed we were a bit surprised by your situation indeed what we have through our quality assurance team is that In that they assessed that the IoT Agent was able to manage up to 140 updates per second, i.e., 140 requests from devices. We don’t have the number of devices as these updates may come from one or several devices. For 100 concurrent threads, it was able to manage in average 114 updates/sec. If we increment the number of concurrent threads, 140 updates/sec will be the cap. I hope this will give you a reference figure. just ket us know which are your requirements and we will come back to you with additional info. 4. "Is this actually out of maintenance?" yes it is. the plan is to have a new version of the IoT Agent to be fully compliant with FIWARE NGSI v2. indeed, i guess you already spotted it the current version refers only to FIWARE NGSI v1. in addition new enhancements will appear soon. as you might know all the developments of fiware technologies are done by the fiware open source community and as any open source community we are committed to fix bugs, if any, and to provide enhancements, but this according to the priorities of the community members. as fiware foundation we have very limited resources but flexible enough to put them at disposal of things that we found critical (as this bug you found), but the large bunch of the work is done by the community. hope all of the above helps you in solving your issues and understanding the matter. serving people like you is our mission! ciao, stefano de panfilis chief operating officer fiware foundation
        Hide
        fw.ext.user FW External User added a comment -

        Comment by jose@secmotic.com :

        Hello Stefano, thanks for your response!

        > 1. stackoverflow.

        > To solve it, you must create an attribute called: 'devicedRegitry', in the payload config.js file as follows:
        > just let us know if this fixes your problem.

        We noticed this and added the device registry and changed to mongodb, which did resolve the first error (memory exhaust).
        But we are now facing a CPU error, which I'm describing now:

        We are sending 1 request per second approximately to 200 devices, which shouldn't be an issue, but the CPU goes up to 100% and the processed is killed.

        Our guess is that our request is too big compared to your tests, so we cannot reach your 140 requests per second.
        Could you please send me that request or test?
        That would be very helpful

        Thanks in advance, for your response and for the great work you are doing in the fiware community

        Confidentiality notice:
        This e-mail message, including any attachments, may contain legally privileged and/or confidential information. If you are not the intended recipient(s), or the employee or agent responsible for delivery of this message to the intended recipient(s), you are hereby notified that any dissemination, distribution, or copying of this e-mail message is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete this e-mail message from your computer.

        > El 25 ago 2017, a las 9:02, Help-Desk <jira-help-desk@jira.fiware.org> escribió:
        >
        >
        >

        Show
        fw.ext.user FW External User added a comment - Comment by jose@secmotic.com : Hello Stefano, thanks for your response! > 1. stackoverflow. > To solve it, you must create an attribute called: 'devicedRegitry', in the payload config.js file as follows: > just let us know if this fixes your problem. We noticed this and added the device registry and changed to mongodb, which did resolve the first error (memory exhaust). But we are now facing a CPU error, which I'm describing now: We are sending 1 request per second approximately to 200 devices, which shouldn't be an issue, but the CPU goes up to 100% and the processed is killed. Our guess is that our request is too big compared to your tests, so we cannot reach your 140 requests per second. Could you please send me that request or test? That would be very helpful Thanks in advance, for your response and for the great work you are doing in the fiware community – Confidentiality notice: This e-mail message, including any attachments, may contain legally privileged and/or confidential information. If you are not the intended recipient(s), or the employee or agent responsible for delivery of this message to the intended recipient(s), you are hereby notified that any dissemination, distribution, or copying of this e-mail message is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete this e-mail message from your computer. > El 25 ago 2017, a las 9:02, Help-Desk <jira-help-desk@jira.fiware.org> escribió: > > >
        Hide
        Stefano Stefano De Panfilis added a comment -

        dear jose,

        may be the case is that your request is larger in size.
        anyhow, as a first step i'm forwarding you the test report our qa team did on idas. this way you can eventually replicate the tests and/or we have to make a different test.

        here is the link to the report: https://forge.fiware.org/docman/view.php/7/5836/FIWARE-GEri_test_IDAS_5.3.0.pdf

        thank you very much for your help on the matter.

        ciao,
        stefano

        Show
        Stefano Stefano De Panfilis added a comment - dear jose, may be the case is that your request is larger in size. anyhow, as a first step i'm forwarding you the test report our qa team did on idas. this way you can eventually replicate the tests and/or we have to make a different test. here is the link to the report: https://forge.fiware.org/docman/view.php/7/5836/FIWARE-GEri_test_IDAS_5.3.0.pdf thank you very much for your help on the matter. ciao, stefano
        Hide
        fw.ext.user FW External User added a comment -

        Comment by jose@secmotic.com :

        Thanks a lot Stefano,

        Just one thing, the GE tested is IDAS Iot Agent, but it doesn't specify which one of the Agents have been tested.

        These results seems pretty good and they are similar to our results for the Ultralight 2.0 agent, but we are facing a complete different scenario with the LWM2M IoT Agent (in its last version).
        The CPU in the LWM2M reaches the 100% of usage for the same inputs that in the report (but with larger json in the requests).

        Is it possible that the results are different from one IoT Agent to another?

        Best regards

        Confidentiality notice:
        This e-mail message, including any attachments, may contain legally privileged and/or confidential information. If you are not the intended recipient(s), or the employee or agent responsible for delivery of this message to the intended recipient(s), you are hereby notified that any dissemination, distribution, or copying of this e-mail message is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete this e-mail message from your computer.

        > El 31 ago 2017, a las 15:13, Help-Desk <jira-help-desk@jira.fiware.org> escribió:
        >
        >
        >

        Show
        fw.ext.user FW External User added a comment - Comment by jose@secmotic.com : Thanks a lot Stefano, Just one thing, the GE tested is IDAS Iot Agent, but it doesn't specify which one of the Agents have been tested. These results seems pretty good and they are similar to our results for the Ultralight 2.0 agent, but we are facing a complete different scenario with the LWM2M IoT Agent (in its last version). The CPU in the LWM2M reaches the 100% of usage for the same inputs that in the report (but with larger json in the requests). Is it possible that the results are different from one IoT Agent to another? Best regards – Confidentiality notice: This e-mail message, including any attachments, may contain legally privileged and/or confidential information. If you are not the intended recipient(s), or the employee or agent responsible for delivery of this message to the intended recipient(s), you are hereby notified that any dissemination, distribution, or copying of this e-mail message is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete this e-mail message from your computer. > El 31 ago 2017, a las 15:13, Help-Desk <jira-help-desk@jira.fiware.org> escribió: > > >
        Hide
        Stefano Stefano De Panfilis added a comment -

        dear jose,

        for whatever reasons i missed to finalise our conversation and fixing this important aspect.
        i'm now checking with the fiware qa team if this situation still holds.

        anything else i shall report to them from your side?

        ciao,
        stefano

        Show
        Stefano Stefano De Panfilis added a comment - dear jose, for whatever reasons i missed to finalise our conversation and fixing this important aspect. i'm now checking with the fiware qa team if this situation still holds. anything else i shall report to them from your side? ciao, stefano

          People

          • Assignee:
            Stefano Stefano De Panfilis
            Reporter:
            fw.ext.user FW External User
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: