Created question in FIWARE Q/A platform on 16-07-2024 at 16:07
Please, ANSWER this question AT https://stackoverflow.com/questions/78755128/solved-fiware-data-loss-with-orion-ld-and-a-stress-test

Question:
SOLVED - Fiware - data loss with orion-ld and a stress test

Description:
During a series of tests we realised that we have a problem with notifications generated by subscriptions. In particular, not all changes correctly received by orion and made to the entities monitored by the subscriptions generate notifications to the downstream systems.
To identify the problem, the test environment was reduced to

orion-ld (both versions 1.5.1 and 1.6.0)
mongodb (both versions 3.3 and 4.4)
a fake http server that receives notifications
Load tests from github/fiware
The services run on the same machine using Docker containers.

During the test, 100 entities are created, a single subscription for all of them, and 5 updates with an interval of 1s. The test is run for both LD and V2. In both cases, a significant number of notifications are not sent to the fake server.
Analysing the logs with debug and trace enabled, we found that notifications are not sent when this condition occurs:
[1318]:addTriggeredSubscriptions_withCache | msg=Subscription: 66964843ed9b9fa7aa4d3bc8
[1319]:addTriggeredSubscriptions_withCache | msg=NOW: 1721124935.260880
[1320]:addTriggeredSubscriptions_withCache | msg=lastNotificationTime: 1721124935.265951
[1321]:addTriggeredSubscriptions_withCache | msg=DIFF: -0.005070
[1322]:addTriggeredSubscriptions_withCache | msg=throttling: 0.000000
[1323]:addTriggeredSubscriptions_withCache | msg=lastSuccess: 1721124935.265951
[1324]:addTriggeredSubscriptions_withCache | msg=lastFailure: 0.000000
[1329]:addTriggeredSubscriptions_withCache | msg=No notification due to throttling (last: 1721124935.260880 vs now: 1721124935.265951)

By enabling the -experimental flag on the same configuration, no notifications are lost on the LD side, but obviously the problem persists on the V2 side, which does not benefit from the flag changes.
Another test was done using fiware/orion instead of orion-ld and the V2 side showed no problems.
The issue appears even with relatively small numbers, such as 20 records submitted and 14 notified to the fake server ( this means 6 losts ), which makes us think there is something wrong with us that we cannot identify.
We need both parts, the V2 and the LD, this issue is pushing us to split services with the two different services, kind of an overkill solution.
Does anyone know what the problem could be or how to fix it?
During the test, 100 entities are created, a single subscription for all of them, and 5 updates with an interval of 1s.
the fake http server should receive all the changes.
Pushing a json file using the Apache benchmarking tool ( "ab" ) , using a sequential approach everything works fine.
Using concurrent ingestion ( i.e. 20 input with 10 requests in a time ) I observe only ( more or less ) 14 records notified to the fake server.
Other tests were done with other tools ( even with https://github.com/FIWARE/load-tests )

Solution:
As suggested here:
https://github.com/FIWARE/context.Orion-LD/issues/1650
a throttling value of -1 seems to bring Orion to ignore throttling settings for V2 and LD APIs.