FW External User added a comment - 18/Nov/22 6:59 PM Hi, For our platform DYDAS, developed with a CEF co-founded project, we would need some support as indicated in my colleague mail below, regarding the compliance of our integration of the Context Broker, which is required by CINEA. We had some difficulties in succeeding in passing all tests, whereas we followed the instructions by the letter, we couldn’t pass all compliance. Can you support us with the tests or help us achieve compliance? We used the Context Broker for some time without problem, but for the completion of the project we need the compliance report. Detailed information is indicated in the mail below. Thank you Regards Iulian Gabriel Coltea R&D Manager KEY TO BUSINESS S.R.L. [cid:image009.png@01D7C061.3BA7C390] +393480936357 | +390698358320 [cid:image010.png@01D7C061.3BA7C390] i.coltea@key2.it< i.coltea@key2.it > [cid:image011.png@01D7C061.3BA7C390] https://www.key2.it/ < https://www.key2.it/it/ > [cid:image012.png@01D7C061.3BA7C390] Viale dell'Esperanto, 71, 00144 Roma (RM) Via Lunigiana 40, 20125 Milano (MI) SS 17 BIS SNC, 67100 L'Aquila (AQ) [cid:image013.png@01D7C061.3BA7C390] [cid:image014.png@01D7C061.3BA7C390] < https://www.instagram.com/k2business/?hl=it > [cid:image015.png@01D7C061.3BA7C390] < https://www.facebook.com/key2.it > [cid:image016.png@01D7C061.3BA7C390] < https://www.linkedin.com/company/key-2/ > Da: Elena Fulvio< e.fulvio@key2.it > Inviato: giovedì 10 novembre 2022 11:15 A: fiware-tech-help@lists.fiware.org< fiware-tech-help@lists.fiware.org > Cc: Iulian Gabriel Coltea< i.coltea@key2.it > Oggetto: CEF funded project compliance tests Good evening, we are trying to run the conformance test for the Orion Context Broker that is included in a project funded by CEF, but we are facing some difficulties. We are running the Orion Context Broker via Kubernetes. The context broker was installed by helm (following the instruction at https://github.com/FIWARE/helm-charts < https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FFIWARE%2Fhelm-charts&data=05%7C01%7Ci.coltea%40key2.it%7Ce4eacb40f6b047f5f48108dac30470d6%7C8c586c92cabd4ce5983abe9869cdc625%7C0%7C0%7C638036721083018882%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=917TiVHmjkpNEaOeKj3OYv4mZm%2BJAudIHPq2J6dDpc0%3D&reserved=0 > ), resulting in a container based on the image quay.io/fiware/orion-ld:1.1.0 (for reference the mongodb image, installed along via the helm chart is quay.io/opencloudio/ibm-mongodb:4.0.24). The containers are up and running and we are able to successfully execute the REST apis described in https://fiware-orion.readthedocs.io/en/master/ < https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffiware-orion.readthedocs.io%2Fen%2Fmaster%2F&data=05%7C01%7Ci.coltea%40key2.it%7Ce4eacb40f6b047f5f48108dac30470d6%7C8c586c92cabd4ce5983abe9869cdc625%7C0%7C0%7C638036721083018882%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=w6Vjjb2RB%2B2OlrYyl%2FBGEcu0Wq%2FWiv2dGG7vNHHaXgE%3D&reserved=0 > , however the test suites we are running seem to fail. We found two different test suites, the first one (NGSI-LD Test Suite) which is linked in CEF digital page https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/Conformance+testing < https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fec.europa.eu%2Fcefdigital%2Fwiki%2Fdisplay%2FCEFDIGITAL%2FConformance%2Btesting&data=05%7C01%7Ci.coltea%40key2.it%7Ce4eacb40f6b047f5f48108dac30470d6%7C8c586c92cabd4ce5983abe9869cdc625%7C0%7C0%7C638036721083018882%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yMUQyroRE6RVLmNyZHuuygLakP1B56G1%2B7JF27bgZ6Q%3D&reserved=0 > and the second one (ETSI NGSI-LD API Conformance Test Suite) on the FIWARE github page. https://ec.europa.eu/cefdigital/wiki/cefdigital/wiki/rest/documentConversion/latest/conversion/thumbnail/356876566/1 < https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fec.europa.eu%2Fcefdigital%2Fwiki%2Fdisplay%2FCEFDIGITAL%2FConformance%2Btesting&data=05%7C01%7Ci.coltea%40key2.it%7Ce4eacb40f6b047f5f48108dac30470d6%7C8c586c92cabd4ce5983abe9869cdc625%7C0%7C0%7C638036721083018882%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yMUQyroRE6RVLmNyZHuuygLakP1B56G1%2B7JF27bgZ6Q%3D&reserved=0 > Conformance testing - ec.europa.eu< https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fec.europa.eu%2Fcefdigital%2Fwiki%2Fdisplay%2FCEFDIGITAL%2FConformance%2Btesting&data=05%7C01%7Ci.coltea%40key2.it%7Ce4eacb40f6b047f5f48108dac30470d6%7C8c586c92cabd4ce5983abe9869cdc625%7C0%7C0%7C638036721083018882%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yMUQyroRE6RVLmNyZHuuygLakP1B56G1%2B7JF27bgZ6Q%3D&reserved=0 > CEF Context Broker offers two different types of testing services: ec.europa.eu In spite of the fact that the first one seems to be deprecated and the second one is still under development we tried both of them: [cid:image001.png@01D8FB7F.AF7701A0]       NGSI-LD Test Suite:             We downloaded the test suite from : https://github.com/FIWARE/NGSI-LD_TestSuite < https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FFIWARE%2FNGSI-LD_TestSuite&data=05%7C01%7Ci.coltea%40key2.it%7Ce4eacb40f6b047f5f48108dac30470d6%7C8c586c92cabd4ce5983abe9869cdc625%7C0%7C0%7C638036721083175990%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bgN%2BIdYuKU92YV5ppBnQ9kOEGiO3qtloUsS%2B5UiVtVU%3D&reserved=0 >             The first test was directed to the helm instance I mentioned before and resulted in :             Test Suites: 15 failed, 22 passed, 37 total             Tests: 49 failed, 120 passed, 169 total             Snapshots: 0 total             Time: 117.426s, estimated 118s            which is odd, considering the fact that the Orion Context Broker instance we are testing is the default image provided by the official helm chart (the complete logs are available in the attachment OUTPUT_NGIS-LD_helm.txt).            We also tried to run the same test suite using the docker containers that are provided in the suite itself (under /docker-compose/orion.yml) and it resulted in:            Test Suites: 11 failed, 26 passed, 37 total             Tests: 39 failed, 130 passed, 169 total             Snapshots: 0 total             Time: 102.645s             which is extremely odd since in this case Orion-LD, mongodb, accumulator server & notification server are all provided by the suite itself .       ETSI Test Suite:             We downloaded the test suite from: https://forge.etsi.org/rep/cim/ngsi-ld-test-suite/-/tree/develop since the master branch is still empty, and we tried to test the helm instance again.             We were actually able to launch part of the tests but the testing process seems to freeze at some point during the execution. Considering all of the above, how we can achieve the CEF compliance for the FIWARE Orion Context broker implementation? Are those tests sufficient to be compliant? Thank you for your time and sorry for the long email. Best regards, Elena Fulvio Solution Architect KEY TO BUSINESS S.R.L. https://www.key2.it/img/loghi_mail/Email.png e.fulvio@key2.it< e.fulvio@key2.it > https://www.key2.it/img/loghi_mail/Sito%20web.png https://www.key2.it/ < https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.key2.it%2Fit%2F&data=05%7C01%7Ci.coltea%40key2.it%7Ce4eacb40f6b047f5f48108dac30470d6%7C8c586c92cabd4ce5983abe9869cdc625%7C0%7C0%7C638036721083175990%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=W3E2FM6%2Ff6tAoONNKGjno0el54Q%2FeUx9DPggWtvfhn0%3D&reserved=0 > https://www.key2.it/img/loghi_mail/Indirizzo.png Viale dell'Esperanto, 71, 00144 Roma (RM) Via Lunigiana 40, 20125 Milano (MI) SS 17 BIS SNC, 67100 L'Aquila (AQ) https://www.key2.it/img/loghi_mail/logo_k2_blue_xxs.png https://www.key2.it/img/loghi_mail/Instagram.png < https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fk2business%2F%3Fhl%3Dit&data=05%7C01%7Ci.coltea%40key2.it%7Ce4eacb40f6b047f5f48108dac30470d6%7C8c586c92cabd4ce5983abe9869cdc625%7C0%7C0%7C638036721083175990%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=59XnL5%2FMgwRJNqYhV%2BdSs4%2B4wqxeqhWRWTVodnV%2F6pg%3D&reserved=0 > https://www.key2.it/img/loghi_mail/Facebook.png < https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fkey2.it&data=05%7C01%7Ci.coltea%40key2.it%7Ce4eacb40f6b047f5f48108dac30470d6%7C8c586c92cabd4ce5983abe9869cdc625%7C0%7C0%7C638036721083175990%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oFeRQmGMC%2FXn49uL5iHeQu00%2BTDHSI4yvG%2FE56F%2BZnc%3D&reserved=0 > https://www.key2.it/img/loghi_mail/Linkedin.png < https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fkey-2%2F&data=05%7C01%7Ci.coltea%40key2.it%7Ce4eacb40f6b047f5f48108dac30470d6%7C8c586c92cabd4ce5983abe9869cdc625%7C0%7C0%7C638036721083175990%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=waQFQ0xYhs2XwMcUlYnfQpozRmPSMScFIY1AWhkJdnk%3D&reserved=0 >

Ken Zangelin added a comment - 22/Nov/22 3:39 PM - edited It seems like we have some kind of mix between Orion (ngsiv2) and Orion-LD (NGSI-LD + outdated ngsiv2). I assume you want to test Orion-LD, as that's my responsibility while Orion is Telefonicas. Now, the test suite used for Orion-LD (and for Orion as well), is part of the repository of the broker. The test cases are found under the directory test/functionalTest/cases . I'm happy to assist you in making the test suite work. This test suite is started as part of any PR, in github actions, and no PR is ever merged unless the entire test suite works to 100% Well, unless it's a PR strictly about documentation and there's some glitch in the test suite - that would be the exception. About the other two test suites ... The first one was discontinued about 3 years ago. At that time, Orion-LD had a ~98% success rate with it (a few tests were about things I still hadn't implemented back then). I know that NEC have put some effort in to improve the test suite, adding test cases, possible cases on things that Orion-LD does not support. Anyway, it's the old inoffical test suite that kind of died around 3 years ago. I'm not using it and neither should you. The second test suite was a TTF in ETSI ISG CIM and there was not funding enough to actually produce anything useful. We (ETSI ISG CIM) are applying for another round of TTF funding from ETSI and with that, hopefully we can fix the test suite so it can actually be used. The way the test suite is right now, it is 100% unusable. It's full of bugs and if a test fails, there is next to no information about what exactly went wrong. I did a test with it once, over a year ago. Took me 35 minutes to figure out what went wrong in the first error reported. In the end it turned out to be a bug in the test suite. NEC has spent more time on the suite and they are of the exact same opinion. The test suite needs to be fixed before it can be used. This leaves you with the test suite of Orion-LD's repository and, as I said, I'd be happy to help you get it running. You'll need to compile the broker (and run it) locally, and you also need mongodb to run locally. No support for containers here, sorry. It's something I'll add, some day. Have a look at the README.md of the repo and ... follow the installation instructions