Created question in FIWARE Q/A platform on 10-03-2016 at 16:03
Please, ANSWER this question AT https://stackoverflow.com/questions/35921306/fiware-cygnus-issue-with-collections-names

Question:
Fiware - Cygnus: Issue with collection's names

Description:
I have encountered a strange behaviour on the Cygnus module. I’m using Context Broker version 0.28 and Cygnus version 0.13. Let’s suppose that I have loaded on the CB (Context Broker) some entities similars to this one:

(curl 172.21.0.33:1026/v1/updateContext -s -S
--header 'Fiware-Service: cb33_1003_06'
--header 'Fiware-ServicePath: /Lugar'
--header 'Content-Type: application/json'
-header 'Accept: application/json' -d @ | python -mjson.tool) <<EOF

{
"contextElements": [
{
"type": "Acceso_Wifi",
"isPattern": "false",
"id": "AP_1.3",
"attributes": [

,

,
{
"name": "position",
"type": "coords",
"value": "13.322326, -1.983824",
"metadatas": [

]
}
]
}
],
"updateAction": "APPEND"
}

EOF

And now I subscribe the Cygnus:

(curl 172.21.0.33:1026/v1/subscribeContext -s -S
--header 'Fiware-Service: cb33_1003_06'
--header 'Fiware-ServicePath: /Lugar'
--header 'Content-Type: application/json'
-header 'Accept: application/json' -d @ | python -mjson.tool) <<EOF
{
"entities": [

],
"attributes": [
"temperature",
"pressure"
],
"reference": "http://172.21.0.33:5050/notify",
"duration": "P1M",
"notifyConditions": [

],
"throttling": "PT1S"
}
EOF

The CB will send one notification to the Cygnus for each matching entity loaded previously on the CB database. Let’s assume that there are 3 entities that meet the subscription’s criteria. In that case, one collection will be created on the Cygnus for each one of these entities (I’m using the default dm-by-entity data model). The problem is that the collection's name are not well formed. The Fiware-ServicePath is concatenated once per each entity, on each collection name:

cygnus2_/Lugar_Lugar_Lugar_AP_1.1_Acceso_Wifi
cygnus2_/Lugar_Lugar_Lugar_AP_1.2_Acceso_Wifi
cygnus2_/Lugar_Lugar_Lugar_AP_1.3_Acceso_Wifi

If we update one of these entities, a new collection will be created with a correct name:

cygnus2_/Lugar_AP_1.1_Acceso_Wifi
cygnus2_/Lugar_Lugar_Lugar_AP_1.1_Acceso_Wifi
cygnus2_/Lugar_Lugar_Lugar_AP_1.2_Acceso_Wifi
cygnus2_/Lugar_Lugar_Lugar_AP_1.3_Acceso_Wifi

On this example there were just 3 entities, but as the amount of matching entities increases, the longer will be the collection's name. And when it arrives to 50 characters, a warning will be raised and no collection will be created.

org.apache.flume.source.http.HTTPBadRequestException: 'fiware-servicePath' header length greater than 50)
at com.telefonica.iot.cygnus.handlers.OrionRestHandler.getEvents(OrionRestHandler.java:209)
at org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:184)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

¿It’s that behaviour due to an incorrect configuration, or it’s Cygnus’s problem?

Here is the OrionMongoSink's configuration:

1. OrionMongoSink configuration
2. sink class, must not be changed
   cygnusagent.sinks.mongo-sink.type = com.telefonica.iot.cygnus.sinks.OrionMongoSink
3. channel name from where to read notification events
   cygnusagent.sinks.mongo-sink.channel = mongo-channel
   # true if the grouping feature is enabled for this sink, false otherwise
   cygnusagent.sinks.mongo-sink.enable_grouping = false
4. true if lower case is wanted to forced in all the element names, false otherwise
   cygnusagent.sinks.mongo-sink.enable_lowercase = false
5. FQDN/IP:port where the MongoDB server runs (standalone case) or comma-separated list of FQDN/IP:port pairs where the MongoDB replica set members run
   cygnusagent.sinks.mongo-sink.mongo_hosts = mongodb1.spider.net:27017,mongodb2.spider.net:27017,mongodb3.spider.net:27017
   ###cygnusagent.sinks.mongo-sink.mongo_hosts = 172.21.0.25:27017,172.21.0.26:27017,172.21.0.28:27017
6. a valid user in the MongoDB server (or empty if authentication is not enabled in MongoDB)
   cygnusagent.sinks.mongo-sink.mongo_username =
7. password for the user above (or empty if authentication is not enabled in MongoDB)
   cygnusagent.sinks.mongo-sink.mongo_password =
8. prefix for the MongoDB databases
   cygnusagent.sinks.mongo-sink.db_prefix = cygnus2_
9. prefix pro the MongoDB collections
   cygnusagent.sinks.mongo-sink.collection_prefix = cygnus2_
10. true is collection names are based on a hash, false for human redable collections
    cygnusagent.sinks.mongo-sink.should_hash = false
11. Must be dm-by-service-path or dm-by-entity
    cygnusagent.sinks.mongo-sink.data_model = dm-by-entity
12. how the attributes are stored, either per row either per column (row, column)
    cygnusagent.sinks.mongo-sink.attr_persistence = column
    # number of notifications to be included within a processing batch
    cygnusagent.sinks.mongo-sink.batch_size = 1
13. timeout for batch accumulation
    cygnusagent.sinks.mongo-sink.batch_timeout = 30
    # number of retries upon persistence error
    cygnusagent.sinks.mongo-sink.batch_ttl = 10
14. Collections will be removed if older than the value specified in seconds. Set to 0 if not wanting this policy.
    cygnusagent.sinks.mongo-sink.data_expiration = 0
15. The oldest data (according to insertion time) will be removed if the size of the data collection gets bigger than the value specified in bytes. Minimum value (different than 0) is 4096 bytes.
    cygnusagent.sinks.mongo-sink.collection_size = 0
16. The oldest data (according to insertion time) will be removed if the number of documents in the data collections goes beyond the specified value. Set to 0 if not wanting this policy
    cygnusagent.sinks.mongo-sink.max_documents = 0

And another agent's configuration:

1. source configuration
2. channel name where to write the notification events
   #####cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel postgresql-channel ckan-channel mongo-channel sth-channel kafka-channel
   cygnusagent.sources.http-source.channels = mongo-channel
3. source class, must not be changed
   cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
4. listening port the Flume source will use for receiving incoming notifications
   cygnusagent.sources.http-source.port = 5050
5. Flume handler that will parse the notifications, must not be changed
   cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
6. URL target
   cygnusagent.sources.http-source.handler.notification_target = /notify
7. Default service (service semantic depends on the persistence sink)
   cygnusagent.sources.http-source.handler.default_service = def_serv
8. Default service path (service path semantic depends on the persistence sink)
   cygnusagent.sources.http-source.handler.default_service_path = def_serv_path
9. Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
   cygnusagent.sources.http-source.handler.events_ttl = -1
10. Source interceptors, do not change
    cygnusagent.sources.http-source.interceptors = ts gi
11. TimestampInterceptor, do not change
    cygnusagent.sources.http-source.interceptors.ts.type = timestamp
12. GroupinInterceptor, do not change
    cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
13. Grouping rules for the GroupingInterceptor, put the right absolute path to the file if necessary
14. See the doc/design/interceptors document for more details
    #####cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf

Channel's conf:

1. mongo-channel configuration
2. channel type (must not be changed)
   cygnusagent.channels.mongo-channel.type = memory
3. capacity of the channel
   cygnusagent.channels.mongo-channel.capacity = 10000
4. amount of bytes that can be sent per transaction
   cygnusagent.channels.mongo-channel.transactionCapacity = 100