Details
-
Type: Monitor
-
Status: Closed
-
Priority: Major
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: 2021
-
Component/s: FIWARE-TECH-HELP
-
Labels:
-
HD-Chapter:Data
-
HD-Enabler:Orion
Description
Created question in FIWARE Q/A platform on 28-04-2016 at 14:04
Please, ANSWER this question AT https://stackoverflow.com/questions/36914589/how-to-combine-multiple-orion-entities-to-one-ckan-dataset-using-cygnus
Question:
how to combine multiple orion entities to one ckan dataset using cygnus?
Description:
Right now I'm trying to create a dataset for my orion data with all the entitys info.
I set up all correctly and when some data changes orion send a notification to cygnus and then, cygnus will add the data to ckan.
As seen in this image.
The problem is that cygnus create a ckan resource for each entity.
What could I do to make cygnus put all data in only one resource?
Thanks.
EDIT 1:
I just found that I can simplify by making just one subscription in orion.
{
"entities": [
] ...
Then I noticed the grouping feature (http://fiware-cygnus.readthedocs.io/en/develop/installation_and_administration_guide/grouping_rules_conf/index.html) in cygnus but right now I'm lost because cygnus don't want to load my grouping config file. It shows this error:
time=2016-05-03T05:32:29.658CDT | lvl=INFO | trans= | srv= | subsrv= | function=<init> | comp=Cygnus | msg=com.telefonica.iot.cygnus.interceptors.GroupingRules[58] : No grouping rules have been read
Exception in thread "Thread-1" java.lang.NullPointerException
at java.io.File.<init>(File.java:277)
at com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$ConfigurationReader.run(GroupingInterceptor.java:244)
In my agent config file I have:
cygnusagent.sources.http-source.interceptors.gi.gropuing_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
And in grouping_rules.conf:
{
"grouping_rules": [
]
}
Edit 2:
I have all the lines as documentation said:
cygnusagent.sources = http-source
cygnusagent.sinks = ckan-sink
cygnusagent.channels = ckan-channel
cygnusagent.sources.http-source.channels = ckan-channel
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnusagent.sources.http-source.port = 5050
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
cygnusagent.sources.http-source.handler.notification_target = /notify
cygnusagent.sources.http-source.handler.default_service = test
cygnusagent.sources.http-source.handler.default_service_path = /myhouse
cygnusagent.sources.http-source.handler.events_ttl = 5
cygnusagent.sources.http-source.interceptors = ts gi
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
cygnusagent.sources.http-source.interceptors.gi.gropuing_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
cygnusagent.channels.ckan-channel.type = memory
cygnusagent.channels.ckan-channel.capacity = 1000
cygnusagent.channels.ckan-channel.transactionCapacity = 100
- ============================================
- OrionCKANSink configuration
- channel name from where to read notification events
cygnusagent.sinks.ckan-sink.channel = ckan-channel
- sink class, must not be changed
cygnusagent.sinks.ckan-sink.type = com.telefonica.iot.cygnus.sinks.OrionCKANSink
- true if the grouping feature is enabled for this sink, false otherwise
cygnusagent.sinks.ckan-sink.enable_grouping = true
- true if lower case is wanted to forced in all the element names, false otherwise
cygnusagent.sinks.hdfs-sink.enable_lowercase = false
- the CKAN API key to use
cygnusagent.sinks.ckan-sink.api_key = 436fffc8-b397-478a-92fd-bbc5ffaf8269
- the FQDN/IP address for the CKAN API endpoint
cygnusagent.sinks.ckan-sink.ckan_host = ckan-demo.ckan.io
- the port for the CKAN API endpoint
cygnusagent.sinks.ckan-sink.ckan_port = 80
- Orion URL used to compose the resource URL with the convenience operation URL to query it
cygnusagent.sinks.ckan-sink.orion_url = http://localhost:1026
- how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.ckan-sink.attr_persistence = column
- enable SSL for secure Http transportation; 'true' or 'false'
cygnusagent.sinks.ckan-sink.ssl = false
- number of notifications to be included within a processing batch
cygnusagent.sinks.ckan-sink.batch_size = 100
- timeout for batch accumulation
cygnusagent.sinks.ckan-sink.batch_timeout = 60
- number of retries upon persistence error
cygnusagent.sinks.ckan-sink.batch_ttl = 10
The file rights seems ok to me:
[root@VM013cen-Prod conf]# ls *.conf -l
-rwxrwxrwx 1 cygnus cygnus 2675 may 3 06:45 agent_test.conf
-rwxrwxrwx 1 cygnus cygnus 258 may 3 05:08 grouping_rules.conf
-rwxr-xr-x 1 cygnus cygnus 135 mar 1 02:50 krb5_login.conf
The test pases ok:
Results :
Tests run: 80, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:05.862s
[INFO] Finished at: Tue May 03 06:53:44 CDT 2016
[INFO] Final Memory: 41M/105M
[INFO] ------------------------------------------------------------------------
The startup command i use: /usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/agent_test.conf -n cygnusagent -Dflume.root.logger=INFO,console
2017-05-22 15:15|CREATED monitor | # answers= 1, accepted answer= True