Details
-
Type:
Monitor
-
Status: Closed
-
Priority:
Major
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: 2021
-
Component/s: FIWARE-TECH-HELP
-
Labels:
-
HD-Chapter:Data
-
HD-Enabler:Cosmos
Description
Created question in FIWARE Q/A platform on 22-10-2014 at 19:10
Please, ANSWER this question AT https://stackoverflow.com/questions/26513757/storing-sql-in-cosmos
Question:
Storing SQL in Cosmos
Description:
I need to persist data in Cosmos in SQL tables instead of HDFS files.
I've deployed a VM in Cloud section of FI-Lab where I've installed the 0.14.0 Orion version and 0.3 of Cygnus. I've configured Cygnus to store data in HDFS and SQL... Problem is, persistence in HDFS files works fine but it is not possible in SQL tables despite of in the past I got it. That's why I'm confused
I guess if HDFS persistence works, it should be problem of cygnus.config file. So I show it below:
- APACHE_FLUME_HOME/conf/cygnus.conf
- The next tree fields set the sources, sinks and channels used by Cygnus. You could use different names than the
- ones suggested below, but in that case make sure you keep coherence in properties names along the configuration file.
- Regarding sinks, you can use multiple ones at the same time; the only requirement is to provide a channel for each
- one of them (this example shows how to configure 3 sinks at the same time).
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink mysql-sink
cygnusagent.channels = hdfs-channel mysql-channel
#=============================================
- source configuration
- channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel - source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource - listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050 - Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler - URL target
cygnusagent.sources.http-source.handler.notification_target = /notify - Default organization (organization semantic depend on the persistence sink)
cygnusagent.sources.http-source.handler.default_organization = org42
- ============================================
- OrionHDFSSink configuration
- channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel - sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink - The FQDN/IP address of the Cosmos deployment where the notification events will be persisted
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46 - port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000 - default username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = quiquehz - default password for the default username
cygnusagent.sinks.hdfs-sink.cosmos_default_password = 'password' - HDFS backend type (webhdfs, httpfs or infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs - how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.hdfs-sink.attr_persistence = column - prefix for the database and table names, empty if no prefix is desired
cygnusagent.sinks.hdfs-sink.naming_prefix = - Hive port for Hive external table provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000
- ============================================
- OrionMySQLSink configuration
- channel name from where to read notification events
cygnusagent.sinks.mysql-sink.channel = mysql-channel - sink class, must not be changed
cygnusagent.sinks.mysql-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink - the FQDN/IP address where the MySQL server runs
cygnusagent.sinks.mysql-sink.mysql_host = 130.206.80.46 - the port where the MySQL server listes for incomming connections
cygnusagent.sinks.mysql-sink.mysql_port = 3306 - a valid user in the MySQL server
cygnusagent.sinks.mysql-sink.mysql_username = quiquehz - password for the user above
cygnusagent.sinks.mysql-sink.mysql_password = 'password' - how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.mysql-sink.attr_persistence = column - prefix for the database and table names, empty if no prefix is desired
cygnusagent.sinks.mysql-sink.naming_prefix =
#=============================================
- hdfs-channel configuration
- channel type (must not be changed)
cygnusagent.channels.hdfs-channel.type = memory - capacity of the channel
cygnusagent.channels.hdfs-channel.capacity = 1000 - amount of bytes that can be sent per transaction
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
#=============================================
- mysql-channel configuration
- channel type (must not be changed)
cygnusagent.channels.mysql-channel.type = memory - capacity of the channel
cygnusagent.channels.mysql-channel.capacity = 1000 - amount of bytes that can be sent per transaction
cygnusagent.channels.mysql-channel.transactionCapacity = 100
Activity
- All
- Comments
- History
- Activity
- Transitions