Details
-
Type: Monitor
-
Status: Closed
-
Priority: Major
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: 2021
-
Component/s: FIWARE-TECH-HELP
-
Labels:
-
HD-Chapter:Data
-
HD-Enabler:Cosmos
Description
Created question in FIWARE Q/A platform on 17-03-2015 at 13:03
Please, ANSWER this question AT http://stackoverflow.com/questions/29099334/cygnus-0-7-1-does-not-create-tables-mysql-and-hdfs
Question:
Cygnus 0.7.1 does not create tables (MySQL and HDFS)
Description:
I've installed (from source) cygnus 0.7.1, and after configuring it (MySQL and HDFS sinks) I can start it without problems. When I subscribe cygnus to a orion context, it receives the information ok, but there is a problem with MySQL and HDFS. This is the log:
15/03/17 13:58:52 INFO handlers.OrionRestHandler: Starting transaction (1426597123-891-0000000000)
15/03/17 13:58:52 INFO handlers.OrionRestHandler: Received data ({ "subscriptionId" : "5508250c1860a36e55240c84", "originator" : "localhost", "contextResponses" : [ { "contextElement" : { "type" : "ubk-temp", "isPattern" : "false", "id" : "ubk:temp:1", "attributes" : [
] }, "statusCode" :
{ "code" : "200", "reasonPhrase" : "OK" } } ]})
15/03/17 13:58:52 INFO handlers.OrionRestHandler: Event put in the channel (id=1549700267, ttl=10)
15/03/17 13:58:52 INFO sinks.OrionSink: Event got from the channel (id=1549700267, headers=
15/03/17 13:58:52 INFO sinks.OrionSink: Event got from the channel (id=1549700267, headers={fiware-servicepath=ubktemp, destination=ubk_temp_1_ubk-temp, content-type=application/json, fiware-service=ubikwa, ttl=10, transactionId=1426597123-891-0000000000, timestamp=1426597132511}
, bodyLength=462)
15/03/17 13:58:52 INFO sinks.OrionMySQLSink: [mysql-sink] Persisting data at OrionMySQLSink. Database: ubikwa, Table: ubktemp_ubk_temp_1_ubk-temp, Timestamp: 2015-03-17T13:58:52.511, Data (attrs):
, (metadata):
{temperature_md=[]}15/03/17 13:58:53 INFO sinks.OrionHDFSSink: [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (ubikwa/ubktemp/ubk_temp_1_ubk-temp/ubk_temp_1_ubk-temp.txt), Data (
{"recvTime":"2015-03-17T13:58:52.511","temperature":"11", "temperature_md":[]})
15/03/17 13:58:53 WARN sinks.OrionSink: Bad context data (Table 'ubikwa.ubktemp_ubk_temp_1_ubk-temp' doesn't exist)
15/03/17 13:58:53 INFO sinks.OrionSink: Finishing transaction (1426597123-891-0000000000)
The MySQL sink does not raise any errors but no tables are created. And the HDFS sink seems to be unable to create the files. I previously installed cygnus 0.6 and it worked with the same configuration.
Edit:
Here its is my configuration:
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink mysql-sink
cygnusagent.channels = hdfs-channel mysql-channel
#=============================================
- source configuration
- channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel - source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource - listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050 - Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler - URL target
cygnusagent.sources.http-source.handler.notification_target = /notify - Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = ubikwa - Default service path (service path semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service_path = ubktemp - Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10 - Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts de - Timestamp interceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp - Destination extractor interceptor, do not change
cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor$Builder - Matching table for the destination extractor interceptor, put the right absolute path to the file if necessary
- See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.de.matching_table = /opt/cygnus/conf/matching_table.conf
- ============================================
- OrionHDFSSink configuration
- channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel - sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink - Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
- If you are using Kerberos authentication, then the usage of FQDNs instead of IP addresses is mandatory
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46 - port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000 - default username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = *** - default password for the default username
cygnusagent.sinks.hdfs-sink.cosmos_default_password = *** - HDFS backend type (webhdfs, httpfs or infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs - how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.hdfs-sink.attr_persistence = column - Hive FQDN/IP address of the Hive server
cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46 - Hive port for Hive external table provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000 - Kerberos-based authentication enabling
cygnusagent.sinks.hdfs-sink.krb5_auth = false - Kerberos username
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_user = krb5_username - Kerberos password
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_password = xxxxxxxxxxxxx - Kerberos login file
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_login_conf_file = /usr/cygnus/conf/krb5_login.conf - Kerberos configuration file
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_conf_file = /usr/cygnus/conf/krb5.conf
- ============================================
- OrionMySQLSink configuration
- channel name from where to read notification events
cygnusagent.sinks.mysql-sink.channel = mysql-channel - sink class, must not be changed
cygnusagent.sinks.mysql-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink - the FQDN/IP address where the MySQL server runs
cygnusagent.sinks.mysql-sink.mysql_host = 127.0.0.1 - the port where the MySQL server listes for incomming connections
cygnusagent.sinks.mysql-sink.mysql_port = 3306 - a valid user in the MySQL server
cygnusagent.sinks.mysql-sink.mysql_username = *** - password for the user above
cygnusagent.sinks.mysql-sink.mysql_password = *** - how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.mysql-sink.attr_persistence = column
#=============================================
- hdfs-channel configuration
- channel type (must not be changed)
cygnusagent.channels.hdfs-channel.type = memory - capacity of the channel
cygnusagent.channels.hdfs-channel.capacity = 1000 - amount of bytes that can be sent per transaction
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
#=============================================
- mysql-channel configuration
- channel type (must not be changed)
cygnusagent.channels.mysql-channel.type = memory - capacity of the channel
cygnusagent.channels.mysql-channel.capacity = 1000 - amount of bytes that can be sent per transaction
cygnusagent.channels.mysql-channel.transactionCapacity = 100
Any hints?
Thanks
Activity
- All
- Comments
- History
- Activity
- Transitions