Created question in FIWARE Q/A platform on 28-05-2015 at 17:05
Please, ANSWER this question AT https://stackoverflow.com/questions/30511082/cygnus-not-presisting-data-on-mysql-database

Question:
Cygnus not presisting data on MySql database

Description:
So i have read all the documentation and followed the tutorial on MySQL persistence but i can't still presist any kind of data on MySQL database.

Even though i'm puting the presistence mode = row it doesn't create any database nor table.

What am i doing wrong?

My Subscription:

python2.7 SetSubscription.py bustest4 http://localhost:5050/notify

Output:

• Asking to http://localhost:1026/v1/subscribeContext
• Headers: {'Fiware-Service': 'fiwaretestapi', 'content-type': 'application/json ', 'accept': 'application/json', 'X-Auth-Token': 'NULL'}
• Sending PAYLOAD:
  {
  "reference": "http://localhost:5050/notify",
  "throttling": "PT5S",
  "entities": [ { "type": "", "id": "bustest4", "isPattern": "false" }

  ],
  "attributes": [
  "temperature"
  ],
  "duration": "P1M",
  "notifyConditions": [

  { "condValues": [ "temperature" ], "type": "ONCHANGE" }

  ]
  }

...

• Status Code: 200

{
"subscribeResponse" :

{ "subscriptionId" : "5567332298add18cc3e183ac", "duration" : "P1M", "throttling" : "PT5S" }

}

My agent_a1.conf:

#=============================================

1. source configuration
2. channel name where to write the notification events
   cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel ckan-channel
3. source class, must not be changed
   cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
4. listening port the Flume source will use for receiving incoming notifications
   cygnusagent.sources.http-source.port = 5050
5. Flume handler that will parse the notifications, must not be changed
   cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
6. URL target
   cygnusagent.sources.http-source.handler.notification_target = /notify
7. Default service (service semantic depends on the persistence sink)
   cygnusagent.sources.http-source.handler.default_service = fiwaretestapi
8. Default service path (service path semantic depends on the persistence sink)
   cygnusagent.sources.http-source.handler.default_service_path = /
9. Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
   cygnusagent.sources.http-source.handler.events_ttl = 10
10. Source interceptors, do not change
    cygnusagent.sources.http-source.interceptors = ts de
11. Timestamp interceptor, do not change
    cygnusagent.sources.http-source.interceptors.ts.type = timestamp
12. Destination extractor interceptor, do not change
    cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor$Builder
13. Matching table for the destination extractor interceptor, put the right absolute path to the file if necessary
14. See the doc/design/interceptors document for more details
    cygnusagent.sources.http-source.interceptors.de.matching_table = /usr/cygnus/conf/matching_table.conf

1. ============================================
2. OrionMySQLSink configuration
3. channel name from where to read notification events
   cygnusagent.sinks.mysql-sink.channel = mysql-channel
4. sink class, must not be changed
   cygnusagent.sinks.mysql-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink
5. the FQDN/IP address where the MySQL server runs
   cygnusagent.sinks.mysql-sink.mysql_host = localhost
6. the port where the MySQL server listes for incomming connections
   cygnusagent.sinks.mysql-sink.mysql_port = 3306
7. a valid user in the MySQL server
   cygnusagent.sinks.mysql-sink.mysql_username = root
8. password for the user above
   cygnusagent.sinks.mysql-sink.mysql_password = ***********
9. how the attributes are stored, either per row either per column (row, column)
   cygnusagent.sinks.mysql-sink.attr_persistence = row

#=============================================

1. mysql-channel configuration
2. channel type (must not be changed)
   cygnusagent.channels.mysql-channel.type = memory
3. capacity of the channel
   cygnusagent.channels.mysql-channel.capacity = 1000
4. amount of bytes that can be sent per transaction
   cygnusagent.channels.mysql-channel.transactionCapacity = 100

My cygnus_instance_c1.conf:

1. Who to run cygnus as. Note that you may need to use root if you want
2. to run cygnus in a privileged port (<1024)
   CYGNUS_USER=root

1. Where is the config folder
   CONFIG_FOLDER=/usr/cygnus/conf

1. Which is the config file
   CONFIG_FILE=/usr/cygnus/conf/agent_a1.conf

1. Name of the agent. The name of the agent is not trivial, since it is the base for the Flume parameters
2. naming conventions, e.g. it appears in .sources.http-source.channels=...
   AGENT_NAME=cygnususer

1. Name of the logfile located at /var/log/cygnus. It is important to put the extension '.log' in order to the log rotation works properly
   LOGFILE_NAME=cygnus.log

1. Administration port. Must be unique per instance
   ADMIN_PORT=8081

My cygnus.log:

Info: Sourcing environment configuration script /usr/cygnus/conf/flume-env.sh
Warning: JAVA_HOME is not set!
+ exec /usr/bin/java -Xmx20m -Dflume.log.file=cygnus.log -cp '/usr/cygnus/conf:/usr/cygnus/lib/:/usr/cygnus/plugins.d/cygnus/lib/:/usr/cygnus/plugins.d/cygnus/libext/*' -Djava.library.path= es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication -p 8081 -f /usr/cygnus/conf/agent_a1.conf -n root
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/cygnus/lib/cygnus-0.7.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

EDIT:

So after some changes i got the log file to work and i found out that the 8081 port was already in use. Can you explain me what the ADMIN_PORT is used for, what port is it recommended to be?

LOG FILE:

02 Jun 2015 05:16:40,680 INFO [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133) - Reloading configuration file:/usr/cygnus/conf/agent_a1.conf
02 Jun 2015 05:16:40,685 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
02 Jun 2015 05:16:40,686 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
02 Jun 2015 05:16:40,686 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930) - Added sinks: mysql-sink Agent: cygnusagent
02 Jun 2015 05:16:40,686 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
02 Jun 2015 05:16:40,692 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140) - Post-validation flume configuration contains configuration for agents: [cygnusagent]
02 Jun 2015 05:16:40,692 WARN [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:138) - No configuration found for this host:root
02 Jun 2015 05:16:40,693 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:101) - Shutting down configuration: { sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,693 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,694 INFO [conf-file-poller-0] (es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication.startManagementInterface:85) - Starting a Jetty server listening on port 8081 (Management Interface)
02 Jun 2015 05:16:40,695 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:101) - Shutting down configuration: { sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,695 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,695 INFO [conf-file-poller-0] (es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication.startManagementInterface:85) - Starting a Jetty server listening on port 8081 (Management Interface)
02 Jun 2015 05:16:40,696 INFO [Thread-26] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
02 Jun 2015 05:16:40,704 WARN [Thread-26] (org.mortbay.log.Slf4jLog.warn:76) - failed SocketConnector@0.0.0.0:8081: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,704 WARN [Thread-26] (org.mortbay.log.Slf4jLog.warn:76) - failed Server@4f1b95f: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,704 FATAL [Thread-26] (es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer.run:63) - Fatal error running the Management Interface. Details=Address already in use
02 Jun 2015 05:16:40,705 INFO [Thread-27] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
02 Jun 2015 05:16:40,709 WARN [Thread-27] (org.mortbay.log.Slf4jLog.warn:76) - failed SocketConnector@0.0.0.0:8081: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,709 WARN [Thread-27] (org.mortbay.log.Slf4jLog.warn:76) - failed Server@ed4c222: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,709 FATAL [Thread-27] (es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer.run:63) - Fatal error running the Management Interface. Details=Address already in use

EDIT 2:

My script that updates entity on Context Broker:

BASE_URL = 'http://localhost:1026'
UPDATE_URL = BASE_URL+'/ngsi10/updateContext'

HEADERS = {
'Content-Type': 'application/json',
'Accept': 'application/json',
'Fiware-Service' : 'fiwaretestapi',
'Fiware-ServicePath': '/'
}

UPDATE_EXAMPLE = {
"contextElements": [
{
"type": "",
"isPattern": "false",
"id": "bustest4",
"attributes": [

{ "name": "temperature", "type": "int", "value": "99" }

]
}
],
"updateAction": "APPEND"
}

def post(url, data):
""""""
req = urllib2.Request(url, data, HEADERS)
f = urllib2.urlopen(req)
result = json.loads(f.read())
f.close()
return result

if _name_ == "_main_":
print post(UPDATE_URL, json.dumps(UPDATE_EXAMPLE))

EDIT3:

Even though i set the admin port to be 8085 on the cygnus agent configuration it tries to bind to the 8081, is that normal?

Here are the logs from the cygnus:

time=2015-06-12T05:56:39.820EDT | lvl=INFO | trans= | function=start | comp=Cygnu s | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component ty pe: CHANNEL, name: mysql-channel started
time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startAllComponents | comp=Cygnus | msg=org.apache.flume.node.Application[173] : Starting Sink mysql -sink
time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startAllComponents | comp=Cygnus | msg=org.apache.flume.node.Application[184] : Starting Source htt p-source
time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startManagementInt erface | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusAppl ication[85] : Starting a Jetty server listening on port 8081 (Management Interfac e)
time=2015-06-12T05:56:39.822EDT | lvl=INFO | trans= | function=start | comp=Cygnu s | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink[151] : [mysql- sink] Startup completed
time=2015-06-12T05:56:39.823EDT | lvl=INFO | trans= | function=info | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[67] : jetty-6.1.26
time=2015-06-12T05:56:39.824EDT | lvl=INFO | trans= | function=info | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[67] : jetty-6.1.26
time=2015-06-12T05:56:39.825EDT | lvl=INFO | trans= | function=info | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[67] : Started SocketConnector@0.0.0.0:5050
time=2015-06-12T05:56:39.825EDT | lvl=INFO | trans= | function=start | comp=Cygnu s | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component ty pe: SOURCE, name: http-source started
time=2015-06-12T05:56:39.827EDT | lvl=WARN | trans= | function=warn | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[76] : failed SocketConnector@0.0.0.0:8081: java.n et.BindException: Address already in use
time=2015-06-12T05:56:39.827EDT | lvl=WARN | trans= | function=warn | comp=Cygnus | msg=org.mortbay.log.Slf4jLog[76] : failed Server@1c9c5521: java.net.BindExcept ion: Address already in use
time=2015-06-12T05:56:39.827EDT | lvl=FATAL | trans= | function=run | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer[63] : Fatal error r unning the Management Interface. Details=Address already in use

Log when i make a subscription:

time=2015-06-12T06:03:56.529EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle rs.OrionRestHandler[153] : Starting transaction (1434103313-190-0000000000)
time=2015-06-12T06:03:56.535EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle rs.OrionRestHandler[239] : Received data ({ "subscriptionId" : "557aae8c98add18c c3e183b6", "originator" : "localhost", "contextResponses" : [ { "contex tElement" : { "type" : "thing", "isPattern" : "false", "id" : "autocarro1", "attributes" : [ { "name" : "temperatu re", "type" : "int", "value" : "95", "metadatas" : [

{ "name" : "TimeInstant", "type" : "ISO8601", "value" : "2015-06-03T09:17:44.046583Z" }

] } ] }, "statusCode" :

{ "code" : "200", "reasonPhrase" : "OK" }

} ]})
time=2015-06-12T06:03:56.540EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle rs.OrionRestHandler[261] : Event put in the channel (id=1983722072, ttl=10)
time=2015-06-12T06:03:56.724EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or ionSink[126] : Event got from the channel (id=1983722072, headers=

{timestamp=1434 103436542, content-type=application/json, transactionId=1434103313-190-0000000000 , fiware-service=fiwaretestapi, fiware-servicepath=, ttl=10, destination=autocarr o1_thing}

, bodyLength=657)
time=2015-06-12T06:03:57.260EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=persist | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or ionMySQLSink[227] : [mysql-sink] Persisting data at OrionMySQLSink. Database: fiw aretestapi, Table: autocarro1_thing, Data: 1434103436,2015-06-12T06:03:56.542,aut ocarro1,thing,temperature,thing,95,[

{"name":"TimeInstant","type":"ISO8601","value ":"2015-06-03T09:17:44.046583Z"}

]
time=2015-06-12T06:03:57.270EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu nction=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or ionSink[187] : Finishing transaction (1434103313-190-0000000000)