Created question in FIWARE Q/A platform on 06-11-2017 at 14:11
Please, ANSWER this question AT https://stackoverflow.com/questions/47138367/make-fiware-cygnus-storing-to-apache-phoenix-database

Question:
Make Fiware-Cygnus storing to Apache-Phoenix database

Description:
I'm new to Cygnus and to Phoenix db. I want Cygnus to store data in an Apache Phoenix db, so I have to configure the Phoenix Sink to work with Cygnus.

I know that Cygnus is an extension of Apache Flume, so any Flume agent can work with it by default (theoretically). I configured my Flume agent agent_phoenix.conf file for Cygnus like this:

cygnus-ngsi.channels = memoryChannel
cygnus-ngsi.sinks = phoenix-sink

cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.channels = memoryChannel
cygnus-ngsi.sources.http-source.port = 5051
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /opt/apache-flume/conf/grouping_rules.conf

cygnus-ngsi.channels.memoryChannel.type=memory
cygnus-ngsi.channels.memoryChannel.transactionCapacity=100
cygnus-ngsi.channels.memoryChannel.byteCapacityBufferPercentage=20

cygnus-ngsi.sinks.phoenix-sink.type=org.apache.phoenix.flume.sink.PhoenixSink
cygnus-ngsi.sinks.phoenix-sink.channel=memoryChannel
cygnus-ngsi.sinks.phoenix-sink.batchSize=100
cygnus-ngsi.sinks.phoenix-sink.table=PROVA_PHOENIX
cygnus-ngsi.sinks.phoenix-sink.ddl=CREATETABLEIFNOTEXISTSPROVA_PHOENIX(uidVARCHARNOTNULL,f_hostVARCHARCONSTRAINTpkPRIMARYKEY(uid),colonnaProvaVARCHAR)
cygnus-ngsi.sinks.phoenix-sink.zookeeperQuorum=localhost
cygnus-ngsi.sinks.phoenix-sink.serializer=REGEX
cygnus-ngsi.sinks.phoenix-sink.serializer.rowkeyType=uuid
cygnus-ngsi.sinks.phoenix-sink.serializer.columns=colonnaProva

and launched it using this command on CentOS command line:

/usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/agent_phoenix.conf -n cygnus-ngsi -Dflume.root.logger=INFO,console

It starts doing some initializations but then I get this message:

76666 [lifecycleSupervisor-1-8] ERROR org.apache.phoenix.flume.serializer.BaseEventSerializer - error No suitable driver found for jdbc:phoenix:localhost; occurred during initializing connection
176667 [lifecycleSupervisor-1-8] ERROR org.apache.phoenix.flume.sink.PhoenixSink - Error No suitable driver found for jdbc:phoenix:localhost; in initializing the serializer.
176667 [lifecycleSupervisor-1-8] ERROR org.apache.flume.lifecycle.LifecycleSupervisor - Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@29918f3 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.RuntimeException: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost;
at com.google.common.base.Throwables.propagate(Throwables.java:156)
at org.apache.phoenix.flume.sink.PhoenixSink.start(PhoenixSink.java:124)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost;
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:187)
at org.apache.phoenix.flume.serializer.BaseEventSerializer.initialize(BaseEventSerializer.java:140)
at org.apache.phoenix.flume.sink.PhoenixSink.start(PhoenixSink.java:119)
... 10 more

HBase and Phoenix are installed correctly on this machine, infact HBase can start and Phoenix has been tested with SQLline file.
Do you have any solutions to this problem?

---EDIT---
I just made a step forward. I edited my flume-env.sh file inside cygnus/conf directory, and specified as FLUME_CLASSPATH the .jar file containing PhoenixSink java class. Now launching the same command that I wrote above it starts configuring (apparently well) then I get this error msg:

time=2017-10-30T22:27:40.552Z | lvl=ERROR | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=run | msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable[142] : Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load sink type: org.apache.phoenix.flume.sink.PhoenixSink, class: org.apache.phoenix.flume.sink.PhoenixSink
at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69)
at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.phoenix.flume.sink.PhoenixSink
at java.net.URLClassLoader$1.run(URLClassLoader.java:359)
at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:312)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:195)
at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
... 11 more

What is wrong now?

------------EDIT 2------------------

Ok I'm gettint closer, but this error has no sense. I solved the jdbc problem by copying the right phoenix JAR in classpath, then I got some dependecies errors and I solved them by copying other JARs in classpath. Now I get the same old error of jdbc. I really can't figure out what is happening:

[root@fiware fiware]# /usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/agent_phoenix.conf -n cygnus-ngsi -Dflume.root.logger=INFO,console
Info: Sourcing environment configuration script /usr/cygnus/conf/flume-env.sh
+ exec /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -Xms512m -Xmx1g -Dcom.sun.management.jmxremote -Dflume.root.logger=INFO,console -cp '/usr/cygnus/conf:/usr/cygnus/lib/:/usr/cygnus/plugins.d/phoenix-sink/:/usr/cygnus/plugins.d/phoenix-sink/lib/:/usr/cygnus/plugins.d/phoenix-sink/libext/*' -Djava.library.path= com.telefonica.iot.cygnus.nodes.CygnusApplication -f /usr/cygnus/conf/agent_phoenix.conf -n cygnus-ngsi
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/phoenix-sink/lib/cygnus-ngsi-1.7.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
time=2017-10-31T06:59:24.751Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp= | op=main | msg=com.telefonica.iot.cygnus.nodes.CygnusApplication[166] : Starting Cygnus, version 1.7.1.UNKNOWN
time=2017-10-31T06:59:24.897Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=main | msg=com.telefonica.iot.cygnus.nodes.CygnusApplication[289] : Waiting for valid Flume components references...
time=2017-10-31T06:59:24.900Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=start | msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider[61] : Configuration provider starting
time=2017-10-31T06:59:24.909Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=run | msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable[133] : Reloading configuration file:/usr/cygnus/conf/agent_phoenix.conf
time=2017-10-31T06:59:24.915Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.915Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.916Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.916Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.916Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.916Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.916Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.917Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.917Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.917Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[930] : Added sinks: phoenix-sink Agent: cygnus-ngsi
time=2017-10-31T06:59:24.917Z | lvl=WARN | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=isValid | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[319] : Agent configuration for 'c' does not contain any channels. Marking it as invalid.
time=2017-10-31T06:59:24.919Z | lvl=WARN | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=validateConfiguration | msg=org.apache.flume.conf.FlumeConfiguration[127] : Agent configuration invalid for agent 'c'. It will be removed.
time=2017-10-31T06:59:24.934Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=validateConfiguration | msg=org.apache.flume.conf.FlumeConfiguration[140] : Post-validation flume configuration contains configuration for agents: [cygnus-ngsi]
time=2017-10-31T06:59:24.934Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=loadChannels | msg=org.apache.flume.node.AbstractConfigurationProvider[150] : Creating channels
time=2017-10-31T06:59:24.944Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=create | msg=org.apache.flume.channel.DefaultChannelFactory[40] : Creating instance of channel memoryChannel type com.telefonica.iot.cygnus.channels.CygnusMemoryChannel
time=2017-10-31T06:59:24.949Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=loadChannels | msg=org.apache.flume.node.AbstractConfigurationProvider[205] : Created channel memoryChannel
time=2017-10-31T06:59:24.951Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=create | msg=org.apache.flume.source.DefaultSourceFactory[39] : Creating instance of source http-source, type org.apache.flume.source.http.HTTPSource
time=2017-10-31T06:59:24.979Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=configure | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[156] : [NGSIRestHandler] Startup completed
time=2017-10-31T06:59:24.994Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=create | msg=org.apache.flume.sink.DefaultSinkFactory[40] : Creating instance of sink: phoenix-sink, type: org.apache.phoenix.flume.sink.PhoenixSink
time=2017-10-31T06:59:25.247Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=getConfiguration | msg=org.apache.flume.node.AbstractConfigurationProvider[119] : Channel memoryChannel connected to [http-source, phoenix-sink]
time=2017-10-31T06:59:25.255Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=startAllComponents | msg=org.apache.flume.node.Application[138] : Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource

}} sinkRunners:{phoenix-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@6be236d0 counterGroup:{ name:null counters:{} } }} channels:{memoryChannel=com.telefonica.iot.cygnus.channels.CygnusMemoryChannel{name: memoryChannel}} }
time=2017-10-31T06:59:25.256Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=startAllComponents | msg=org.apache.flume.node.Application[145] : Starting Channel memoryChannel
time=2017-10-31T06:59:25.258Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=register | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[110] : Monitoried counter group for type: CHANNEL, name: memoryChannel, registered successfully.
time=2017-10-31T06:59:25.259Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=start | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component type: CHANNEL, name: memoryChannel started
time=2017-10-31T06:59:25.259Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=startAllComponents | msg=org.apache.flume.node.Application[173] : Starting Sink phoenix-sink
time=2017-10-31T06:59:25.261Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=start | msg=org.apache.phoenix.flume.sink.PhoenixSink[116] : Starting sink Phoenix Sink__1
time=2017-10-31T06:59:25.268Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=register | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[110] : Monitoried counter group for type: SINK, name: Phoenix Sink__1, registered successfully.
time=2017-10-31T06:59:25.268Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=startAllComponents | msg=org.apache.flume.node.Application[184] : Starting Source http-source
time=2017-10-31T06:59:25.268Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=start | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component type: SINK, name: Phoenix Sink__1 started
time=2017-10-31T06:59:25.273Z | lvl=WARN | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=<init> | msg=com.telefonica.iot.cygnus.interceptors.CygnusGroupingRules[49] : No grouping rules have been read. Details: /opt/apache-flume/conf/grouping_rules.conf (No such file or directory)
time=2017-10-31T06:59:25.287Z | lvl=WARN | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=<init> | msg=com.telefonica.iot.cygnus.interceptors.CygnusGroupingRules[49] : No grouping rules have been read. Details: /opt/apache-flume/conf/grouping_rules.conf (No such file or directory)
time=2017-10-31T06:59:25.412Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=register | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[110] : Monitoried counter group for type: SOURCE, name: http-source, registered successfully.
time=2017-10-31T06:59:25.412Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=start | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component type: SOURCE, name: http-source started
time=2017-10-31T06:59:25.460Z | lvl=ERROR | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=initialize | msg=org.apache.phoenix.flume.serializer.BaseEventSerializer[200] : error No suitable driver found for jdbc:phoenix:localhost; occurred during initializing connection
time=2017-10-31T06:59:25.460Z | lvl=ERROR | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=start | msg=org.apache.phoenix.flume.sink.PhoenixSink[123] : Error No suitable driver found for jdbc:phoenix:localhost; in initializing the serializer.
time=2017-10-31T06:59:25.462Z | lvl=ERROR | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=run | msg=org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable[253] : Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@6be236d0 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.RuntimeException: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost;
at com.google.common.base.Throwables.propagate(Throwables.java:156)
at org.apache.phoenix.flume.sink.PhoenixSink.start(PhoenixSink.java:124)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost;
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:187)
at org.apache.phoenix.flume.serializer.BaseEventSerializer.initialize(BaseEventSerializer.java:140)
at org.apache.phoenix.flume.sink.PhoenixSink.start(PhoenixSink.java:119)
... 10 more

May someone explain this?