Created question in FIWARE Q/A platform on 09-12-2015 at 14:12
Please, ANSWER this question AT https://stackoverflow.com/questions/34180456/why-wont-cygnus-keep-running

Question:
Why wont Cygnus keep running?

Description:
Following the instructions on this link I only changed 2 config files.

The first one is cygnus_instance_1.conf (actually I just renamed the example file):

#####
#

1. Configuration file for apache-flume
   #
   #####
2. Copyright 2014 Telefonica Investigación y Desarrollo, S.A.U
4. This file is part of fiware-cygnus (FI-WARE project).
6. fiware-cygnus is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General
7. Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any
8. later version.
9. fiware-cygnus is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
10. warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
11. details.
13. You should have received a copy of the GNU Affero General Public License along with fiware-cygnus. If not, see
14. http://www.gnu.org/licenses/.
16. For those usages not covered by the GNU Affero General Public License please contact with iot_support at tid dot es

1. Who to run cygnus as. Note that you may need to use root if you want
2. to run cygnus in a privileged port (<1024)
   CYGNUS_USER=cygnus

1. Where is the config folder
   CONFIG_FOLDER=/usr/cygnus/conf

1. Which is the config file
   CONFIG_FILE=/usr/cygnus/conf/agent.conf

1. Name of the agent. The name of the agent is not trivial, since it is the base for the Flume parameters
2. naming conventions, e.g. it appears in .sources.http-source.channels=...
   AGENT_NAME=cygnusagent

1. Name of the logfile located at /var/log/cygnus. It is important to put the extension '.log' in order to the log rotation works properly
   LOGFILE_NAME=cygnus.log

1. Administration port. Must be unique per instance
   ADMIN_PORT=8081

1. Polling interval (seconds) for the configuration reloading
   POLLING_INTERVAL=30

And the second config is agent_1.conf (where I removed all but the mysql DB because that is the one I am using):

1. Copyright 2014 Telefónica Investigación y Desarrollo, S.A.U
3. This file is part of fiware-cygnus (FI-WARE project).
5. fiware-cygnus is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General
6. Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any
7. later version.
8. fiware-cygnus is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
9. warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
10. details.
12. You should have received a copy of the GNU Affero General Public License along with fiware-cygnus. If not, see
13. http://www.gnu.org/licenses/.
15. For those usages not covered by the GNU Affero General Public License please contact with iot_support at tid dot es

#=============================================

1. To be put in APACHE_FLUME_HOME/conf/agent.conf
   #
2. General configuration template explaining how to setup a sink of each of the available types (HDFS, CKAN, MySQL).

#=============================================

1. The next tree fields set the sources, sinks and channels used by Cygnus. You could use different names than the
2. ones suggested below, but in that case make sure you keep coherence in properties names along the configuration file.
3. Regarding sinks, you can use multiple types at the same time; the only requirement is to provide a channel for each
4. one of them (this example shows how to configure 3 sink types at the same time). Even, you can define more than one
5. sink of the same type and sharing the channel in order to improve the performance (this is like having
6. multi-threading).
   cygnusagent.sources = http-source
   cygnusagent.sinks = mysql-sink
   cygnusagent.channels = mysql-channel

#=============================================

1. source configuration
2. channel name where to write the notification events
   cygnusagent.sources.http-source.channels = mysql-channel
3. source class, must not be changed
   cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
4. listening port the Flume source will use for receiving incoming notifications
   cygnusagent.sources.http-source.port = 5050
5. Flume handler that will parse the notifications, must not be changed
   cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
6. URL target
   cygnusagent.sources.http-source.handler.notification_target = /notify
7. Default service (service semantic depends on the persistence sink)
   cygnusagent.sources.http-source.handler.default_service = Trace_Data
8. Default service path (service path semantic depends on the persistence sink)
   cygnusagent.sources.http-source.handler.default_service_path = Sensor
9. Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
   cygnusagent.sources.http-source.handler.events_ttl = 10
10. Source interceptors, do not change
    cygnusagent.sources.http-source.interceptors = ts gi
11. TimestampInterceptor, do not change
    cygnusagent.sources.http-source.interceptors.ts.type = timestamp
12. GroupinInterceptor, do not change
    cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
13. Grouping rules for the GroupingInterceptor, put the right absolute path to the file if necessary
14. See the doc/design/interceptors document for more details
    cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf

1. ============================================
2. OrionMySQLSink configuration
3. channel name from where to read notification events
   cygnusagent.sinks.mysql-sink.channel = mysql-channel
4. sink class, must not be changed
   cygnusagent.sinks.mysql-sink.type = com.telefonica.iot.cygnus.sinks.OrionMySQLSink
   # true if the grouping feature is enabled for this sink, false otherwise
   cygnusagent.sinks.mysql-sink.enable_grouping = false
5. the FQDN/IP address where the MySQL server runs
   cygnusagent.sinks.mysql-sink.mysql_host = 127.0.0.1
6. the port where the MySQL server listes for incomming connections
   cygnusagent.sinks.mysql-sink.mysql_port = 3306
7. a valid user in the MySQL server
   cygnusagent.sinks.mysql-sink.mysql_username = root
8. password for the user above
   cygnusagent.sinks.mysql-sink.mysql_password = klasika
9. how the attributes are stored, either per row either per column (row, column)
   cygnusagent.sinks.mysql-sink.attr_persistence = column
10. select the table type from table-by-destination and table-by-service-path
    cygnusagent.sinks.mysql-sink.table_type = table-by-destination
    # number of notifications to be included within a processing batch
    cygnusagent.sinks.mysql-sink.batch_size = 100
11. timeout for batch accumulation
    cygunsagent.sinks.mysql-sink.batch_timeout = 30

#=============================================

1. mysql-channel configuration
2. channel type (must not be changed)
   cygnusagent.channels.mysql-channel.type = memory
3. capacity of the channel
   cygnusagent.channels.mysql-channel.capacity = 1000
4. amount of bytes that can be sent per transaction
   cygnusagent.channels.mysql-channel.transactionCapacity = 100

#============================================

I even did the test successfully and got the following output:

Tests run: 72, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:28.380s
[INFO] Finished at: Tue Dec 08 15:25:16 CET 2015
[INFO] Final Memory: 43M/112M
[INFO] ------------------------------------------------------------------------

The test worked because I properly configured java and maven:

[root@localhost fiware-cygnus]# echo $JAVA_HOME
/usr/java/jdk1.8.0_65
[root@localhost fiware-cygnus]# java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
[root@localhost fiware-cygnus]# mvn -version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 14:51:28+0100)
Maven home: /usr/local/maven
Java version: 1.8.0_65, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_65/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-504.12.2.el6.x86_64", arch: "amd64", family: "unix"

So all good when I hit service cygnus start because I get:

Starting Cygnus 1... [ OK ]

But then 2 seconds later when I hit service cygnus status I get:

cygnus-flume-ng dead but pid file exists

Context broker is working perfectly by the way, but it just doesnt know that the subscriptions Im entering are not going anywere...

This is the log from cygnus (there is no timestamp so I ill post the first and the last one):

Warning: JAVA_HOME is not set!
+ exec /usr/bin/java -Xmx20m -Dflume.log.file=cygnus.log -cp '/usr/cygnus/conf:/usr/cygnus/lib/:/usr/cygnus/plugins.d/cygnus/lib/:/usr/cygnus/plugins.d/cygnus/libext/*' -Djava.library.path= com.telefonica.iot.cygnus.nodes.CygnusApplic$
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/cygnus/lib/cygnus-0.10.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: ./logs/cygnus.log (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)
at org.apache.flume.node.Application.<clinit>(Application.java:58)

Warning: JAVA_HOME is not set!
+ exec /usr/bin/java -Xmx20m -Dflume.log.file=cygnus.log -cp '/usr/cygnus/conf:/usr/cygnus/lib/:/usr/cygnus/plugins.d/cygnus/lib/:/usr/cygnus/plugins.d/cygnus/libext/*' -Djava.library.path= com.telefonica.iot.cygnus.nodes.CygnusApplic$
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/cygnus/lib/cygnus-0.10.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: ./logs/cygnus.log (No such file or directory)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)
at org.apache.flume.node.Application.<clinit>(Application.java:58)

I demonstrated that I have the latest version of java running and that its home env is set. What am I doing wrong here? I should stress that I havent touched any other config files what so ever. Here is the full list that I found:

[root@localhost conf]# ls
agent_1.conf cygnus_instance_1.conf flume-env.sh.template grouping_rules.conf.template krb5_login.conf log4j.properties.template
agent.conf.template cygnus_instance.conf.template grouping_rules.conf krb5.conf.template log4j.properties README.md

EDIT1: log4.properties as requested:

#

1. Licensed to the Apache Software Foundation (ASF) under one
2. or more contributor license agreements. See the NOTICE file
3. distributed with this work for additional information
4. regarding copyright ownership. The ASF licenses this file
5. to you under the Apache License, Version 2.0 (the
6. "License"); you may not use this file except in compliance
7. with the License. You may obtain a copy of the License at
   #
8. http://www.apache.org/licenses/LICENSE-2.0
   #
9. Unless required by applicable law or agreed to in writing,
10. software distributed under the License is distributed on an
11. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
12. KIND, either express or implied. See the License for the
13. specific language governing permissions and limitations
14. under the License.
    #

1. Define some default values that can be overridden by system properties.
   #
2. For testing, it may also be convenient to specify
3. -Dflume.root.logger=DEBUG,console when launching flume.

#flume.root.logger=DEBUG,console
flume.root.logger=INFO,LOGFILE
flume.log.dir=./logs
flume.log.file=flume.log

log4j.logger.org.apache.flume.lifecycle = INFO
log4j.logger.org.jboss = WARN
log4j.logger.org.mortbay = INFO
log4j.logger.org.apache.avro.ipc.NettyTransceiver = WARN
log4j.logger.org.apache.hadoop = INFO

1. Define the root logger to the system property "flume.root.logger".
   log4j.rootLogger=$ {flume.root.logger}

1. Stock log4j rolling file appender
2. Default log rotation configuration
   log4j.appender.LOGFILE=org.apache.log4j.RollingFileAppender
   log4j.appender.LOGFILE.MaxFileSize=100MB
   log4j.appender.LOGFILE.MaxBackupIndex=10
   log4j.appender.LOGFILE.File=$ {flume.log.dir}/${flume.log.file}
   log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
   log4j.appender.LOGFILE.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n
   
   
   # Warning: If you enable the following appender it will fill up your disk if you don't have a cleanup job!
   # This uses the updated rolling file appender from log4j-extras that supports a reliable time-based rolling policy.
   # See http://logging.apache.org/log4j/companions/extras/apidocs/org/apache/log4j/rolling/TimeBasedRollingPolicy.html
   # Add "DAILY" to flume.root.logger above if you want to use this
   log4j.appender.DAILY=org.apache.log4j.rolling.RollingFileAppender
   log4j.appender.DAILY.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
   log4j.appender.DAILY.rollingPolicy.ActiveFileName=${flume.log.dir}

   /$

   {flume.log.file}
   log4j.appender.DAILY.rollingPolicy.FileNamePattern=${flume.log.dir}/${flume.log.file}

   .%d

   {yyyy-MM-dd}

   log4j.appender.DAILY.layout=org.apache.log4j.PatternLayout
   log4j.appender.DAILY.layout.ConversionPattern=%d

   {dd MMM yyyy HH:mm:ss,SSS}

   %-5p [%t] (%C.%M:%L) %x - %m%n

1. console
2. Add "console" to flume.root.logger above if you want to use this
   log4j.appender.console=org.apache.log4j.ConsoleAppender
   log4j.appender.console.target=System.err
   log4j.appender.console.layout=org.apache.log4j.PatternLayout
   log4j.appender.console.layout.ConversionPattern=%d (%t) [%p - %l] %m%n

EDIT2 Here is the result of the ls -l command:

[root@localhost conf]# ls -l
total 64
-rwxr-xr-x. 1 root root 5289 Nov 30 12:23 agent_1.conf
-rwxr-xr-x. 1 cygnus cygnus 14481 Nov 5 15:45 agent.conf.template
-rwxr-xr-x. 1 root root 1732 Nov 30 09:29 cygnus_instance_1.conf
-rwxr-xr-x. 1 cygnus cygnus 1732 Nov 5 15:45 cygnus_instance.conf.template
-rwxr-xr-x. 1 cygnus cygnus 1197 Nov 5 15:45 flume-env.sh.template
-rwxr-xr-x. 1 root root 3597 Dec 7 12:15 grouping_rules.conf
-rwxr-xr-x. 1 cygnus cygnus 3597 Nov 5 15:45 grouping_rules.conf.template
-rwxr-xr-x. 1 cygnus cygnus 325 Nov 5 15:45 krb5.conf.template
-rwxr-xr-x. 1 cygnus cygnus 135 Nov 5 15:45 krb5_login.conf
-rwxr-xr-x. 1 cygnus cygnus 3063 Nov 5 15:45 log4j.properties
-rwxr-xr-x. 1 cygnus cygnus 3659 Nov 5 15:45 log4j.properties.template
-rwxr-xr-x. 1 cygnus cygnus 2195 Nov 5 15:45 README.md

EDIT 2.1 Here is the ownership of var/log/cygnus:

[root@localhost conf]# ls -l /var/log/cygnus
total 16
rw-rw-r-. 1 cygnus cygnus 14152 Dec 9 14:37 cygnus.log

EDIT 3 My newest log:

Warning: JAVA_HOME is not set!
+ exec /usr/bin/java -Xmx20m -Dflume.log.file=cygnus.log -cp '/usr/cygnus/conf:/usr/cygnus/lib/:/usr/cygnus/plugins.d/cygnus/lib/:/usr/cygnus/plugins.d/cygnus/libext/*' -Djava.library.path= com.telefonica.iot.cygnus.nodes.CygnusApplic$
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/cygnus/lib/cygnus-0.10.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
11 Dec 2015 11:16:13,101 ERROR [main] (com.telefonica.iot.cygnus.nodes.CygnusApplication.main:250) - A fatal error occurred while running. Exception follows. Details=The specified configuration file does not exist: /usr/cygnus/conf/agent.conf

If you are not familiar with FIWARE technologies, please ignore this question!