Details
-
Type:
extRequest
-
Status: Closed
-
Priority:
Major
-
Resolution: Done
-
Component/s: INCENSe
-
Labels:None
Description
I have looked at the log and found the following:
10 Nov 2015 16:39:57,903 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persistAggregation:918) -
[hdfs-sink] Persisting data at OrionHDFSSink. HDFS file
(def_serv/def_servpath/8_registervalues/8_registervalues.txt), Data
(
)
10 Nov 2015 16:39:58,183 ERROR
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionSink.process:305) - Persistence
error (The /user/ms/def_serv/def_servpath/8_registervalues directory could
not be created in HDFS. Server response: 503 Service unavailable)
10 Nov 2015 16:39:58,183 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionSink.process:307) - Finishing
transaction (1447169873-477-0000000000)
Is there anything wrong with the configuration I am using?
Agent_1.conf:
- Copyright 2014 Telefֳ³nica Investigaciֳ³n y Desarrollo, S.A.U
# - This file is part of fiware-cygnus (FI-WARE project).
# - fiware-cygnus is free software: you can redistribute it and/or modify it
under the terms of the GNU Affero General - Public License as published by the Free Software Foundation, either
version 3 of the License, or (at your option) any - later version.
- fiware-cygnus is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied - warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more - details.
# - You should have received a copy of the GNU Affero General Public License
along with fiware-cygnus. If not, see - http://www.gnu.org/licenses/.
# - For those usages not covered by the GNU Affero General Public License
please contact with iot_support at tid dot es
#=============================================
- To be put in APACHE_FLUME_HOME/conf/agent.conf
# - General configuration template explaining how to setup a sink of each of
the available types (HDFS, CKAN, MySQL).
#=============================================
- The next tree fields set the sources, sinks and channels used by Cygnus.
You could use different names than the - ones suggested below, but in that case make sure you keep coherence in
properties names along the configuration file. - Regarding sinks, you can use multiple types at the same time; the only
requirement is to provide a channel for each - one of them (this example shows how to configure 3 sink types at the same
time). Even, you can define more than one - sink of the same type and sharing the channel in order to improve the
performance (this is like having - multi-threading).
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink
cygnusagent.channels = hdfs-channel
#=============================================
- source configuration
- channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel - source class, must not be changed
cygnusagent.sources.http-source.type =
org.apache.flume.source.http.HTTPSource - listening port the Flume source will use for receiving incoming
notifications
cygnusagent.sources.http-source.port = 5050 - Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler =
com.telefonica.iot.cygnus.handlers.OrionRestHandler - URL target
cygnusagent.sources.http-source.handler.notification_target = /notify - Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = def_serv - Default service path (service path semantic depends on the persistence
sink)
cygnusagent.sources.http-source.handler.default_service_path = def_servpath - Number of channel re-injection retries before a Flume event is definitely
discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10 - Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts gi - TimestampInterceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp - GroupinInterceptor, do not change
cygnusagent.sources.http-source.interceptors.gi.type =
com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder - Grouping rules for the GroupingInterceptor, put the right absolute path
to the file if necessary - See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file =
/usr/cygnus/conf/grouping_rules.conf
- ============================================
- OrionHDFSSink configuration
- channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel - sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type =
com.telefonica.iot.cygnus.sinks.OrionHDFSSink
#ֲ true if the grouping feature is enabled for this sink, false otherwise
cygnusagent.sinks.hdfs-sink.enable_grouping = false - rest if the interaction with HDFS will be WebHDFS/HttpFS-based, binary if
based on the Hadoop API
cygnusagent.sinks.hdfs-sink.backend_impl = rest - Comma-separated list of FQDN/IP address regarding the HDFS Namenode
endpoints - If you are using Kerberos authentication, then the usage of FQDNs instead
of IP addresses is mandatory
cygnusagent.sinks.hdfs-sink.hdfs_host = cosmos.lab.fiware.org - port of the HDFS service listening for persistence operations; 14000 for
httpfs, 50070 for webhdfs
cygnusagent.sinks.hdfs-sink.hdfs_port = 14000 - username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.hdfs_username = COSMOS_USER
#ֲ password for the above username; this is only required for Hive
authentication
cygnusagent.sinks.hdfs-sink.hdfs_password = COSMOS_PASSWORD - OAuth2 token for HDFS authentication
#cygnusagent.sinks.hdfs-sink.oauth2_token = TOKEN - how the attributes are stored, available formats are json-row,
json-column, csv-row and csv-column
cygnusagent.sinks.hdfs-sink.file_format = json-column
#ֲ number of notifications to be included within a processing batch
cygnusagent.sinks.hdfs-sink.batch_size = 100 - timeout for batch accumulation
cygunsagent.sinks.hdfs-sink.batch_timeout = 30 - Hive enabling
cygnusagent.sinks.hdfs-sink.hive = true - Hive server version, 1 or 2 (ignored if hive is false)
cygnusagent.sinks.hdfs-sink.hive.server_version = 2 - Hive FQDN/IP address of the Hive server (ignored if hive is false)
cygnusagent.sinks.hdfs-sink.hive.host = cosmos.lab.fiware.org - Hive port for Hive external table provisioning (ignored if hive is false)
cygnusagent.sinks.hdfs-sink.hive.port = 10000 - Kerberos-based authentication enabling
cygnusagent.sinks.hdfs-sink.krb5_auth = false - Kerberos username (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_user = krb5_username - Kerberos password (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_password = xxxxxxxxxxxxx - Kerberos login file (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_login_conf_file =
/usr/cygnus/conf/krb5_login.conf - Kerberos configuration file (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_conf_file =
/usr/cygnus/conf/krb5.conf
#=============================================
- hdfs-channel configuration
- channel type (must not be changed)
cygnusagent.channels.hdfs-channel.type = memory - capacity of the channel
cygnusagent.channels.hdfs-channel.capacity = 1000 - amount of bytes that can be sent per transaction
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
CYGNUS_INSTANCE_1.CONF
#####
#
- Configuration file for apache-flume
#
##### - Copyright 2014 Telefonica Investigaciֳ³n y Desarrollo, S.A.U
# - This file is part of fiware-cygnus (FI-WARE project).
# - fiware-cygnus is free software: you can redistribute it and/or modify it
under the terms of the GNU Affero General - Public License as published by the Free Software Foundation, either
version 3 of the License, or (at your option) any - later version.
- fiware-cygnus is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied - warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more - details.
# - You should have received a copy of the GNU Affero General Public License
along with fiware-cygnus. If not, see - http://www.gnu.org/licenses/.
# - For those usages not covered by the GNU Affero General Public License
please contact with iot_support at tid dot es
- Who to run cygnus as. Note that you may need to use root if you want
- to run cygnus in a privileged port (<1024)
CYGNUS_USER=cygnus
- Where is the config folder
CONFIG_FOLDER=/usr/cygnus/conf
- Which is the config file
CONFIG_FILE=/usr/cygnus/conf/agent_1.conf
- Name of the agent. The name of the agent is not trivial, since it is the
base for the Flume parameters - naming conventions, e.g. it appears in .sources.http-source.channels=...
AGENT_NAME=cygnusagent
- Name of the logfile located at /var/log/cygnus. It is important to put
the extension '.log' in order to the log rotation works properly
LOGFILE_NAME=cygnus.log
- Administration port. Must be unique per instance
ADMIN_PORT=8081
- Polling interval (seconds) for the configuration reloading
POLLING_INTERVAL=30
Thanks you!
–
Menachem Shtalhaim
menachem.shtalhaim@n-join.com <elkana@n-join.com> | +972-5
<%2B972-527-213941>47241959 | www.n-join.com
n-Join – Maximizing industrial potential
Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost.
Please, send your messages using the new domain (Fiware-incense-coaching@lists.fiware.org) instead of the old one.
_______________________________________________
Fiware-incense-coaching mailing list
Fiware-incense-coaching@lists.fiware.org
https://lists.fiware.org/listinfo/fiware-incense-coaching
[Created via e-mail received from: Menachem Shtalhaim <menachem.shtalhaim@n-join.com>]
duplicated of
HELC-1132