Details
-
Type: extRequest
-
Status: Closed
-
Priority: Major
-
Resolution: Done
-
Fix Version/s: 2021
-
Component/s: FIWARE-TECH-HELP
-
Labels:None
-
Sender Email:
-
HD-Chapter:Data
-
HD-Enabler:Cosmos
Description
When trying to persist an entity from Cygnus to Cosmos global instance it
fails.
Looking at the log file I see something like that:
"2015 15:31:50,006 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:273) -
>> GET
/webhdfs/v1/user/ms/def_serv/def_servpath/6_registervalues/6_registervalues.txt?op=getfilestatus&
user.name=ms HTTP/1.1 12 Nov 2015 15:31:50,006 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276) -
>> X-Auth-Token: xxxxxxxxxxxxxxxxxxx 12 Nov 2015 15:31:50,006 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276) -
>> Host: cosmos.lab.fiware.org:14000 12 Nov 2015 15:31:50,006 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276) -
>> Connection: Keep-Alive 12 Nov 2015 15:31:50,007 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276) -
>> User-Agent: Apache-HttpClient/4.2.1 (java 1.5) 12 Nov 2015 15:31:50,284
DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.close:169) - Connection
0.0.0.0:59834<->130.206.80.46:14000 closed 12 Nov 2015 15:31:50,285 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.client.DefaultRequestDirector.tryExecute:716) -
Closing the connection. 12 Nov 2015 15:31:50,285 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.close:169) - Connection
0.0.0.0:59834<->130.206.80.46:14000 closed 12 Nov 2015 15:31:50,286 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.client.DefaultRequestDirector.tryExecute:723) - I/O
exception (java.net.SocketException) caught when processing request:
Connection reset 12 Nov 2015 15:31:50,287 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.client.DefaultRequestDirector.tryExecute:728) -
Connection reset java.net.SocketException: Connection reset at
java.net.SocketInputStream.read(SocketInputStream.java:185) at
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
"
(I have obfuscated the token in the example above.)
for some reason the connection is closed.
Orion and Cygnus are installed on a fiware lab private instance in Spain2.
When trying to fire the same request from the command line from the Orion
machine, I get the same disconnected response.
curl -X GET "
http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/ms/def_serv/def_servpath/6_registervalues/6_registervalues.txt?op=getfilestatus&user.name=ms"
-H "X-Auth-Token: xxxxxxxxxxxxxxxxxxxxxx" curl: (56) Failure when receiving
data from the peer
But surprisingly, when doing the same request from the command line from
the Cosmos global instance, it works!
See:
[ms@cosmosmaster-gi ~]$ curl -X GET "
http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/ms/def_serv/def_servpath/6_registervalues/6_registervalues.txt?op=getfilestatus&user.name=ms"
-H "X-Auth-Token: xxxxxxxxxxxxxxxxx"
{"FileStatus":{"pathSuffix":"","type":"FILE","length":2,"owner":"ms","group":"ms","permission":"644","accessTime":1447332222908,"modificationTime":1447253745711,"blockSize":67108864,"replication":3}}
[ms@cosmosmaster-gi ~]$
Seems like Cosmos will not service connection arriving from a fiware lab
instance running on Spain2.
Does anyone knows what's the issue here, and how to correct this.
Thanks!
Menachem S.
On Nov 11, 2015 13:00, <fiware-incense-coaching-request@lists.fiware.org>
wrote:
Send Fiware-incense-coaching mailing list submissions to
fiware-incense-coaching@lists.fiware.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.fiware.org/listinfo/fiware-incense-coaching
or, via email, send a message with subject or body 'help' to
fiware-incense-coaching-request@lists.fiware.org
You can reach the person managing the list at
fiware-incense-coaching-owner@lists.fiware.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Fiware-incense-coaching digest..."
Today's Topics:
1. Heaving problems perssiting data from Cygnus to Cosmos global
instanse (Menachem Shtalhaim)
----------------------------------------------------------------------
Message: 1
Date: Tue, 10 Nov 2015 17:56:06 +0200
From: Menachem Shtalhaim <menachem.shtalhaim@n-join.com>
To: fiware-incense-coaching@lists.fiware.org
Subject: [Fiware-incense-coaching] Heaving problems perssiting data
from Cygnus to Cosmos global instanse
Message-ID:
<CAJgi6OaNek66JsYXizp7O2g=-9f8APSKbFO0d+wdQWDRgJeMQw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
I have looked at the log and found the following:
10 Nov 2015 16:39:57,903 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persistAggregation:918) -
[hdfs-sink] Persisting data at OrionHDFSSink. HDFS file
(def_serv/def_servpath/8_registervalues/8_registervalues.txt), Data
(
)
10 Nov 2015 16:39:58,183 ERROR
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionSink.process:305) - Persistence
error (The /user/ms/def_serv/def_servpath/8_registervalues directory could
not be created in HDFS. Server response: 503 Service unavailable)
10 Nov 2015 16:39:58,183 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionSink.process:307) - Finishing
transaction (1447169873-477-0000000000)
Is there anything wrong with the configuration I am using?
Agent_1.conf:
- Copyright 2014 Telef??nica Investigaci??n y Desarrollo, S.A.U
# - This file is part of fiware-cygnus (FI-WARE project).
# - fiware-cygnus is free software: you can redistribute it and/or modify it
under the terms of the GNU Affero General - Public License as published by the Free Software Foundation, either
version 3 of the License, or (at your option) any - later version.
- fiware-cygnus is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied - warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more - details.
# - You should have received a copy of the GNU Affero General Public License
along with fiware-cygnus. If not, see - http://www.gnu.org/licenses/.
# - For those usages not covered by the GNU Affero General Public License
please contact with iot_support at tid dot es
#=============================================
- To be put in APACHE_FLUME_HOME/conf/agent.conf
# - General configuration template explaining how to setup a sink of each of
the available types (HDFS, CKAN, MySQL).
#=============================================
- The next tree fields set the sources, sinks and channels used by Cygnus.
You could use different names than the - ones suggested below, but in that case make sure you keep coherence in
properties names along the configuration file. - Regarding sinks, you can use multiple types at the same time; the only
requirement is to provide a channel for each - one of them (this example shows how to configure 3 sink types at the same
time). Even, you can define more than one - sink of the same type and sharing the channel in order to improve the
performance (this is like having - multi-threading).
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink
cygnusagent.channels = hdfs-channel
#=============================================
- source configuration
- channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel - source class, must not be changed
cygnusagent.sources.http-source.type =
org.apache.flume.source.http.HTTPSource - listening port the Flume source will use for receiving incoming
notifications
cygnusagent.sources.http-source.port = 5050 - Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler =
com.telefonica.iot.cygnus.handlers.OrionRestHandler - URL target
cygnusagent.sources.http-source.handler.notification_target = /notify - Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = def_serv - Default service path (service path semantic depends on the persistence
sink)
cygnusagent.sources.http-source.handler.default_service_path = def_servpath - Number of channel re-injection retries before a Flume event is definitely
discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10 - Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts gi - TimestampInterceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp - GroupinInterceptor, do not change
cygnusagent.sources.http-source.interceptors.gi.type =
com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder - Grouping rules for the GroupingInterceptor, put the right absolute path
to the file if necessary - See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file =
/usr/cygnus/conf/grouping_rules.conf
- ============================================
- OrionHDFSSink configuration
- channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel - sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type =
com.telefonica.iot.cygnus.sinks.OrionHDFSSink
#? true if the grouping feature is enabled for this sink, false otherwise
cygnusagent.sinks.hdfs-sink.enable_grouping = false - rest if the interaction with HDFS will be WebHDFS/HttpFS-based, binary if
based on the Hadoop API
cygnusagent.sinks.hdfs-sink.backend_impl = rest - Comma-separated list of FQDN/IP address regarding the HDFS Namenode
endpoints - If you are using Kerberos authentication, then the usage of FQDNs instead
of IP addresses is mandatory
cygnusagent.sinks.hdfs-sink.hdfs_host = cosmos.lab.fiware.org - port of the HDFS service listening for persistence operations; 14000 for
httpfs, 50070 for webhdfs
cygnusagent.sinks.hdfs-sink.hdfs_port = 14000 - username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.hdfs_username = COSMOS_USER
#? password for the above username; this is only required for Hive
authentication
cygnusagent.sinks.hdfs-sink.hdfs_password = COSMOS_PASSWORD - OAuth2 token for HDFS authentication
#cygnusagent.sinks.hdfs-sink.oauth2_token = TOKEN - how the attributes are stored, available formats are json-row,
json-column, csv-row and csv-column
cygnusagent.sinks.hdfs-sink.file_format = json-column
#? number of notifications to be included within a processing batch
cygnusagent.sinks.hdfs-sink.batch_size = 100 - timeout for batch accumulation
cygunsagent.sinks.hdfs-sink.batch_timeout = 30 - Hive enabling
cygnusagent.sinks.hdfs-sink.hive = true - Hive server version, 1 or 2 (ignored if hive is false)
cygnusagent.sinks.hdfs-sink.hive.server_version = 2 - Hive FQDN/IP address of the Hive server (ignored if hive is false)
cygnusagent.sinks.hdfs-sink.hive.host = cosmos.lab.fiware.org - Hive port for Hive external table provisioning (ignored if hive is false)
cygnusagent.sinks.hdfs-sink.hive.port = 10000 - Kerberos-based authentication enabling
cygnusagent.sinks.hdfs-sink.krb5_auth = false - Kerberos username (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_user = krb5_username - Kerberos password (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_password = xxxxxxxxxxxxx - Kerberos login file (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_login_conf_file =
/usr/cygnus/conf/krb5_login.conf - Kerberos configuration file (ignored if krb5_auth is false)
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_conf_file =
/usr/cygnus/conf/krb5.conf
#=============================================
- hdfs-channel configuration
- channel type (must not be changed)
cygnusagent.channels.hdfs-channel.type = memory - capacity of the channel
cygnusagent.channels.hdfs-channel.capacity = 1000 - amount of bytes that can be sent per transaction
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
CYGNUS_INSTANCE_1.CONF
#####
#
- Configuration file for apache-flume
#
##### - Copyright 2014 Telefonica Investigaci??n y Desarrollo, S.A.U
# - This file is part of fiware-cygnus (FI-WARE project).
# - fiware-cygnus is free software: you can redistribute it and/or modify it
under the terms of the GNU Affero General - Public License as published by the Free Software Foundation, either
version 3 of the License, or (at your option) any - later version.
- fiware-cygnus is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied - warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more - details.
# - You should have received a copy of the GNU Affero General Public License
along with fiware-cygnus. If not, see - http://www.gnu.org/licenses/.
# - For those usages not covered by the GNU Affero General Public License
please contact with iot_support at tid dot es
- Who to run cygnus as. Note that you may need to use root if you want
- to run cygnus in a privileged port (<1024)
CYGNUS_USER=cygnus
- Where is the config folder
CONFIG_FOLDER=/usr/cygnus/conf
- Which is the config file
CONFIG_FILE=/usr/cygnus/conf/agent_1.conf
- Name of the agent. The name of the agent is not trivial, since it is the
base for the Flume parameters - naming conventions, e.g. it appears in .sources.http-source.channels=...
AGENT_NAME=cygnusagent
- Name of the logfile located at /var/log/cygnus. It is important to put
the extension '.log' in order to the log rotation works properly
LOGFILE_NAME=cygnus.log
- Administration port. Must be unique per instance
ADMIN_PORT=8081
- Polling interval (seconds) for the configuration reloading
POLLING_INTERVAL=30
Thanks you!
–
Menachem Shtalhaim
menachem.shtalhaim@n-join.com <elkana@n-join.com> | +972-5
<%2B972-527-213941>47241959 | www.n-join.com
n-Join ? Maximizing industrial potential
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://lists.fiware.org/pipermail/fiware-incense-coaching/attachments/20151110/4b6d91e3/attachment-0001.html
>
------------------------------
Since January 1st, old domains won't be supported and messages sent to any
domain different to @lists.fiware.org will be lost.
Please, send your messages using the new domain (
Fiware-incense-coaching@lists.fiware.org) instead of the old one.
_______________________________________________
Fiware-incense-coaching mailing list
Fiware-incense-coaching@lists.fiware.org
https://lists.fiware.org/listinfo/fiware-incense-coaching
End of Fiware-incense-coaching Digest, Vol 13, Issue 3
******************************************************
Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost.
Please, send your messages using the new domain (Fiware-incense-coaching@lists.fiware.org) instead of the old one.
_______________________________________________
Fiware-incense-coaching mailing list
Fiware-incense-coaching@lists.fiware.org
https://lists.fiware.org/listinfo/fiware-incense-coaching
[Created via e-mail received from: Menachem Shtalhaim <menachem.shtalhaim@n-join.com>]
Issue Links
- relates to
-
HELC-1132 FIWARE.Request.Coach.INCENSe.Cygnus can not persist data on Cosmos global instance
- Closed
Activity
Transition | Time In Source Status | Execution Times | Last Executer | Last Execution Date | |||||
---|---|---|---|---|---|---|---|---|---|
|
6m 41s | 1 | Pasquale Andriani | 19/Nov/15 2:50 PM | |||||
|
3d 16h 32m | 1 | Francisco Romero | 23/Nov/15 7:23 AM | |||||
|
2s | 1 | Francisco Romero | 23/Nov/15 7:23 AM |