When trying to persist an entity from Cygnus to Cosmos global instance it
fails.

Looking at the log file I see something like that:

"2015 15:31:50,006 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:273) -
>> GET
/webhdfs/v1/user/ms/def_serv/def_servpath/6_registervalues/6_registervalues.txt?op=getfilestatus&
user.name=ms HTTP/1.1 12 Nov 2015 15:31:50,006 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276) -
>> X-Auth-Token: xxxxxxxxxxxxxxxxxxx 12 Nov 2015 15:31:50,006 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276) -
>> Host: cosmos.lab.fiware.org:14000 12 Nov 2015 15:31:50,006 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276) -
>> Connection: Keep-Alive 12 Nov 2015 15:31:50,007 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276) -
>> User-Agent: Apache-HttpClient/4.2.1 (java 1.5) 12 Nov 2015 15:31:50,284
DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.close:169) - Connection
0.0.0.0:59834<->130.206.80.46:14000 closed 12 Nov 2015 15:31:50,285 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.client.DefaultRequestDirector.tryExecute:716) -
Closing the connection. 12 Nov 2015 15:31:50,285 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.conn.DefaultClientConnection.close:169) - Connection
0.0.0.0:59834<->130.206.80.46:14000 closed 12 Nov 2015 15:31:50,286 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.client.DefaultRequestDirector.tryExecute:723) - I/O
exception (java.net.SocketException) caught when processing request:
Connection reset 12 Nov 2015 15:31:50,287 DEBUG
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.http.impl.client.DefaultRequestDirector.tryExecute:728) -
Connection reset java.net.SocketException: Connection reset at
java.net.SocketInputStream.read(SocketInputStream.java:185) at
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
"

(I have obfuscated the token in the example above.)

for some reason the connection is closed.

Orion and Cygnus are installed on a fiware lab private instance in Spain2.

When trying to fire the same request from the command line from the Orion
machine, I get the same disconnected response.

curl -X GET "
http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/ms/def_serv/def_servpath/6_registervalues/6_registervalues.txt?op=getfilestatus&user.name=ms"
-H "X-Auth-Token: xxxxxxxxxxxxxxxxxxxxxx" curl: (56) Failure when receiving
data from the peer

But surprisingly, when doing the same request from the command line from
the Cosmos global instance, it works!

See:

[ms@cosmosmaster-gi ~]$ curl -X GET "
http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/ms/def_serv/def_servpath/6_registervalues/6_registervalues.txt?op=getfilestatus&user.name=ms"
-H "X-Auth-Token: xxxxxxxxxxxxxxxxx"
{"FileStatus":{"pathSuffix":"","type":"FILE","length":2,"owner":"ms","group":"ms","permission":"644","accessTime":1447332222908,"modificationTime":1447253745711,"blockSize":67108864,"replication":3}}
[ms@cosmosmaster-gi ~]$

Seems like Cosmos will not service connection arriving from a fiware lab
instance running on Spain2.

Does anyone knows what's the issue here, and how to correct this.

Thanks!

Menachem S.
On Nov 11, 2015 13:00, <fiware-incense-coaching-request@lists.fiware.org>
wrote:

Send Fiware-incense-coaching mailing list submissions to
fiware-incense-coaching@lists.fiware.org

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.fiware.org/listinfo/fiware-incense-coaching
or, via email, send a message with subject or body 'help' to
fiware-incense-coaching-request@lists.fiware.org

You can reach the person managing the list at
fiware-incense-coaching-owner@lists.fiware.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Fiware-incense-coaching digest..."

Today's Topics:

1. Heaving problems perssiting data from Cygnus to Cosmos global
instanse (Menachem Shtalhaim)

----------------------------------------------------------------------

Message: 1
Date: Tue, 10 Nov 2015 17:56:06 +0200
From: Menachem Shtalhaim <menachem.shtalhaim@n-join.com>
To: fiware-incense-coaching@lists.fiware.org
Subject: [Fiware-incense-coaching] Heaving problems perssiting data
from Cygnus to Cosmos global instanse
Message-ID:
<CAJgi6OaNek66JsYXizp7O2g=-9f8APSKbFO0d+wdQWDRgJeMQw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I have looked at the log and found the following:
10 Nov 2015 16:39:57,903 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persistAggregation:918) -
[hdfs-sink] Persisting data at OrionHDFSSink. HDFS file
(def_serv/def_servpath/8_registervalues/8_registervalues.txt), Data
(

)
10 Nov 2015 16:39:58,183 ERROR
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionSink.process:305) - Persistence
error (The /user/ms/def_serv/def_servpath/8_registervalues directory could
not be created in HDFS. Server response: 503 Service unavailable)
10 Nov 2015 16:39:58,183 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(com.telefonica.iot.cygnus.sinks.OrionSink.process:307) - Finishing
transaction (1447169873-477-0000000000)

Is there anything wrong with the configuration I am using?

Agent_1.conf:

1. Copyright 2014 Telef??nica Investigaci??n y Desarrollo, S.A.U
   #
2. This file is part of fiware-cygnus (FI-WARE project).
   #
3. fiware-cygnus is free software: you can redistribute it and/or modify it
   under the terms of the GNU Affero General
4. Public License as published by the Free Software Foundation, either
   version 3 of the License, or (at your option) any
5. later version.
6. fiware-cygnus is distributed in the hope that it will be useful, but
   WITHOUT ANY WARRANTY; without even the implied
7. warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
   GNU Affero General Public License for more
8. details.
   #
9. You should have received a copy of the GNU Affero General Public License
   along with fiware-cygnus. If not, see
10. http://www.gnu.org/licenses/.
    #
11. For those usages not covered by the GNU Affero General Public License
    please contact with iot_support at tid dot es

#=============================================

1. To be put in APACHE_FLUME_HOME/conf/agent.conf
   #
2. General configuration template explaining how to setup a sink of each of
   the available types (HDFS, CKAN, MySQL).

#=============================================

1. The next tree fields set the sources, sinks and channels used by Cygnus.
   You could use different names than the
2. ones suggested below, but in that case make sure you keep coherence in
   properties names along the configuration file.
3. Regarding sinks, you can use multiple types at the same time; the only
   requirement is to provide a channel for each
4. one of them (this example shows how to configure 3 sink types at the same
   time). Even, you can define more than one
5. sink of the same type and sharing the channel in order to improve the
   performance (this is like having
6. multi-threading).
   cygnusagent.sources = http-source
   cygnusagent.sinks = hdfs-sink
   cygnusagent.channels = hdfs-channel

#=============================================

1. source configuration
2. channel name where to write the notification events
   cygnusagent.sources.http-source.channels = hdfs-channel
3. source class, must not be changed
   cygnusagent.sources.http-source.type =
   org.apache.flume.source.http.HTTPSource
4. listening port the Flume source will use for receiving incoming
   notifications
   cygnusagent.sources.http-source.port = 5050
5. Flume handler that will parse the notifications, must not be changed
   cygnusagent.sources.http-source.handler =
   com.telefonica.iot.cygnus.handlers.OrionRestHandler
6. URL target
   cygnusagent.sources.http-source.handler.notification_target = /notify
7. Default service (service semantic depends on the persistence sink)
   cygnusagent.sources.http-source.handler.default_service = def_serv
8. Default service path (service path semantic depends on the persistence
   sink)
   cygnusagent.sources.http-source.handler.default_service_path = def_servpath
9. Number of channel re-injection retries before a Flume event is definitely
   discarded (-1 means infinite retries)
   cygnusagent.sources.http-source.handler.events_ttl = 10
10. Source interceptors, do not change
    cygnusagent.sources.http-source.interceptors = ts gi
11. TimestampInterceptor, do not change
    cygnusagent.sources.http-source.interceptors.ts.type = timestamp
12. GroupinInterceptor, do not change
    cygnusagent.sources.http-source.interceptors.gi.type =
    com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
13. Grouping rules for the GroupingInterceptor, put the right absolute path
    to the file if necessary
14. See the doc/design/interceptors document for more details
    cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file =
    /usr/cygnus/conf/grouping_rules.conf

1. ============================================
2. OrionHDFSSink configuration
3. channel name from where to read notification events
   cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
4. sink class, must not be changed
   cygnusagent.sinks.hdfs-sink.type =
   com.telefonica.iot.cygnus.sinks.OrionHDFSSink
   #? true if the grouping feature is enabled for this sink, false otherwise
   cygnusagent.sinks.hdfs-sink.enable_grouping = false
5. rest if the interaction with HDFS will be WebHDFS/HttpFS-based, binary if
   based on the Hadoop API
   cygnusagent.sinks.hdfs-sink.backend_impl = rest
6. Comma-separated list of FQDN/IP address regarding the HDFS Namenode
   endpoints
7. If you are using Kerberos authentication, then the usage of FQDNs instead
   of IP addresses is mandatory
   cygnusagent.sinks.hdfs-sink.hdfs_host = cosmos.lab.fiware.org
8. port of the HDFS service listening for persistence operations; 14000 for
   httpfs, 50070 for webhdfs
   cygnusagent.sinks.hdfs-sink.hdfs_port = 14000
9. username allowed to write in HDFS
   cygnusagent.sinks.hdfs-sink.hdfs_username = COSMOS_USER
   #? password for the above username; this is only required for Hive
   authentication
   cygnusagent.sinks.hdfs-sink.hdfs_password = COSMOS_PASSWORD
10. OAuth2 token for HDFS authentication
    #cygnusagent.sinks.hdfs-sink.oauth2_token = TOKEN
11. how the attributes are stored, available formats are json-row,
    json-column, csv-row and csv-column
    cygnusagent.sinks.hdfs-sink.file_format = json-column
    #? number of notifications to be included within a processing batch
    cygnusagent.sinks.hdfs-sink.batch_size = 100
12. timeout for batch accumulation
    cygunsagent.sinks.hdfs-sink.batch_timeout = 30
13. Hive enabling
    cygnusagent.sinks.hdfs-sink.hive = true
14. Hive server version, 1 or 2 (ignored if hive is false)
    cygnusagent.sinks.hdfs-sink.hive.server_version = 2
15. Hive FQDN/IP address of the Hive server (ignored if hive is false)
    cygnusagent.sinks.hdfs-sink.hive.host = cosmos.lab.fiware.org
16. Hive port for Hive external table provisioning (ignored if hive is false)
    cygnusagent.sinks.hdfs-sink.hive.port = 10000
17. Kerberos-based authentication enabling
    cygnusagent.sinks.hdfs-sink.krb5_auth = false
18. Kerberos username (ignored if krb5_auth is false)
    cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_user = krb5_username
19. Kerberos password (ignored if krb5_auth is false)
    cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_password = xxxxxxxxxxxxx
20. Kerberos login file (ignored if krb5_auth is false)
    cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_login_conf_file =
    /usr/cygnus/conf/krb5_login.conf
21. Kerberos configuration file (ignored if krb5_auth is false)
    cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_conf_file =
    /usr/cygnus/conf/krb5.conf

#=============================================

1. hdfs-channel configuration
2. channel type (must not be changed)
   cygnusagent.channels.hdfs-channel.type = memory
3. capacity of the channel
   cygnusagent.channels.hdfs-channel.capacity = 1000
4. amount of bytes that can be sent per transaction
   cygnusagent.channels.hdfs-channel.transactionCapacity = 100

CYGNUS_INSTANCE_1.CONF

#####
#

1. Configuration file for apache-flume
   #
   #####
2. Copyright 2014 Telefonica Investigaci??n y Desarrollo, S.A.U
   #
3. This file is part of fiware-cygnus (FI-WARE project).
   #
4. fiware-cygnus is free software: you can redistribute it and/or modify it
   under the terms of the GNU Affero General
5. Public License as published by the Free Software Foundation, either
   version 3 of the License, or (at your option) any
6. later version.
7. fiware-cygnus is distributed in the hope that it will be useful, but
   WITHOUT ANY WARRANTY; without even the implied
8. warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
   GNU Affero General Public License for more
9. details.
   #
10. You should have received a copy of the GNU Affero General Public License
    along with fiware-cygnus. If not, see
11. http://www.gnu.org/licenses/.
    #
12. For those usages not covered by the GNU Affero General Public License
    please contact with iot_support at tid dot es

1. Who to run cygnus as. Note that you may need to use root if you want
2. to run cygnus in a privileged port (<1024)
   CYGNUS_USER=cygnus

1. Where is the config folder
   CONFIG_FOLDER=/usr/cygnus/conf

1. Which is the config file
   CONFIG_FILE=/usr/cygnus/conf/agent_1.conf

1. Name of the agent. The name of the agent is not trivial, since it is the
   base for the Flume parameters
2. naming conventions, e.g. it appears in .sources.http-source.channels=...
   AGENT_NAME=cygnusagent

1. Name of the logfile located at /var/log/cygnus. It is important to put
   the extension '.log' in order to the log rotation works properly
   LOGFILE_NAME=cygnus.log

1. Administration port. Must be unique per instance
   ADMIN_PORT=8081

1. Polling interval (seconds) for the configuration reloading
   POLLING_INTERVAL=30

Thanks you!

–

Menachem Shtalhaim

menachem.shtalhaim@n-join.com <elkana@n-join.com> | +972-5
<%2B972-527-213941>47241959 | www.n-join.com

n-Join ? Maximizing industrial potential
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://lists.fiware.org/pipermail/fiware-incense-coaching/attachments/20151110/4b6d91e3/attachment-0001.html
>

------------------------------

Since January 1st, old domains won't be supported and messages sent to any
domain different to @lists.fiware.org will be lost.
Please, send your messages using the new domain (
Fiware-incense-coaching@lists.fiware.org) instead of the old one.
_______________________________________________
Fiware-incense-coaching mailing list
Fiware-incense-coaching@lists.fiware.org
https://lists.fiware.org/listinfo/fiware-incense-coaching

End of Fiware-incense-coaching Digest, Vol 13, Issue 3
******************************************************

Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost.
Please, send your messages using the new domain (Fiware-incense-coaching@lists.fiware.org) instead of the old one.
_______________________________________________
Fiware-incense-coaching mailing list
Fiware-incense-coaching@lists.fiware.org
https://lists.fiware.org/listinfo/fiware-incense-coaching
[Created via e-mail received from: Menachem Shtalhaim <menachem.shtalhaim@n-join.com>]