Details
-
Type:
extRequest
-
Status: Closed
-
Priority:
Major
-
Resolution: Done
-
Component/s: FINODEX
-
Labels:None
Description
Hi everyone,
I'm Carlos Corrales, part of the development team at SAPIN (FINODEX012). We
have been working hard in the application and now that it's finished and we
are doing tests with final users, we are improving our code. A couple of
months ago we were able to write data to our HDFS in the COSMOS Fiware
machine (cosmos.lab.fi-ware.org) with our user (idelgado). As we are not
using Cosmos for production right now (it will be used in the future), we
left it aside. Now that we are checking that everything works as expected,
we can see that our Cygnus connector is not able to write properly in HDFS.
We haven't changed anything in the configuration, but we receive the
following error when trying to write (flume.log file)
time=2015-09-13T15:11:36.744CEST | lvl=INFO |
trans=1442149853-658-0000000000 | function=persist | comp=Cygnus |
msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink[367] :
[hdfs-sink] Persisting data at OrionHDFSSink. HDFS file
(idelgado_SAPINnull), Data
(
)
time=2015-09-13T15:11:36.803CEST | lvl=ERROR |
trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] :
Persistence error (The unknownt/idelgado_SAPINnull directory could not be
created in HDFS. HttpFS response: 401 Unauthorized)
time=2015-09-13T15:11:36.804CEST | lvl=INFO |
trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[150] : An event
was put again in the channel (id=1444248959, ttl=0)
time=2015-09-13T15:11:36.804CEST | lvl=INFO |
trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[175] : Finishing
transaction (1442149853-658-0000000000)
As you can see, the error is 401 Unauthorized. However, we can access the
Cosmos machine using SSH, and the user/password in the cygnus.conf file. We
also checked that the IP address of the Cosmos machine hasn't changed.
Could you please help us? We don't know what else to check...
Best regards
–
Carlos Corrales
Cofounder Secmotic Innovation S.L.
http://www.secmotic.com
_______________________________________________
Fiware-finodex-coaching mailing list
Fiware-finodex-coaching@lists.fiware.org
https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
[Created via e-mail received from: Carlos Corrales <carlos@secmotic.com>]
Activity
Hi Francisco,
Yes, we were using cygnus 0.5-91... It is the default version when using
the image available in FIWARE lab (orion-psb-image-R4.2
<https://cloud.lab.fiware.org/#nova/images/7dd7a0cc-3e3b-4eb3-a50c-4a2e3e498df3>
).
After updating the version and configuring it properly, everything is
working again.
Thank you very much!
2015-09-15 9:45 GMT+02:00 MANUEL ESCRICHE VICENTE <
–
Carlos Corrales
Cofounder Secmotic Innovation S.L.
http://www.secmotic.com
_______________________________________________
Fiware-finodex-coaching mailing list
Fiware-finodex-coaching@lists.fiware.org
https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
Hi Carlos,
I will aggregate through this email all the other emails you sent.
Regarding the version, it is printed in the log at the very beginning of the execution. In addition, if you installed via RPM you can check for the installed version by typing:
$ rpm –qa | grep cygnus
Nevertheless, looking at your configuration, it seems you are using a very old version of Cygnus, or you are mixing parameters from different versions. I mean:
- Current version is 0.8.2
- The "cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink” part is previous to 0.8.0. From 0.8.0 it must be “com.telefonica.iot.cygnus.sinks.OrionCKANSink”
- The OAuth2 token was introduced in 0.8.2
My recommendation is you install the latest version from RPM (https://github.com/telefonicaid/fiware-cygnus/tree/master#section4.1) and you configure Cygnus following the guidelines appearing in the master or release-0.8.2 branches (they are synchronized):
https://github.com/telefonicaid/fiware-cygnus/blob/master/README.md
https://github.com/telefonicaid/fiware-cygnus/blob/release/0.8.2/README.md<https://github.com/telefonicaid/fiware-cygnus/tree/release/0.8.2>
Regarding the fiware-service and fiware-servicePath values, if they are not notified as http headers, then the default configured values in Cygnus’ HTTPSource are used. Again, it seems you are not using/configuring these values.
Regards,
Francisco
De: Carlos Corrales <carlos@secmotic.com<carlos@secmotic.com
>>
Fecha: lunes, 14 de septiembre de 2015, 18:22
Para: MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>
CC: JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com<juanjose.hierro@telefonica.com>>, "fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>" <fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>>, "jose@secmotic.com<jose@secmotic.com>" <jose@secmotic.com<jose@secmotic.com>>, Francisco Romero Bueno <francisco.romerobueno@telefonica.com<francisco.romerobueno@telefonica.com>>
Asunto: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
Hi everyone,
Thanks for your responses, we have been testing the whole day and trying to find out the issue. It is not a problem with neither the password, nor the oauth2 token. We are able to access using ssh user/password and the curl command proposed by Pasquale with Oauth2 token. This is what I received using the Oauth2 token curl proposed by Pasquale
curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado" -H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i"
{"FileStatuses":{"FileStatus":[
,
{"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0},
{"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0},
{"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0},
{"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0},
{"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0},
{"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0},
{"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0}]}}
Everything seems to be correct. However, using the same token and updating ORION, I receive the following error in Cygnus:
time=2015-09-14T18:10:44.419CEST | lvl=ERROR | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] : Persistence error (The org42/idelgado_tablanull directory could not be created in HDFS. HttpFS response: 401 Unauthorized)
time=2015-09-14T18:10:44.419CEST | lvl=WARN | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[153] : The event TTL has expired, it is no more re-injected in the channel (id=1444248959, ttl=0)
However, if I try to access the directory shown in the error, I'm able to do so
ssh idelgado@cosmos.lab.fi-ware.org<idelgado@cosmos.lab.fi-ware.org>
hadoop fs -ls /user/idelgado/org42/idelgado_tablanull
Found 1 items
rw-r---- 3 idelgado idelgado 1416 2015-07-01 13:11 /user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt
My cygnus.conf file content is:
- ============================================
- OrionHDFSSink configuration
- channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel - sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink - Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46 - port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free c\
hoice for inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000 - default username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado - default password for the default username
#cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX
cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i - HDFS backend type (webhdfs, httpfs or infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs - how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.hdfs-sink.attr_persistence = column - prefix for the database and table names, empty if no prefix is desired
cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla - Hive FQDN/IP address of the Hive server
cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46 - Hive port for Hive external table provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000
Could you please help us?
Thanks in advance!
2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>:
In this case I see both Pasquale (Coach) and Francisco (GE Owner) have reacted to the request.
In the normal flow GE owners enter into action upon request from Coaches.
Kind regards,
Manuel
From: JUAN JOSE HIERRO SUREDA
Sent: lunes, 14 de septiembre de 2015 0:37
To: Carlos Corrales
Cc: fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>; jose@secmotic.com<jose@secmotic.com>; FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA
Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list)
Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket.
You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution.
Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is.
Best regards,
– Juanjo
Great to hear that!
Regarding the image containing the bundle Orion+Cygnus, it must be updated for sure. Adding it to our backlog.
Regards,
Francisco
De: Carlos Corrales <carlos@secmotic.com<carlos@secmotic.com>>
Fecha: miércoles, 16 de septiembre de 2015, 11:11
Para: MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>
CC: Francisco Romero Bueno <francisco.romerobueno@telefonica.com<francisco.romerobueno@telefonica.com>>, JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com<juanjose.hierro@telefonica.com>>, "fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>" <fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>>, "jose@secmotic.com<jose@secmotic.com>" <jose@secmotic.com<jose@secmotic.com>>
Asunto: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
Hi Francisco,
Yes, we were using cygnus 0.5-91... It is the default version when using the image available in FIWARE lab (orion-psb-image-R4.2<https://cloud.lab.fiware.org/#nova/images/7dd7a0cc-3e3b-4eb3-a50c-4a2e3e498df3>).
After updating the version and configuring it properly, everything is working again.
Thank you very much!
2015-09-15 9:45 GMT+02:00 MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>:
Be aware there exist this issue in JIRA for this exchange: https://jira.fiware.org/browse/HELC-1025
From: FRANCISCO ROMERO BUENO
Sent: martes, 15 de septiembre de 2015 9:12
To: Carlos Corrales; MANUEL ESCRICHE VICENTE
Cc: JUAN JOSE HIERRO SUREDA; fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>; jose@secmotic.com<jose@secmotic.com>
Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
Hi Carlos,
I will aggregate through this email all the other emails you sent.
Regarding the version, it is printed in the log at the very beginning of the execution. In addition, if you installed via RPM you can check for the installed version by typing:
$ rpm –qa | grep cygnus
Nevertheless, looking at your configuration, it seems you are using a very old version of Cygnus, or you are mixing parameters from different versions. I mean:
- Current version is 0.8.2
- The "cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink” part is previous to 0.8.0. From 0.8.0 it must be “com.telefonica.iot.cygnus.sinks.OrionCKANSink”
- The OAuth2 token was introduced in 0.8.2
My recommendation is you install the latest version from RPM (https://github.com/telefonicaid/fiware-cygnus/tree/master#section4.1) and you configure Cygnus following the guidelines appearing in the master or release-0.8.2 branches (they are synchronized):
https://github.com/telefonicaid/fiware-cygnus/blob/master/README.md
https://github.com/telefonicaid/fiware-cygnus/blob/release/0.8.2/README.md<https://github.com/telefonicaid/fiware-cygnus/tree/release/0.8.2>
Regarding the fiware-service and fiware-servicePath values, if they are not notified as http headers, then the default configured values in Cygnus’ HTTPSource are used. Again, it seems you are not using/configuring these values.
Regards,
Francisco
De: Carlos Corrales <carlos@secmotic.com<carlos@secmotic.com>>
Fecha: lunes, 14 de septiembre de 2015, 18:22
Para: MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>
CC: JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com<juanjose.hierro@telefonica.com>>, "fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>" <fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>>, "jose@secmotic.com<jose@secmotic.com>" <jose@secmotic.com<jose@secmotic.com>>, Francisco Romero Bueno <francisco.romerobueno@telefonica.com<francisco.romerobueno@telefonica.com>>
Asunto: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
Hi everyone,
Thanks for your responses, we have been testing the whole day and trying to find out the issue. It is not a problem with neither the password, nor the oauth2 token. We are able to access using ssh user/password and the curl command proposed by Pasquale with Oauth2 token. This is what I received using the Oauth2 token curl proposed by Pasquale
curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado" -H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i"
{"FileStatuses":{"FileStatus":[
,
{"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0},
{"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0},
{"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0},
{"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0},
{"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0},
{"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0},
{"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0}]}}
Everything seems to be correct. However, using the same token and updating ORION, I receive the following error in Cygnus:
time=2015-09-14T18:10:44.419CEST | lvl=ERROR | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] : Persistence error (The org42/idelgado_tablanull directory could not be created in HDFS. HttpFS response: 401 Unauthorized)
time=2015-09-14T18:10:44.419CEST | lvl=WARN | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[153] : The event TTL has expired, it is no more re-injected in the channel (id=1444248959, ttl=0)
However, if I try to access the directory shown in the error, I'm able to do so
ssh idelgado@cosmos.lab.fi-ware.org<idelgado@cosmos.lab.fi-ware.org>
hadoop fs -ls /user/idelgado/org42/idelgado_tablanull
Found 1 items
rw-r---- 3 idelgado idelgado 1416 2015-07-01 13:11 /user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt
My cygnus.conf file content is:
- ============================================
- OrionHDFSSink configuration
- channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel - sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink - Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46 - port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free c\
hoice for inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000 - default username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado - default password for the default username
#cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX
cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i - HDFS backend type (webhdfs, httpfs or infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs - how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.hdfs-sink.attr_persistence = column - prefix for the database and table names, empty if no prefix is desired
cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla - Hive FQDN/IP address of the Hive server
cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46 - Hive port for Hive external table provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000
Could you please help us?
Thanks in advance!
2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>:
In this case I see both Pasquale (Coach) and Francisco (GE Owner) have reacted to the request.
In the normal flow GE owners enter into action upon request from Coaches.
Kind regards,
Manuel
From: JUAN JOSE HIERRO SUREDA
Sent: lunes, 14 de septiembre de 2015 0:37
To: Carlos Corrales
Cc: fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>; jose@secmotic.com<jose@secmotic.com>; FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA
Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list)
Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket.
You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution.
Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is.
Best regards,
– Juanjo
In this case I see both Pasquale (Coach) and Francisco (GE Owner) have reacted to the request.
In the normal flow GE owners enter into action upon request from Coaches.
Kind regards,
Manuel
From: JUAN JOSE HIERRO SUREDA
Sent: lunes, 14 de septiembre de 2015 0:37
To: Carlos Corrales
Cc: fiware-finodex-coaching@lists.fi-ware.org; jose@secmotic.com; FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA
Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list)
Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket.
You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution.
Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is.
Best regards,
– Juanjo
Be aware there exist this issue in JIRA for this exchange: https://jira.fiware.org/browse/HELC-1025
From: FRANCISCO ROMERO BUENO
Sent: martes, 15 de septiembre de 2015 9:12
To: Carlos Corrales; MANUEL ESCRICHE VICENTE
Cc: JUAN JOSE HIERRO SUREDA; fiware-finodex-coaching@lists.fi-ware.org; jose@secmotic.com
Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
Hi Carlos,
I will aggregate through this email all the other emails you sent.
Regarding the version, it is printed in the log at the very beginning of the execution. In addition, if you installed via RPM you can check for the installed version by typing:
$ rpm -qa | grep cygnus
Nevertheless, looking at your configuration, it seems you are using a very old version of Cygnus, or you are mixing parameters from different versions. I mean:
- Current version is 0.8.2
- The "cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink" part is previous to 0.8.0. From 0.8.0 it must be "com.telefonica.iot.cygnus.sinks.OrionCKANSink"
- The OAuth2 token was introduced in 0.8.2
My recommendation is you install the latest version from RPM (https://github.com/telefonicaid/fiware-cygnus/tree/master#section4.1) and you configure Cygnus following the guidelines appearing in the master or release-0.8.2 branches (they are synchronized):
https://github.com/telefonicaid/fiware-cygnus/blob/master/README.md
https://github.com/telefonicaid/fiware-cygnus/blob/release/0.8.2/README.md<https://github.com/telefonicaid/fiware-cygnus/tree/release/0.8.2>
Regarding the fiware-service and fiware-servicePath values, if they are not notified as http headers, then the default configured values in Cygnus' HTTPSource are used. Again, it seems you are not using/configuring these values.
Regards,
Francisco
De: Carlos Corrales <carlos@secmotic.com<carlos@secmotic.com>>
Fecha: lunes, 14 de septiembre de 2015, 18:22
Para: MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>
CC: JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com<juanjose.hierro@telefonica.com>>, "fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>" <fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>>, "jose@secmotic.com<jose@secmotic.com>" <jose@secmotic.com<jose@secmotic.com>>, Francisco Romero Bueno <francisco.romerobueno@telefonica.com<francisco.romerobueno@telefonica.com>>
Asunto: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
Hi everyone,
Thanks for your responses, we have been testing the whole day and trying to find out the issue. It is not a problem with neither the password, nor the oauth2 token. We are able to access using ssh user/password and the curl command proposed by Pasquale with Oauth2 token. This is what I received using the Oauth2 token curl proposed by Pasquale
curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado" -H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i"
{"FileStatuses":{"FileStatus":[
,
{"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0},
{"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0},
{"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0},
{"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0},
{"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0},
{"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0},
{"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0}]}}
Everything seems to be correct. However, using the same token and updating ORION, I receive the following error in Cygnus:
time=2015-09-14T18:10:44.419CEST | lvl=ERROR | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] : Persistence error (The org42/idelgado_tablanull directory could not be created in HDFS. HttpFS response: 401 Unauthorized)
time=2015-09-14T18:10:44.419CEST | lvl=WARN | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[153] : The event TTL has expired, it is no more re-injected in the channel (id=1444248959, ttl=0)
However, if I try to access the directory shown in the error, I'm able to do so
ssh idelgado@cosmos.lab.fi-ware.org<idelgado@cosmos.lab.fi-ware.org>
hadoop fs -ls /user/idelgado/org42/idelgado_tablanull
Found 1 items
rw-r---- 3 idelgado idelgado 1416 2015-07-01 13:11 /user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt
My cygnus.conf file content is:
- ============================================
- OrionHDFSSink configuration
- channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel - sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink - Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46 - port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free c\
hoice for inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000 - default username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado - default password for the default username
#cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX
cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i - HDFS backend type (webhdfs, httpfs or infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs - how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.hdfs-sink.attr_persistence = column - prefix for the database and table names, empty if no prefix is desired
cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla - Hive FQDN/IP address of the Hive server
cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46 - Hive port for Hive external table provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000
Could you please help us?
Thanks in advance!
2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>:
In this case I see both Pasquale (Coach) and Francisco (GE Owner) have reacted to the request.
In the normal flow GE owners enter into action upon request from Coaches.
Kind regards,
Manuel
From: JUAN JOSE HIERRO SUREDA
Sent: lunes, 14 de septiembre de 2015 0:37
To: Carlos Corrales
Cc: fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>; jose@secmotic.com<jose@secmotic.com>; FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA
Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance
I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list)
Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket.
You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution.
Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is.
Best regards,
– Juanjo
Hi,
I copy Francisco (Paco) Romero who should be able to provide some help on this. I guess he won't be able to work on the matter until tomorrow since support is not granted out of office working hours.
Nevertheless, I believe that it is rather important that you open an issue so that a ticket on our help-desk system is created and then follow-up of support is feasible.
You may create a ticket in our helpdesk simply by sending an email to either fiware-lab-help (if it happens to be an issue which is mostly related to FIWARE Lab or the global COSMOS instance setup on the FIWARE Lab) or fiware-tech-help (if it happens to be an issue mostly related with the technology, i.e., with COSMOS in general and not the particular instance of COSMOS running on the FIWARE Lab).
Best regards,
– Juanjo