Uploaded image for project: 'Help-Coaches-Desk'
  1. Help-Coaches-Desk
  2. HELC-1025

FIWARE.Request.Coach.FINODEX.FIWARE.Request.Lab.Data.BigData-Analysis.CygnusPersistenceError

        Details

        • Type: extRequest
        • Status: Closed
        • Priority: Major
        • Resolution: Done
        • Component/s: FINODEX
        • Labels:
          None

          Description

          Hi everyone,

          I'm Carlos Corrales, part of the development team at SAPIN (FINODEX012). We
          have been working hard in the application and now that it's finished and we
          are doing tests with final users, we are improving our code. A couple of
          months ago we were able to write data to our HDFS in the COSMOS Fiware
          machine (cosmos.lab.fi-ware.org) with our user (idelgado). As we are not
          using Cosmos for production right now (it will be used in the future), we
          left it aside. Now that we are checking that everything works as expected,
          we can see that our Cygnus connector is not able to write properly in HDFS.
          We haven't changed anything in the configuration, but we receive the
          following error when trying to write (flume.log file)

          time=2015-09-13T15:11:36.744CEST | lvl=INFO |
          trans=1442149853-658-0000000000 | function=persist | comp=Cygnus |
          msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink[367] :
          [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file
          (idelgado_SAPINnull), Data
          (

          {"recvTime":"2015-09-13T15:11:00.413","age":"25", "age_md":[],"exercisetype":"Light", "exercisetype_md":[],"hr":"36", "hr_md":[],"id_session":"1234", "id_session_md":[]}

          )
          time=2015-09-13T15:11:36.803CEST | lvl=ERROR |
          trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
          msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] :
          Persistence error (The unknownt/idelgado_SAPINnull directory could not be
          created in HDFS. HttpFS response: 401 Unauthorized)
          time=2015-09-13T15:11:36.804CEST | lvl=INFO |
          trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
          msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[150] : An event
          was put again in the channel (id=1444248959, ttl=0)
          time=2015-09-13T15:11:36.804CEST | lvl=INFO |
          trans=1442149853-658-0000000000 | function=process | comp=Cygnus |
          msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[175] : Finishing
          transaction (1442149853-658-0000000000)

          As you can see, the error is 401 Unauthorized. However, we can access the
          Cosmos machine using SSH, and the user/password in the cygnus.conf file. We
          also checked that the IP address of the Cosmos machine hasn't changed.

          Could you please help us? We don't know what else to check...

          Best regards

          –
          Carlos Corrales
          Cofounder Secmotic Innovation S.L.
          http://www.secmotic.com

          _______________________________________________
          Fiware-finodex-coaching mailing list
          Fiware-finodex-coaching@lists.fiware.org
          https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
          [Created via e-mail received from: Carlos Corrales <carlos@secmotic.com>]

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            juanjo Juanjo Hierro added a comment -

            Hi,

              I copy Francisco (Paco) Romero who should be able to provide some help on this.   I guess he won't be able to work on the matter until tomorrow since support is not granted out of office working hours.

              Nevertheless, I believe that it is rather important that you open an issue so that a ticket on our help-desk system is created and then follow-up of support is feasible.

              You may create a ticket in our helpdesk simply by sending an email to either fiware-lab-help (if it happens to be an issue which is mostly related to FIWARE Lab or the global COSMOS instance setup on the FIWARE Lab) or fiware-tech-help (if it happens to be an issue mostly related with the technology, i.e., with COSMOS in general and not the particular instance of COSMOS running on the FIWARE Lab).

              Best regards,

            – Juanjo

            Show
            juanjo Juanjo Hierro added a comment - Hi,   I copy Francisco (Paco) Romero who should be able to provide some help on this.   I guess he won't be able to work on the matter until tomorrow since support is not granted out of office working hours.   Nevertheless, I believe that it is rather important that you open an issue so that a ticket on our help-desk system is created and then follow-up of support is feasible.   You may create a ticket in our helpdesk simply by sending an email to either fiware-lab-help (if it happens to be an issue which is mostly related to FIWARE Lab or the global COSMOS instance setup on the FIWARE Lab) or fiware-tech-help (if it happens to be an issue mostly related with the technology, i.e., with COSMOS in general and not the particular instance of COSMOS running on the FIWARE Lab).   Best regards, – Juanjo
            Hide
            Permalink
            fw.ext.user FW External User added a comment -

            Hi Juanjo,

            Thanks for you feedback, we sent it today because we wanted you to answer
            it tomorrow morning .

            As you can see, a ticket has been created in Jira (
            https://jira.fiware.org/browse/HELC-1025). I used the
            fiware-finodex-coaching list, because we are a FINODEX project and they
            told us to do so. Should we create another ticket in the fiware-tech-help
            list also?

            Best regards

            –
            Carlos Corrales
            Cofounder Secmotic Innovation S.L.
            http://www.secmotic.com

            _______________________________________________
            Fiware-finodex-coaching mailing list
            Fiware-finodex-coaching@lists.fiware.org
            https://lists.fiware.org/listinfo/fiware-finodex-coaching-new

            Show
            fw.ext.user FW External User added a comment - Hi Juanjo, Thanks for you feedback, we sent it today because we wanted you to answer it tomorrow morning . As you can see, a ticket has been created in Jira ( https://jira.fiware.org/browse/HELC-1025 ). I used the fiware-finodex-coaching list, because we are a FINODEX project and they told us to do so. Should we create another ticket in the fiware-tech-help list also? Best regards – Carlos Corrales Cofounder Secmotic Innovation S.L. http://www.secmotic.com _______________________________________________ Fiware-finodex-coaching mailing list Fiware-finodex-coaching@lists.fiware.org https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
            Hide
            Permalink
            fw.ext.user FW External User added a comment -

            Hi Carlos,

            I do not know if this will help. The same thing happened to us.
            http://stackoverflow.com/questions/31310111/oauth2-in-cygnus/31312196#31312196

            Regards

            firma
            *SensoWave
            Ignacio Gómez Maqueda*
            *
            *
            Director Técnico

            móvil: + 34 644350613

            _imaqueda@sensowave.es <candres@sensowave.es>_

            @queronea

            _______________________________________________
            Fiware-finodex-coaching mailing list
            Fiware-finodex-coaching@lists.fiware.org
            https://lists.fiware.org/listinfo/fiware-finodex-coaching-new

            Show
            fw.ext.user FW External User added a comment - Hi Carlos, I do not know if this will help. The same thing happened to us. http://stackoverflow.com/questions/31310111/oauth2-in-cygnus/31312196#31312196 Regards firma *SensoWave Ignacio Gómez Maqueda* * * Director Técnico móvil: + 34 644350613 _ imaqueda @sensowave.es < candres@sensowave.es >_ @queronea _______________________________________________ Fiware-finodex-coaching mailing list Fiware-finodex-coaching@lists.fiware.org https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
            Hide
            Permalink
            juanjo Juanjo Hierro added a comment -

              I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list)

              Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket.

              You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution.

              Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is.

              Best regards,

            – Juanjo

            Show
            juanjo Juanjo Hierro added a comment -   I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list)   Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket.   You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution.   Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is.   Best regards, – Juanjo
            Hide
            Permalink
            Stefano Stefano De Panfilis added a comment -

            dear carlos,

            in fact, as an sme working in finodex you did correctly.
            it will be your fiware coach (pasquale vitale in cc) that if not able to
            solve himself the issue, and i guess this time he will not, he will turn
            the ticket to the ge owner (francisco romero as indicated by juanjo).

            ciao,
            stefano

            –
            Stefano De Panfilis
            Chief Innovation Officer
            Engineering Ingegneria Informatica S.p.A.
            via Riccardo Morandi 32
            00148 Roma
            Italy

            tel (direct): +39-06-8759-4253
            tel (secr.): +39-068307-4513
            fax: +39-068307-4200
            cell: +39-335-7542-567
            skype: depa01
            twitter: @depa01

            _______________________________________________
            Fiware-finodex-coaching mailing list
            Fiware-finodex-coaching@lists.fiware.org
            https://lists.fiware.org/listinfo/fiware-finodex-coaching-new

            Show
            Stefano Stefano De Panfilis added a comment - dear carlos, in fact, as an sme working in finodex you did correctly. it will be your fiware coach (pasquale vitale in cc) that if not able to solve himself the issue, and i guess this time he will not, he will turn the ticket to the ge owner (francisco romero as indicated by juanjo). ciao, stefano – Stefano De Panfilis Chief Innovation Officer Engineering Ingegneria Informatica S.p.A. via Riccardo Morandi 32 00148 Roma Italy tel (direct): +39-06-8759-4253 tel (secr.): +39-068307-4513 fax: +39-068307-4200 cell: +39-335-7542-567 skype: depa01 twitter: @depa01 _______________________________________________ Fiware-finodex-coaching mailing list Fiware-finodex-coaching@lists.fiware.org https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
            Hide
            Permalink
            pasquale.vitale Pasquale Vitale added a comment -

            see the helc 1026

            Show
            pasquale.vitale Pasquale Vitale added a comment - see the helc 1026
            Hide
            Permalink
            fw.ext.user FW External User added a comment -

            Hi everyone,

            Thanks for your responses, we have been testing the whole day and trying to
            find out the issue. It is not a problem with neither the password, nor the
            oauth2 token. We are able to access using ssh user/password and the curl
            command proposed by Pasquale with Oauth2 token. This is what I received
            using the Oauth2 token curl proposed by Pasquale

            curl -X GET "
            http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado"
            -H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i"
            {"FileStatuses":{"FileStatus":[

            {"pathSuffix":".staging","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"700","accessTime":0,"modificationTime":1437059783102,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0}

            ]}}

            Everything seems to be correct. However, using the same token and updating
            ORION, I receive the following error in Cygnus:

            time=2015-09-14T18:10:44.419CEST | lvl=ERROR |
            trans=1442246992-282-0000000000 | function=process | comp=Cygnus |
            msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] :
            Persistence error (The org42/idelgado_tablanull directory could not be
            created in HDFS. HttpFS response: 401 Unauthorized)
            time=2015-09-14T18:10:44.419CEST | lvl=WARN |
            trans=1442246992-282-0000000000 | function=process | comp=Cygnus |
            msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[153] : The event
            TTL has expired, it is no more re-injected in the channel (id=1444248959,
            ttl=0)

            However, if I try to access the directory shown in the error, I'm able to
            do so

            ssh idelgado@cosmos.lab.fi-ware.org
            hadoop fs -ls /user/idelgado/org42/idelgado_tablanull
            Found 1 items
            rw-r---- 3 idelgado idelgado 1416 2015-07-01 13:11
            /user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt

            My cygnus.conf file content is:

            #
            ============================================

            1. OrionHDFSSink
              configuration
            1. channel name from where to read notification
              events
              cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
            2. sink class, must not be
              changed

            cygnusagent.sinks.hdfs-sink.type =
            es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink

            1. Comma-separated list of FQDN/IP address regarding the Cosmos Namenode
              endpoints
              cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
            2. port of the Cosmos service listening for persistence operations; 14000
              for httpfs, 50070 for webhdfs and free c\
              hoice for
              inifinty

            cygnusagent.sinks.hdfs-sink.cosmos_port = 14000

            1. default username allowed to write in
              HDFS
              cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado
            2. default password for the default
              username

            #cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX

            cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i

            1. HDFS backend type (webhdfs, httpfs or
              infinity)
              cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
            2. how the attributes are stored, either per row either per column (row,
              column)
              cygnusagent.sinks.hdfs-sink.attr_persistence = column
            3. prefix for the database and table names, empty if no prefix is
              desired
              cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla
            4. Hive FQDN/IP address of the Hive
              server

            cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46

            1. Hive port for Hive external table
              provisioning

            cygnusagent.sinks.hdfs-sink.hive_port = 10000

            Could you please help us?

            Thanks in advance!

            2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE <

            –
            Carlos Corrales
            Cofounder Secmotic Innovation S.L.
            http://www.secmotic.com

            _______________________________________________
            Fiware-finodex-coaching mailing list
            Fiware-finodex-coaching@lists.fiware.org
            https://lists.fiware.org/listinfo/fiware-finodex-coaching-new

            Show
            fw.ext.user FW External User added a comment - Hi everyone, Thanks for your responses, we have been testing the whole day and trying to find out the issue. It is not a problem with neither the password, nor the oauth2 token. We are able to access using ssh user/password and the curl command proposed by Pasquale with Oauth2 token. This is what I received using the Oauth2 token curl proposed by Pasquale curl -X GET " http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado " -H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i" {"FileStatuses":{"FileStatus":[ {"pathSuffix":".staging","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"700","accessTime":0,"modificationTime":1437059783102,"blockSize":0,"replication":0} , {"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0} , {"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0} , {"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0} , {"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0} , {"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0} , {"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0} , {"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0} ]}} Everything seems to be correct. However, using the same token and updating ORION, I receive the following error in Cygnus: time=2015-09-14T18:10:44.419CEST | lvl=ERROR | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink [140] : Persistence error (The org42/idelgado_tablanull directory could not be created in HDFS. HttpFS response: 401 Unauthorized) time=2015-09-14T18:10:44.419CEST | lvl=WARN | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink [153] : The event TTL has expired, it is no more re-injected in the channel (id=1444248959, ttl=0) However, if I try to access the directory shown in the error, I'm able to do so ssh idelgado@cosmos.lab.fi-ware.org hadoop fs -ls /user/idelgado/org42/idelgado_tablanull Found 1 items rw-r ---- 3 idelgado idelgado 1416 2015-07-01 13:11 /user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt My cygnus.conf file content is: # ============================================ OrionHDFSSink configuration channel name from where to read notification events cygnusagent.sinks.hdfs-sink.channel = hdfs-channel sink class, must not be changed cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46 port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free c\ hoice for inifinty cygnusagent.sinks.hdfs-sink.cosmos_port = 14000 default username allowed to write in HDFS cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado default password for the default username #cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i HDFS backend type (webhdfs, httpfs or infinity) cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs how the attributes are stored, either per row either per column (row, column) cygnusagent.sinks.hdfs-sink.attr_persistence = column prefix for the database and table names, empty if no prefix is desired cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla Hive FQDN/IP address of the Hive server cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46 Hive port for Hive external table provisioning cygnusagent.sinks.hdfs-sink.hive_port = 10000 Could you please help us? Thanks in advance! 2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE < – Carlos Corrales Cofounder Secmotic Innovation S.L. http://www.secmotic.com _______________________________________________ Fiware-finodex-coaching mailing list Fiware-finodex-coaching@lists.fiware.org https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
            Hide
            Permalink
            fw.ext.user FW External User added a comment -

            One more thing, Francisco, how can I find the Cygnus version?

            Best regards

            –
            Carlos Corrales
            Cofounder Secmotic Innovation S.L.
            http://www.secmotic.com

            _______________________________________________
            Fiware-finodex-coaching mailing list
            Fiware-finodex-coaching@lists.fiware.org
            https://lists.fiware.org/listinfo/fiware-finodex-coaching-new

            Show
            fw.ext.user FW External User added a comment - One more thing, Francisco, how can I find the Cygnus version? Best regards – Carlos Corrales Cofounder Secmotic Innovation S.L. http://www.secmotic.com _______________________________________________ Fiware-finodex-coaching mailing list Fiware-finodex-coaching@lists.fiware.org https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
            Hide
            Permalink
            fw.ext.user FW External User added a comment -

            Hi everyone again,

            Could it be related to the Fiware-Service and Fiware-ServicePath headers?
            We didn't use them at the beginning, then we tried to use them, and now, we
            notice that if we didn't use them the directory used by Cygnus to write in
            Cosmos is /org42/idelgado_tablanull, while if we use for instance

            --header 'Fiware-Service: unknownT' --header 'Fiware-ServicePath: /'

            the directory written is /unknownt/idelgado_tablanull.

            Any clues?

            Best regards

            –
            Carlos Corrales
            Cofounder Secmotic Innovation S.L.
            http://www.secmotic.com

            _______________________________________________
            Fiware-finodex-coaching mailing list
            Fiware-finodex-coaching@lists.fiware.org
            https://lists.fiware.org/listinfo/fiware-finodex-coaching-new

            Show
            fw.ext.user FW External User added a comment - Hi everyone again, Could it be related to the Fiware-Service and Fiware-ServicePath headers? We didn't use them at the beginning, then we tried to use them, and now, we notice that if we didn't use them the directory used by Cygnus to write in Cosmos is /org42/idelgado_tablanull, while if we use for instance --header 'Fiware-Service: unknownT' --header 'Fiware-ServicePath: /' the directory written is /unknownt/idelgado_tablanull. Any clues? Best regards – Carlos Corrales Cofounder Secmotic Innovation S.L. http://www.secmotic.com _______________________________________________ Fiware-finodex-coaching mailing list Fiware-finodex-coaching@lists.fiware.org https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
            Hide
            Permalink
            pasquale.vitale Pasquale Vitale added a comment -

            The issue has been emailed:

            • Time sent: 15/Sep/15 1:08 PM
            • To: Fiware-finodex-coaching@lists.fiware.org
            • with subject: *(HELC-1025) [Fiware-finodex-coaching] Problems with COSMOS persistance *

            Dear Carlos,
            Fiware-Service and Fiware-ServicePath headers are important when you want to use Cygnus with a Database. They are the name of table in the database.

            Best regards,
            Pasquale

            Show
            pasquale.vitale Pasquale Vitale added a comment - The issue has been emailed: Time sent: 15/Sep/15 1:08 PM To: Fiware-finodex-coaching@lists.fiware.org with subject: *( HELC-1025 ) [Fiware-finodex-coaching] Problems with COSMOS persistance * Dear Carlos, Fiware-Service and Fiware-ServicePath headers are important when you want to use Cygnus with a Database. They are the name of table in the database. Best regards, Pasquale
            Hide
            Permalink
            frb Francisco Romero added a comment -

            From: FRANCISCO ROMERO BUENO
            Sent: martes, 15 de septiembre de 2015 9:12
            To: Carlos Corrales; MANUEL ESCRICHE VICENTE
            Cc: JUAN JOSE HIERRO SUREDA; fiware-finodex-coaching@lists.fi-ware.org; jose@secmotic.com
            Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance

            Hi Carlos,

            I will aggregate through this email all the other emails you sent.

            Regarding the version, it is printed in the log at the very beginning of the execution. In addition, if you installed via RPM you can check for the installed version by typing:

            $ rpm –qa | grep cygnus

            Nevertheless, looking at your configuration, it seems you are using a very old version of Cygnus, or you are mixing parameters from different versions. I mean:
            Current version is 0.8.2
            The "cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink” part is previous to 0.8.0. From 0.8.0 it must be “com.telefonica.iot.cygnus.sinks.OrionCKANSink”
            The OAuth2 token was introduced in 0.8.2
            My recommendation is you install the latest version from RPM (https://github.com/telefonicaid/fiware-cygnus/tree/master#section4.1) and you configure Cygnus following the guidelines appearing in the master or release-0.8.2 branches (they are synchronized):
            https://github.com/telefonicaid/fiware-cygnus/blob/master/README.md
            https://github.com/telefonicaid/fiware-cygnus/blob/release/0.8.2/README.md

            Regarding the fiware-service and fiware-servicePath values, if they are not notified as http headers, then the default configured values in Cygnus’ HTTPSource are used. Again, it seems you are not using/configuring these values.

            Regards,
            Francisco

            Show
            frb Francisco Romero added a comment - From: FRANCISCO ROMERO BUENO Sent: martes, 15 de septiembre de 2015 9:12 To: Carlos Corrales; MANUEL ESCRICHE VICENTE Cc: JUAN JOSE HIERRO SUREDA; fiware-finodex-coaching@lists.fi-ware.org; jose@secmotic.com Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance Hi Carlos, I will aggregate through this email all the other emails you sent. Regarding the version, it is printed in the log at the very beginning of the execution. In addition, if you installed via RPM you can check for the installed version by typing: $ rpm –qa | grep cygnus Nevertheless, looking at your configuration, it seems you are using a very old version of Cygnus, or you are mixing parameters from different versions. I mean: Current version is 0.8.2 The "cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink” part is previous to 0.8.0. From 0.8.0 it must be “com.telefonica.iot.cygnus.sinks.OrionCKANSink” The OAuth2 token was introduced in 0.8.2 My recommendation is you install the latest version from RPM ( https://github.com/telefonicaid/fiware-cygnus/tree/master#section4.1 ) and you configure Cygnus following the guidelines appearing in the master or release-0.8.2 branches (they are synchronized): https://github.com/telefonicaid/fiware-cygnus/blob/master/README.md https://github.com/telefonicaid/fiware-cygnus/blob/release/0.8.2/README.md Regarding the fiware-service and fiware-servicePath values, if they are not notified as http headers, then the default configured values in Cygnus’ HTTPSource are used. Again, it seems you are not using/configuring these values. Regards, Francisco
            Hide
            Permalink
            fw.ext.user FW External User added a comment -

            Hi Francisco,

            Yes, we were using cygnus 0.5-91... It is the default version when using
            the image available in FIWARE lab (orion-psb-image-R4.2
            <https://cloud.lab.fiware.org/#nova/images/7dd7a0cc-3e3b-4eb3-a50c-4a2e3e498df3>
            ).

            After updating the version and configuring it properly, everything is
            working again.

            Thank you very much!

            2015-09-15 9:45 GMT+02:00 MANUEL ESCRICHE VICENTE <

            –
            Carlos Corrales
            Cofounder Secmotic Innovation S.L.
            http://www.secmotic.com

            _______________________________________________
            Fiware-finodex-coaching mailing list
            Fiware-finodex-coaching@lists.fiware.org
            https://lists.fiware.org/listinfo/fiware-finodex-coaching-new

            Show
            fw.ext.user FW External User added a comment - Hi Francisco, Yes, we were using cygnus 0.5-91... It is the default version when using the image available in FIWARE lab (orion-psb-image-R4.2 < https://cloud.lab.fiware.org/#nova/images/7dd7a0cc-3e3b-4eb3-a50c-4a2e3e498df3 > ). After updating the version and configuring it properly, everything is working again. Thank you very much! 2015-09-15 9:45 GMT+02:00 MANUEL ESCRICHE VICENTE < – Carlos Corrales Cofounder Secmotic Innovation S.L. http://www.secmotic.com _______________________________________________ Fiware-finodex-coaching mailing list Fiware-finodex-coaching@lists.fiware.org https://lists.fiware.org/listinfo/fiware-finodex-coaching-new
            Hide
            Permalink
            fw.ext.user FW External User added a comment -

            Hi Carlos,

            I will aggregate through this email all the other emails you sent.

            Regarding the version, it is printed in the log at the very beginning of the execution. In addition, if you installed via RPM you can check for the installed version by typing:

            $ rpm –qa | grep cygnus

            Nevertheless, looking at your configuration, it seems you are using a very old version of Cygnus, or you are mixing parameters from different versions. I mean:

            • Current version is 0.8.2
            • The "cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink” part is previous to 0.8.0. From 0.8.0 it must be “com.telefonica.iot.cygnus.sinks.OrionCKANSink”
            • The OAuth2 token was introduced in 0.8.2

            My recommendation is you install the latest version from RPM (https://github.com/telefonicaid/fiware-cygnus/tree/master#section4.1) and you configure Cygnus following the guidelines appearing in the master or release-0.8.2 branches (they are synchronized):
            https://github.com/telefonicaid/fiware-cygnus/blob/master/README.md
            https://github.com/telefonicaid/fiware-cygnus/blob/release/0.8.2/README.md<https://github.com/telefonicaid/fiware-cygnus/tree/release/0.8.2>

            Regarding the fiware-service and fiware-servicePath values, if they are not notified as http headers, then the default configured values in Cygnus’ HTTPSource are used. Again, it seems you are not using/configuring these values.

            Regards,
            Francisco

            De: Carlos Corrales <carlos@secmotic.com<carlos@secmotic.com>>
            Fecha: lunes, 14 de septiembre de 2015, 18:22
            Para: MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>
            CC: JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com<juanjose.hierro@telefonica.com>>, "fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>" <fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>>, "jose@secmotic.com<jose@secmotic.com>" <jose@secmotic.com<jose@secmotic.com>>, Francisco Romero Bueno <francisco.romerobueno@telefonica.com<francisco.romerobueno@telefonica.com>>
            Asunto: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance

            Hi everyone,

            Thanks for your responses, we have been testing the whole day and trying to find out the issue. It is not a problem with neither the password, nor the oauth2 token. We are able to access using ssh user/password and the curl command proposed by Pasquale with Oauth2 token. This is what I received using the Oauth2 token curl proposed by Pasquale

            curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado" -H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i"
            {"FileStatuses":{"FileStatus":[

            {"pathSuffix":".staging","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"700","accessTime":0,"modificationTime":1437059783102,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0}

            ]}}

            Everything seems to be correct. However, using the same token and updating ORION, I receive the following error in Cygnus:

            time=2015-09-14T18:10:44.419CEST | lvl=ERROR | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] : Persistence error (The org42/idelgado_tablanull directory could not be created in HDFS. HttpFS response: 401 Unauthorized)
            time=2015-09-14T18:10:44.419CEST | lvl=WARN | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[153] : The event TTL has expired, it is no more re-injected in the channel (id=1444248959, ttl=0)

            However, if I try to access the directory shown in the error, I'm able to do so

            ssh idelgado@cosmos.lab.fi-ware.org<idelgado@cosmos.lab.fi-ware.org>
            hadoop fs -ls /user/idelgado/org42/idelgado_tablanull
            Found 1 items
            rw-r---- 3 idelgado idelgado 1416 2015-07-01 13:11 /user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt

            My cygnus.conf file content is:

            1. ============================================
            2. OrionHDFSSink configuration
            3. channel name from where to read notification events
              cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
            4. sink class, must not be changed
              cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
            5. Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
              cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
            6. port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free c\
              hoice for inifinty
              cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
            7. default username allowed to write in HDFS
              cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado
            8. default password for the default username
              #cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX
              cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i
            9. HDFS backend type (webhdfs, httpfs or infinity)
              cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
            10. how the attributes are stored, either per row either per column (row, column)
              cygnusagent.sinks.hdfs-sink.attr_persistence = column
            11. prefix for the database and table names, empty if no prefix is desired
              cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla
            12. Hive FQDN/IP address of the Hive server
              cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
            13. Hive port for Hive external table provisioning
              cygnusagent.sinks.hdfs-sink.hive_port = 10000

            Could you please help us?

            Thanks in advance!

            2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>:
            In this case I see both Pasquale (Coach) and Francisco (GE Owner) have reacted to the request.
            In the normal flow GE owners enter into action upon request from Coaches.
            Kind regards,
            Manuel

            From: JUAN JOSE HIERRO SUREDA
            Sent: lunes, 14 de septiembre de 2015 0:37
            To: Carlos Corrales
            Cc: fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>; jose@secmotic.com<jose@secmotic.com>; FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA
            Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance

            I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list)

            Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket.

            You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution.

            Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is.

            Best regards,

            – Juanjo

            Show
            fw.ext.user FW External User added a comment - Hi Carlos, I will aggregate through this email all the other emails you sent. Regarding the version, it is printed in the log at the very beginning of the execution. In addition, if you installed via RPM you can check for the installed version by typing: $ rpm –qa | grep cygnus Nevertheless, looking at your configuration, it seems you are using a very old version of Cygnus, or you are mixing parameters from different versions. I mean: Current version is 0.8.2 The "cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink” part is previous to 0.8.0. From 0.8.0 it must be “com.telefonica.iot.cygnus.sinks.OrionCKANSink” The OAuth2 token was introduced in 0.8.2 My recommendation is you install the latest version from RPM ( https://github.com/telefonicaid/fiware-cygnus/tree/master#section4.1 ) and you configure Cygnus following the guidelines appearing in the master or release-0.8.2 branches (they are synchronized): https://github.com/telefonicaid/fiware-cygnus/blob/master/README.md https://github.com/telefonicaid/fiware-cygnus/blob/release/0.8.2/README.md < https://github.com/telefonicaid/fiware-cygnus/tree/release/0.8.2 > Regarding the fiware-service and fiware-servicePath values, if they are not notified as http headers, then the default configured values in Cygnus’ HTTPSource are used. Again, it seems you are not using/configuring these values. Regards, Francisco De: Carlos Corrales <carlos@secmotic.com< carlos@secmotic.com >> Fecha: lunes, 14 de septiembre de 2015, 18:22 Para: MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com< manuel.escrichevicente@telefonica.com >> CC: JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com< juanjose.hierro@telefonica.com >>, "fiware-finodex-coaching@lists.fi-ware.org< fiware-finodex-coaching@lists.fi-ware.org >" <fiware-finodex-coaching@lists.fi-ware.org< fiware-finodex-coaching@lists.fi-ware.org >>, "jose@secmotic.com< jose@secmotic.com >" <jose@secmotic.com< jose@secmotic.com >>, Francisco Romero Bueno <francisco.romerobueno@telefonica.com< francisco.romerobueno@telefonica.com >> Asunto: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance Hi everyone, Thanks for your responses, we have been testing the whole day and trying to find out the issue. It is not a problem with neither the password, nor the oauth2 token. We are able to access using ssh user/password and the curl command proposed by Pasquale with Oauth2 token. This is what I received using the Oauth2 token curl proposed by Pasquale curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado" -H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i" {"FileStatuses":{"FileStatus":[ {"pathSuffix":".staging","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"700","accessTime":0,"modificationTime":1437059783102,"blockSize":0,"replication":0} , {"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0} , {"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0} , {"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0} , {"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0} , {"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0} , {"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0} , {"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0} ]}} Everything seems to be correct. However, using the same token and updating ORION, I receive the following error in Cygnus: time=2015-09-14T18:10:44.419CEST | lvl=ERROR | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink [140] : Persistence error (The org42/idelgado_tablanull directory could not be created in HDFS. HttpFS response: 401 Unauthorized) time=2015-09-14T18:10:44.419CEST | lvl=WARN | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink [153] : The event TTL has expired, it is no more re-injected in the channel (id=1444248959, ttl=0) However, if I try to access the directory shown in the error, I'm able to do so ssh idelgado@cosmos.lab.fi-ware.org< idelgado@cosmos.lab.fi-ware.org > hadoop fs -ls /user/idelgado/org42/idelgado_tablanull Found 1 items rw-r ---- 3 idelgado idelgado 1416 2015-07-01 13:11 /user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt My cygnus.conf file content is: ============================================ OrionHDFSSink configuration channel name from where to read notification events cygnusagent.sinks.hdfs-sink.channel = hdfs-channel sink class, must not be changed cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46 port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free c\ hoice for inifinty cygnusagent.sinks.hdfs-sink.cosmos_port = 14000 default username allowed to write in HDFS cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado default password for the default username #cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i HDFS backend type (webhdfs, httpfs or infinity) cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs how the attributes are stored, either per row either per column (row, column) cygnusagent.sinks.hdfs-sink.attr_persistence = column prefix for the database and table names, empty if no prefix is desired cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla Hive FQDN/IP address of the Hive server cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46 Hive port for Hive external table provisioning cygnusagent.sinks.hdfs-sink.hive_port = 10000 Could you please help us? Thanks in advance! 2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com< manuel.escrichevicente@telefonica.com >>: In this case I see both Pasquale (Coach) and Francisco (GE Owner) have reacted to the request. In the normal flow GE owners enter into action upon request from Coaches. Kind regards, Manuel From: JUAN JOSE HIERRO SUREDA Sent: lunes, 14 de septiembre de 2015 0:37 To: Carlos Corrales Cc: fiware-finodex-coaching@lists.fi-ware.org< fiware-finodex-coaching@lists.fi-ware.org >; jose@secmotic.com< jose@secmotic.com >; FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list) Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket. You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution. Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is. Best regards, – Juanjo
            Hide
            Permalink
            fw.ext.user FW External User added a comment -

            Great to hear that!

            Regarding the image containing the bundle Orion+Cygnus, it must be updated for sure. Adding it to our backlog.

            Regards,
            Francisco

            De: Carlos Corrales <carlos@secmotic.com<carlos@secmotic.com>>
            Fecha: miércoles, 16 de septiembre de 2015, 11:11
            Para: MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>
            CC: Francisco Romero Bueno <francisco.romerobueno@telefonica.com<francisco.romerobueno@telefonica.com>>, JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com<juanjose.hierro@telefonica.com>>, "fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>" <fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>>, "jose@secmotic.com<jose@secmotic.com>" <jose@secmotic.com<jose@secmotic.com>>
            Asunto: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance

            Hi Francisco,

            Yes, we were using cygnus 0.5-91... It is the default version when using the image available in FIWARE lab (orion-psb-image-R4.2<https://cloud.lab.fiware.org/#nova/images/7dd7a0cc-3e3b-4eb3-a50c-4a2e3e498df3>).

            After updating the version and configuring it properly, everything is working again.

            Thank you very much!

            2015-09-15 9:45 GMT+02:00 MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>:
            Be aware there exist this issue in JIRA for this exchange: https://jira.fiware.org/browse/HELC-1025

            From: FRANCISCO ROMERO BUENO
            Sent: martes, 15 de septiembre de 2015 9:12
            To: Carlos Corrales; MANUEL ESCRICHE VICENTE
            Cc: JUAN JOSE HIERRO SUREDA; fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>; jose@secmotic.com<jose@secmotic.com>

            Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance

            Hi Carlos,

            I will aggregate through this email all the other emails you sent.

            Regarding the version, it is printed in the log at the very beginning of the execution. In addition, if you installed via RPM you can check for the installed version by typing:

            $ rpm –qa | grep cygnus

            Nevertheless, looking at your configuration, it seems you are using a very old version of Cygnus, or you are mixing parameters from different versions. I mean:

            Regarding the fiware-service and fiware-servicePath values, if they are not notified as http headers, then the default configured values in Cygnus’ HTTPSource are used. Again, it seems you are not using/configuring these values.

            Regards,
            Francisco

            De: Carlos Corrales <carlos@secmotic.com<carlos@secmotic.com>>
            Fecha: lunes, 14 de septiembre de 2015, 18:22
            Para: MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>
            CC: JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com<juanjose.hierro@telefonica.com>>, "fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>" <fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>>, "jose@secmotic.com<jose@secmotic.com>" <jose@secmotic.com<jose@secmotic.com>>, Francisco Romero Bueno <francisco.romerobueno@telefonica.com<francisco.romerobueno@telefonica.com>>
            Asunto: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance

            Hi everyone,
            Thanks for your responses, we have been testing the whole day and trying to find out the issue. It is not a problem with neither the password, nor the oauth2 token. We are able to access using ssh user/password and the curl command proposed by Pasquale with Oauth2 token. This is what I received using the Oauth2 token curl proposed by Pasquale

            curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado" -H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i"
            {"FileStatuses":{"FileStatus":[

            {"pathSuffix":".staging","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"700","accessTime":0,"modificationTime":1437059783102,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0}

            ]}}

            Everything seems to be correct. However, using the same token and updating ORION, I receive the following error in Cygnus:

            time=2015-09-14T18:10:44.419CEST | lvl=ERROR | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] : Persistence error (The org42/idelgado_tablanull directory could not be created in HDFS. HttpFS response: 401 Unauthorized)
            time=2015-09-14T18:10:44.419CEST | lvl=WARN | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[153] : The event TTL has expired, it is no more re-injected in the channel (id=1444248959, ttl=0)
            However, if I try to access the directory shown in the error, I'm able to do so

            ssh idelgado@cosmos.lab.fi-ware.org<idelgado@cosmos.lab.fi-ware.org>
            hadoop fs -ls /user/idelgado/org42/idelgado_tablanull
            Found 1 items
            rw-r---- 3 idelgado idelgado 1416 2015-07-01 13:11 /user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt
            My cygnus.conf file content is:

            1. ============================================
            2. OrionHDFSSink configuration
            3. channel name from where to read notification events
              cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
            4. sink class, must not be changed
              cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
            5. Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
              cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
            6. port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free c\
              hoice for inifinty
              cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
            7. default username allowed to write in HDFS
              cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado
            8. default password for the default username
              #cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX
              cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i
            9. HDFS backend type (webhdfs, httpfs or infinity)
              cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
            10. how the attributes are stored, either per row either per column (row, column)
              cygnusagent.sinks.hdfs-sink.attr_persistence = column
            11. prefix for the database and table names, empty if no prefix is desired
              cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla
            12. Hive FQDN/IP address of the Hive server
              cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
            13. Hive port for Hive external table provisioning
              cygnusagent.sinks.hdfs-sink.hive_port = 10000
              Could you please help us?
              Thanks in advance!

            2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>:
            In this case I see both Pasquale (Coach) and Francisco (GE Owner) have reacted to the request.
            In the normal flow GE owners enter into action upon request from Coaches.
            Kind regards,
            Manuel

            From: JUAN JOSE HIERRO SUREDA
            Sent: lunes, 14 de septiembre de 2015 0:37
            To: Carlos Corrales
            Cc: fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>; jose@secmotic.com<jose@secmotic.com>; FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA
            Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance

            I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list)

            Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket.

            You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution.

            Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is.

            Best regards,

            – Juanjo

            Show
            fw.ext.user FW External User added a comment - Great to hear that! Regarding the image containing the bundle Orion+Cygnus, it must be updated for sure. Adding it to our backlog. Regards, Francisco De: Carlos Corrales <carlos@secmotic.com< carlos@secmotic.com >> Fecha: miércoles, 16 de septiembre de 2015, 11:11 Para: MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com< manuel.escrichevicente@telefonica.com >> CC: Francisco Romero Bueno <francisco.romerobueno@telefonica.com< francisco.romerobueno@telefonica.com >>, JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com< juanjose.hierro@telefonica.com >>, "fiware-finodex-coaching@lists.fi-ware.org< fiware-finodex-coaching@lists.fi-ware.org >" <fiware-finodex-coaching@lists.fi-ware.org< fiware-finodex-coaching@lists.fi-ware.org >>, "jose@secmotic.com< jose@secmotic.com >" <jose@secmotic.com< jose@secmotic.com >> Asunto: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance Hi Francisco, Yes, we were using cygnus 0.5-91... It is the default version when using the image available in FIWARE lab (orion-psb-image-R4.2< https://cloud.lab.fiware.org/#nova/images/7dd7a0cc-3e3b-4eb3-a50c-4a2e3e498df3 >). After updating the version and configuring it properly, everything is working again. Thank you very much! 2015-09-15 9:45 GMT+02:00 MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com< manuel.escrichevicente@telefonica.com >>: Be aware there exist this issue in JIRA for this exchange: https://jira.fiware.org/browse/HELC-1025 From: FRANCISCO ROMERO BUENO Sent: martes, 15 de septiembre de 2015 9:12 To: Carlos Corrales; MANUEL ESCRICHE VICENTE Cc: JUAN JOSE HIERRO SUREDA; fiware-finodex-coaching@lists.fi-ware.org< fiware-finodex-coaching@lists.fi-ware.org >; jose@secmotic.com< jose@secmotic.com > Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance Hi Carlos, I will aggregate through this email all the other emails you sent. Regarding the version, it is printed in the log at the very beginning of the execution. In addition, if you installed via RPM you can check for the installed version by typing: $ rpm –qa | grep cygnus Nevertheless, looking at your configuration, it seems you are using a very old version of Cygnus, or you are mixing parameters from different versions. I mean: Current version is 0.8.2 The "cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink” part is previous to 0.8.0. From 0.8.0 it must be “com.telefonica.iot.cygnus.sinks.OrionCKANSink” The OAuth2 token was introduced in 0.8.2 My recommendation is you install the latest version from RPM ( https://github.com/telefonicaid/fiware-cygnus/tree/master#section4.1 ) and you configure Cygnus following the guidelines appearing in the master or release-0.8.2 branches (they are synchronized): https://github.com/telefonicaid/fiware-cygnus/blob/master/README.md https://github.com/telefonicaid/fiware-cygnus/blob/release/0.8.2/README.md < https://github.com/telefonicaid/fiware-cygnus/tree/release/0.8.2 > Regarding the fiware-service and fiware-servicePath values, if they are not notified as http headers, then the default configured values in Cygnus’ HTTPSource are used. Again, it seems you are not using/configuring these values. Regards, Francisco De: Carlos Corrales <carlos@secmotic.com< carlos@secmotic.com >> Fecha: lunes, 14 de septiembre de 2015, 18:22 Para: MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com< manuel.escrichevicente@telefonica.com >> CC: JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com< juanjose.hierro@telefonica.com >>, "fiware-finodex-coaching@lists.fi-ware.org< fiware-finodex-coaching@lists.fi-ware.org >" <fiware-finodex-coaching@lists.fi-ware.org< fiware-finodex-coaching@lists.fi-ware.org >>, "jose@secmotic.com< jose@secmotic.com >" <jose@secmotic.com< jose@secmotic.com >>, Francisco Romero Bueno <francisco.romerobueno@telefonica.com< francisco.romerobueno@telefonica.com >> Asunto: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance Hi everyone, Thanks for your responses, we have been testing the whole day and trying to find out the issue. It is not a problem with neither the password, nor the oauth2 token. We are able to access using ssh user/password and the curl command proposed by Pasquale with Oauth2 token. This is what I received using the Oauth2 token curl proposed by Pasquale curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado" -H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i" {"FileStatuses":{"FileStatus":[ {"pathSuffix":".staging","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"700","accessTime":0,"modificationTime":1437059783102,"blockSize":0,"replication":0} , {"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0} , {"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0} , {"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0} , {"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0} , {"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0} , {"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0} , {"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0} ]}} Everything seems to be correct. However, using the same token and updating ORION, I receive the following error in Cygnus: time=2015-09-14T18:10:44.419CEST | lvl=ERROR | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink [140] : Persistence error (The org42/idelgado_tablanull directory could not be created in HDFS. HttpFS response: 401 Unauthorized) time=2015-09-14T18:10:44.419CEST | lvl=WARN | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink [153] : The event TTL has expired, it is no more re-injected in the channel (id=1444248959, ttl=0) However, if I try to access the directory shown in the error, I'm able to do so ssh idelgado@cosmos.lab.fi-ware.org< idelgado@cosmos.lab.fi-ware.org > hadoop fs -ls /user/idelgado/org42/idelgado_tablanull Found 1 items rw-r ---- 3 idelgado idelgado 1416 2015-07-01 13:11 /user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt My cygnus.conf file content is: ============================================ OrionHDFSSink configuration channel name from where to read notification events cygnusagent.sinks.hdfs-sink.channel = hdfs-channel sink class, must not be changed cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46 port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free c\ hoice for inifinty cygnusagent.sinks.hdfs-sink.cosmos_port = 14000 default username allowed to write in HDFS cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado default password for the default username #cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i HDFS backend type (webhdfs, httpfs or infinity) cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs how the attributes are stored, either per row either per column (row, column) cygnusagent.sinks.hdfs-sink.attr_persistence = column prefix for the database and table names, empty if no prefix is desired cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla Hive FQDN/IP address of the Hive server cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46 Hive port for Hive external table provisioning cygnusagent.sinks.hdfs-sink.hive_port = 10000 Could you please help us? Thanks in advance! 2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com< manuel.escrichevicente@telefonica.com >>: In this case I see both Pasquale (Coach) and Francisco (GE Owner) have reacted to the request. In the normal flow GE owners enter into action upon request from Coaches. Kind regards, Manuel From: JUAN JOSE HIERRO SUREDA Sent: lunes, 14 de septiembre de 2015 0:37 To: Carlos Corrales Cc: fiware-finodex-coaching@lists.fi-ware.org< fiware-finodex-coaching@lists.fi-ware.org >; jose@secmotic.com< jose@secmotic.com >; FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list) Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket. You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution. Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is. Best regards, – Juanjo
            Hide
            Permalink
            sysadmin System Administrator added a comment -

            In this case I see both Pasquale (Coach) and Francisco (GE Owner) have reacted to the request.
            In the normal flow GE owners enter into action upon request from Coaches.
            Kind regards,
            Manuel

            From: JUAN JOSE HIERRO SUREDA
            Sent: lunes, 14 de septiembre de 2015 0:37
            To: Carlos Corrales
            Cc: fiware-finodex-coaching@lists.fi-ware.org; jose@secmotic.com; FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA
            Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance

            I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list)

            Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket.

            You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution.

            Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is.

            Best regards,

            – Juanjo

            Show
            sysadmin System Administrator added a comment - In this case I see both Pasquale (Coach) and Francisco (GE Owner) have reacted to the request. In the normal flow GE owners enter into action upon request from Coaches. Kind regards, Manuel From: JUAN JOSE HIERRO SUREDA Sent: lunes, 14 de septiembre de 2015 0:37 To: Carlos Corrales Cc: fiware-finodex-coaching@lists.fi-ware.org; jose@secmotic.com; FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list) Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket. You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution. Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is. Best regards, – Juanjo
            Hide
            Permalink
            sysadmin System Administrator added a comment -

            Be aware there exist this issue in JIRA for this exchange: https://jira.fiware.org/browse/HELC-1025

            From: FRANCISCO ROMERO BUENO
            Sent: martes, 15 de septiembre de 2015 9:12
            To: Carlos Corrales; MANUEL ESCRICHE VICENTE
            Cc: JUAN JOSE HIERRO SUREDA; fiware-finodex-coaching@lists.fi-ware.org; jose@secmotic.com
            Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance

            Hi Carlos,

            I will aggregate through this email all the other emails you sent.

            Regarding the version, it is printed in the log at the very beginning of the execution. In addition, if you installed via RPM you can check for the installed version by typing:

            $ rpm -qa | grep cygnus

            Nevertheless, looking at your configuration, it seems you are using a very old version of Cygnus, or you are mixing parameters from different versions. I mean:

            Regarding the fiware-service and fiware-servicePath values, if they are not notified as http headers, then the default configured values in Cygnus' HTTPSource are used. Again, it seems you are not using/configuring these values.

            Regards,
            Francisco

            De: Carlos Corrales <carlos@secmotic.com<carlos@secmotic.com>>
            Fecha: lunes, 14 de septiembre de 2015, 18:22
            Para: MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>
            CC: JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com<juanjose.hierro@telefonica.com>>, "fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>" <fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>>, "jose@secmotic.com<jose@secmotic.com>" <jose@secmotic.com<jose@secmotic.com>>, Francisco Romero Bueno <francisco.romerobueno@telefonica.com<francisco.romerobueno@telefonica.com>>
            Asunto: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance

            Hi everyone,
            Thanks for your responses, we have been testing the whole day and trying to find out the issue. It is not a problem with neither the password, nor the oauth2 token. We are able to access using ssh user/password and the curl command proposed by Pasquale with Oauth2 token. This is what I received using the Oauth2 token curl proposed by Pasquale

            curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado" -H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i"
            {"FileStatuses":{"FileStatus":[

            {"pathSuffix":".staging","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"700","accessTime":0,"modificationTime":1437059783102,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0}

            ,

            {"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0}

            ]}}

            Everything seems to be correct. However, using the same token and updating ORION, I receive the following error in Cygnus:

            time=2015-09-14T18:10:44.419CEST | lvl=ERROR | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[140] : Persistence error (The org42/idelgado_tablanull directory could not be created in HDFS. HttpFS response: 401 Unauthorized)
            time=2015-09-14T18:10:44.419CEST | lvl=WARN | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[153] : The event TTL has expired, it is no more re-injected in the channel (id=1444248959, ttl=0)
            However, if I try to access the directory shown in the error, I'm able to do so

            ssh idelgado@cosmos.lab.fi-ware.org<idelgado@cosmos.lab.fi-ware.org>
            hadoop fs -ls /user/idelgado/org42/idelgado_tablanull
            Found 1 items
            rw-r---- 3 idelgado idelgado 1416 2015-07-01 13:11 /user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt
            My cygnus.conf file content is:

            1. ============================================
            2. OrionHDFSSink configuration
            3. channel name from where to read notification events
              cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
            4. sink class, must not be changed
              cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
            5. Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
              cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
            6. port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free c\
              hoice for inifinty
              cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
            7. default username allowed to write in HDFS
              cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado
            8. default password for the default username
              #cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX
              cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i
            9. HDFS backend type (webhdfs, httpfs or infinity)
              cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
            10. how the attributes are stored, either per row either per column (row, column)
              cygnusagent.sinks.hdfs-sink.attr_persistence = column
            11. prefix for the database and table names, empty if no prefix is desired
              cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla
            12. Hive FQDN/IP address of the Hive server
              cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
            13. Hive port for Hive external table provisioning
              cygnusagent.sinks.hdfs-sink.hive_port = 10000
              Could you please help us?
              Thanks in advance!

            2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com<manuel.escrichevicente@telefonica.com>>:
            In this case I see both Pasquale (Coach) and Francisco (GE Owner) have reacted to the request.
            In the normal flow GE owners enter into action upon request from Coaches.
            Kind regards,
            Manuel

            From: JUAN JOSE HIERRO SUREDA
            Sent: lunes, 14 de septiembre de 2015 0:37
            To: Carlos Corrales
            Cc: fiware-finodex-coaching@lists.fi-ware.org<fiware-finodex-coaching@lists.fi-ware.org>; jose@secmotic.com<jose@secmotic.com>; FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA
            Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance

            I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list)

            Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket.

            You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution.

            Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is.

            Best regards,

            – Juanjo

            Show
            sysadmin System Administrator added a comment - Be aware there exist this issue in JIRA for this exchange: https://jira.fiware.org/browse/HELC-1025 From: FRANCISCO ROMERO BUENO Sent: martes, 15 de septiembre de 2015 9:12 To: Carlos Corrales; MANUEL ESCRICHE VICENTE Cc: JUAN JOSE HIERRO SUREDA; fiware-finodex-coaching@lists.fi-ware.org; jose@secmotic.com Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance Hi Carlos, I will aggregate through this email all the other emails you sent. Regarding the version, it is printed in the log at the very beginning of the execution. In addition, if you installed via RPM you can check for the installed version by typing: $ rpm -qa | grep cygnus Nevertheless, looking at your configuration, it seems you are using a very old version of Cygnus, or you are mixing parameters from different versions. I mean: Current version is 0.8.2 The "cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink" part is previous to 0.8.0. From 0.8.0 it must be "com.telefonica.iot.cygnus.sinks.OrionCKANSink" The OAuth2 token was introduced in 0.8.2 My recommendation is you install the latest version from RPM ( https://github.com/telefonicaid/fiware-cygnus/tree/master#section4.1 ) and you configure Cygnus following the guidelines appearing in the master or release-0.8.2 branches (they are synchronized): https://github.com/telefonicaid/fiware-cygnus/blob/master/README.md https://github.com/telefonicaid/fiware-cygnus/blob/release/0.8.2/README.md < https://github.com/telefonicaid/fiware-cygnus/tree/release/0.8.2 > Regarding the fiware-service and fiware-servicePath values, if they are not notified as http headers, then the default configured values in Cygnus' HTTPSource are used. Again, it seems you are not using/configuring these values. Regards, Francisco De: Carlos Corrales <carlos@secmotic.com< carlos@secmotic.com >> Fecha: lunes, 14 de septiembre de 2015, 18:22 Para: MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com< manuel.escrichevicente@telefonica.com >> CC: JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com< juanjose.hierro@telefonica.com >>, "fiware-finodex-coaching@lists.fi-ware.org< fiware-finodex-coaching@lists.fi-ware.org >" <fiware-finodex-coaching@lists.fi-ware.org< fiware-finodex-coaching@lists.fi-ware.org >>, "jose@secmotic.com< jose@secmotic.com >" <jose@secmotic.com< jose@secmotic.com >>, Francisco Romero Bueno <francisco.romerobueno@telefonica.com< francisco.romerobueno@telefonica.com >> Asunto: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance Hi everyone, Thanks for your responses, we have been testing the whole day and trying to find out the issue. It is not a problem with neither the password, nor the oauth2 token. We are able to access using ssh user/password and the curl command proposed by Pasquale with Oauth2 token. This is what I received using the Oauth2 token curl proposed by Pasquale curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/idelgado?op=liststatus&user.name=idelgado" -H "X-Auth-Token: rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i" {"FileStatuses":{"FileStatus":[ {"pathSuffix":".staging","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"700","accessTime":0,"modificationTime":1437059783102,"blockSize":0,"replication":0} , {"pathSuffix":"cuentapalabras","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433847949370,"blockSize":0,"replication":0} , {"pathSuffix":"cuentapalabras2","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434651036673,"blockSize":0,"replication":0} , {"pathSuffix":"cuentapalabras_fiware","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1434710282953,"blockSize":0,"replication":0} , {"pathSuffix":"input","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433840978953,"blockSize":0,"replication":0} , {"pathSuffix":"input_data","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433841088243,"blockSize":0,"replication":0} , {"pathSuffix":"org42","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1435749061128,"blockSize":0,"replication":0} , {"pathSuffix":"prueba","type":"DIRECTORY","length":0,"owner":"idelgado","group":"idelgado","permission":"740","accessTime":0,"modificationTime":1433842740482,"blockSize":0,"replication":0} ]}} Everything seems to be correct. However, using the same token and updating ORION, I receive the following error in Cygnus: time=2015-09-14T18:10:44.419CEST | lvl=ERROR | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink [140] : Persistence error (The org42/idelgado_tablanull directory could not be created in HDFS. HttpFS response: 401 Unauthorized) time=2015-09-14T18:10:44.419CEST | lvl=WARN | trans=1442246992-282-0000000000 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink [153] : The event TTL has expired, it is no more re-injected in the channel (id=1444248959, ttl=0) However, if I try to access the directory shown in the error, I'm able to do so ssh idelgado@cosmos.lab.fi-ware.org< idelgado@cosmos.lab.fi-ware.org > hadoop fs -ls /user/idelgado/org42/idelgado_tablanull Found 1 items rw-r ---- 3 idelgado idelgado 1416 2015-07-01 13:11 /user/idelgado/org42/idelgado_tablanull/idelgado_tablanull.txt My cygnus.conf file content is: ============================================ OrionHDFSSink configuration channel name from where to read notification events cygnusagent.sinks.hdfs-sink.channel = hdfs-channel sink class, must not be changed cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46 port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free c\ hoice for inifinty cygnusagent.sinks.hdfs-sink.cosmos_port = 14000 default username allowed to write in HDFS cygnusagent.sinks.hdfs-sink.cosmos_default_username = idelgado default password for the default username #cygnusagent.sinks.hdfs-sink.cosmos_default_password = XXXXXXX cygnusagent.sinks.hdfs-sink.oauth2_token = rxcEgpDHqHAqhWnwUB93JlAVJ7uz9i HDFS backend type (webhdfs, httpfs or infinity) cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs how the attributes are stored, either per row either per column (row, column) cygnusagent.sinks.hdfs-sink.attr_persistence = column prefix for the database and table names, empty if no prefix is desired cygnusagent.sinks.hdfs-sink.naming_prefix = idelgado_tabla Hive FQDN/IP address of the Hive server cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46 Hive port for Hive external table provisioning cygnusagent.sinks.hdfs-sink.hive_port = 10000 Could you please help us? Thanks in advance! 2015-09-14 10:26 GMT+02:00 MANUEL ESCRICHE VICENTE <manuel.escrichevicente@telefonica.com< manuel.escrichevicente@telefonica.com >>: In this case I see both Pasquale (Coach) and Francisco (GE Owner) have reacted to the request. In the normal flow GE owners enter into action upon request from Coaches. Kind regards, Manuel From: JUAN JOSE HIERRO SUREDA Sent: lunes, 14 de septiembre de 2015 0:37 To: Carlos Corrales Cc: fiware-finodex-coaching@lists.fi-ware.org< fiware-finodex-coaching@lists.fi-ware.org >; jose@secmotic.com< jose@secmotic.com >; FRANCISCO ROMERO BUENO; MANUEL ESCRICHE VICENTE; JUAN JOSE HIERRO SUREDA Subject: Re: [Fiware-finodex-coaching] Problems with COSMOS persistance I guess that the FINODEX coach will forward the ticket to the FIWARE GE owner (or eventually, if he hadn't know where to redirect it, he would redirect the original question to the fiware-tech-help mailing list) Point is that one way or the other, it would arrive at the inbox of the FIWARE GE owner who has to take care of the ticket. You may have sent the ticket directly to the fiware-tech-help mailing list ... however, I guess that the FINODEX coach team wanted to be informed to keep track of the evolution. Manuel, in copy, is the master of all the helpdesk processes and he can clarify what the actual workflow is. Best regards, – Juanjo

              People

              • Assignee:
                pasquale.vitale Pasquale Vitale
                Reporter:
                fw.ext.user FW External User
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: