Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-3762

FIWARE.Request.Tech.Data.BigData-Analysis.UnsuccessfullDataSaving

    Details

    • Type: extRequest
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Fix Version/s: 2021
    • Component/s: FIWARE-TECH-HELP
    • Labels:
      None

      Description

      Hello,

      in order to use the Cosmos's shared instance (username
      marcosalvatore.zappatore), we tried to save data into Hadoop File System,
      but unsuccessfully.

      Maybe a firewall is blocking us?

      Thank you in advance.

      Sergio De Lorenzis

      Alba Project s.r.l.

      Via Don Luigi Sturzo, 36 - 73100 - Lecce (LE)

      Tel.: +39 349 15 48 139

      Email: <sergio.delorenzis@albaproject.it>
      sergio.delorenzis@albaproject.it

      Skype: sergio.delorenzis

      _______________________________________________
      Fiware-tech-help mailing list
      Fiware-tech-help@lists.fi-ware.org
      https://lists.fi-ware.org/listinfo/fiware-tech-help

      [Created via e-mail received from: Sergio De Lorenzis <sergio.delorenzis@albaproject.it>]

      1. image.jpeg
        2 kB
      2. image002.jpg
        2 kB

        Activity

        Hide
        fw.ext.user FW External User added a comment -

        Dear Sergio,

        I am passing the case to the GE owner who will look into it directly.

        Best regards,

        Miguel

        El 06/08/2015 a las 12:40, Sergio De Lorenzis escribió:
        Hello,
        in order to use the Cosmos's shared instance (username marcosalvatore.zappatore), we tried to save data into Hadoop File System, but unsuccessfully.
        Maybe a firewall is blocking us?

        Thank you in advance.

        Sergio De Lorenzis

        Alba Project s.r.l.
        Via Don Luigi Sturzo, 36 – 73100 – Lecce (LE)
        Tel.: +39 349 15 48 139
        Email: <sergio.delorenzis@albaproject.it> sergio.delorenzis@albaproject.it<sergio.delorenzis@albaproject.it>
        Skype: sergio.delorenzis
        [LOGO Alba Project]

        Show
        fw.ext.user FW External User added a comment - Dear Sergio, I am passing the case to the GE owner who will look into it directly. Best regards, Miguel El 06/08/2015 a las 12:40, Sergio De Lorenzis escribió: Hello, in order to use the Cosmos's shared instance (username marcosalvatore.zappatore), we tried to save data into Hadoop File System, but unsuccessfully. Maybe a firewall is blocking us? Thank you in advance. Sergio De Lorenzis Alba Project s.r.l. Via Don Luigi Sturzo, 36 – 73100 – Lecce (LE) Tel.: +39 349 15 48 139 Email: < sergio.delorenzis@albaproject.it > sergio.delorenzis@albaproject.it< sergio.delorenzis@albaproject.it > Skype: sergio.delorenzis [LOGO Alba Project]
        Hide
        fw.ext.user FW External User added a comment -

        Hi Sergio,

        Can you give us more details regarding the operation you are trying to
        perform? I mean, is it a problem with WebHDFS REST API? Is it a problem
        with Cygnus?

        Best regards,
        Francisco

        El 7/8/15 9:50, "Miguel Carrillo (JIRA)" <jira-help-desk@fi-ware.org>

        ________________________________

        Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

        The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

        Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
        _______________________________________________
        Fiware-tech-help mailing list
        Fiware-tech-help@lists.fi-ware.org
        https://lists.fi-ware.org/listinfo/fiware-tech-help

        Show
        fw.ext.user FW External User added a comment - Hi Sergio, Can you give us more details regarding the operation you are trying to perform? I mean, is it a problem with WebHDFS REST API? Is it a problem with Cygnus? Best regards, Francisco El 7/8/15 9:50, "Miguel Carrillo (JIRA)" <jira-help-desk@fi-ware.org> ________________________________ Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción. The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it. Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição _______________________________________________ Fiware-tech-help mailing list Fiware-tech-help@lists.fi-ware.org https://lists.fi-ware.org/listinfo/fiware-tech-help
        Hide
        fw.ext.user FW External User added a comment -

        Yes Francisco, I'm talking about WebHDFS REST API.

        I tried to execute the API call:

        curl -i
        "http://cosmos.lab.fi-ware.org:50070/webhdfs/v1/tmp?user.name=marcosalvatore
        .zappatore
        <http://cosmos.lab.fi-ware.org:50070/webhdfs/v1/tmp?user.name=marcosalvatore
        .zappatore&op=GETFILESTATUS> &op=GETFILESTATUS"

        from the following hosts:

        1. cosmos.lab.fi-ware.org (the same host, public IP: 130.206.80.46).
        It works.

        2. a FI-LAB machine (associated IP: 193.205.211.99) . It doesn’t work.

        3. An non FI-LAB machine (my PC). It doesn’t work.

        According to the documentation, host 2 should works, while type 3 host
        should not. So the only undesiderable case is the second one and require to
        be fixed.

        The test results don’t change if I replace the hostname
        cosmos.lab.fi-ware.org with the resolved IP, 130.206.80.46; so this is not
        an issue caused by a non resolved hostname by DNS.

        Conversely, port 50070 seems to be closed at the hosts 2 and 3 (test
        executed via telnet).

        Thank you in advance.

        Best regards.

        Sergio.

        ----Messaggio originale----
        Da: FRANCISCO ROMERO BUENO francisco.romerobueno@telefonica.com
        Inviato: venerdì 7 agosto 2015 09:52
        A: fiware-tech-help@lists.fi-ware.org
        Cc: sergio.delorenzis@albaproject.it
        Oggetto: Re: [FI-WARE-JIRA] (HELP-3762) [Fiware-tech-help] FIWARE Lab
        Assistance

        Hi Sergio,

        Can you give us more details regarding the operation you are trying to
        perform? I mean, is it a problem with WebHDFS REST API? Is it a problem with
        Cygnus?

        Best regards,

        Francisco

        El 7/8/15 9:50, "Miguel Carrillo (JIRA)" <
        <jira-help-desk@fi-ware.org> jira-help-desk@fi-ware.org>

        escribió:

        https://jira.fiware.org/browse/HELP-3762?page=com.atlassian.jira.plugin

        https://jira.fiware.org/browse/HELP-3762

        sergio.delorenzis@albaproject.it>

        sergio.delorenzis@albaproject.it

        Fiware-tech-help@lists.fi-ware.org

        https://lists.fi-ware.org/listinfo/fiware-tech-help

        sergio.delorenzis@albaproject.it>]

        ________________________________

        Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario,
        puede contener información privilegiada o confidencial y es para uso
        exclusivo de la persona o entidad de destino. Si no es usted. el
        destinatario indicado, queda notificado de que la lectura, utilización,
        divulgación y/o copia sin autorización puede estar prohibida en virtud de la
        legislación vigente. Si ha recibido este mensaje por error, le rogamos que
        nos lo comunique inmediatamente por esta misma vía y proceda a su
        destrucción.

        The information contained in this transmission is privileged and
        confidential information intended only for the use of the individual or
        entity named above. If the reader of this message is not the intended
        recipient, you are hereby notified that any dissemination, distribution or
        copying of this communication is strictly prohibited. If you have received
        this transmission in error, do not read it. Please immediately reply to the
        sender that you have received this communication in error and then delete
        it.

        Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário,
        pode conter informação privilegiada ou confidencial e é para uso exclusivo
        da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário
        indicado, fica notificado de que a leitura, utilização, divulgação e/ou
        cópia sem autorização pode estar proibida em virtude da legislação vigente.
        Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique
        imediatamente por esta mesma via e proceda a sua destruição

        _______________________________________________
        Fiware-tech-help mailing list
        Fiware-tech-help@lists.fi-ware.org
        https://lists.fi-ware.org/listinfo/fiware-tech-help

        Show
        fw.ext.user FW External User added a comment - Yes Francisco, I'm talking about WebHDFS REST API. I tried to execute the API call: curl -i "http://cosmos.lab.fi-ware.org:50070/webhdfs/v1/tmp?user.name=marcosalvatore .zappatore < http://cosmos.lab.fi-ware.org:50070/webhdfs/v1/tmp?user.name=marcosalvatore .zappatore&op=GETFILESTATUS> &op=GETFILESTATUS" from the following hosts: 1. cosmos.lab.fi-ware.org (the same host, public IP: 130.206.80.46). It works. 2. a FI-LAB machine (associated IP: 193.205.211.99) . It doesn’t work. 3. An non FI-LAB machine (my PC). It doesn’t work. According to the documentation, host 2 should works, while type 3 host should not. So the only undesiderable case is the second one and require to be fixed. The test results don’t change if I replace the hostname cosmos.lab.fi-ware.org with the resolved IP, 130.206.80.46; so this is not an issue caused by a non resolved hostname by DNS. Conversely, port 50070 seems to be closed at the hosts 2 and 3 (test executed via telnet). Thank you in advance. Best regards. Sergio. ---- Messaggio originale ---- Da: FRANCISCO ROMERO BUENO francisco.romerobueno@telefonica.com Inviato: venerdì 7 agosto 2015 09:52 A: fiware-tech-help@lists.fi-ware.org Cc: sergio.delorenzis@albaproject.it Oggetto: Re: [FI-WARE-JIRA] ( HELP-3762 ) [Fiware-tech-help] FIWARE Lab Assistance Hi Sergio, Can you give us more details regarding the operation you are trying to perform? I mean, is it a problem with WebHDFS REST API? Is it a problem with Cygnus? Best regards, Francisco El 7/8/15 9:50, "Miguel Carrillo (JIRA)" < < jira-help-desk@fi-ware.org > jira-help-desk@fi-ware.org> escribió: https://jira.fiware.org/browse/HELP-3762?page=com.atlassian.jira.plugin https://jira.fiware.org/browse/HELP-3762 sergio.delorenzis@albaproject.it > sergio.delorenzis@albaproject.it Fiware-tech-help@lists.fi-ware.org https://lists.fi-ware.org/listinfo/fiware-tech-help sergio.delorenzis@albaproject.it>] ________________________________ Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción. The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it. Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição _______________________________________________ Fiware-tech-help mailing list Fiware-tech-help@lists.fi-ware.org https://lists.fi-ware.org/listinfo/fiware-tech-help
        Hide
        fw.ext.user FW External User added a comment -

        Hi Sergio,

        You are not able to perform such an operation from outside because you
        have to append to the request an OAuth2 token (and use HttpFS port
        instead, see later). This is because the WebHDFS interface authentication
        and authorization mechanisms were recently moved to OAuth2. There was a
        general announcement, nevertheless if you missed it, you have all the
        details in this stack overflow.com question:

        http://stackoverflow.com/questions/31187977/oauth2-access-to-cosmos-webhdfs
        -in-fiware-lab

        Additionally, as said, we are using the HttpFS port (14000 instead of
        50070) because this allows us to hide the cluster details from the outside.

        Please, follow the guide above and tell me if you have any doubts.

        Regarding the enabler documentation, we are working hard to update it, it
        should be ready in the following days.

        Regards,
        Francisco

        El 8/8/15 0:45, "FW External User (JIRA)" <jira-help-desk@fi-ware.org>

        ________________________________

        Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

        The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

        Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
        _______________________________________________
        Fiware-tech-help mailing list
        Fiware-tech-help@lists.fi-ware.org
        https://lists.fi-ware.org/listinfo/fiware-tech-help

        Show
        fw.ext.user FW External User added a comment - Hi Sergio, You are not able to perform such an operation from outside because you have to append to the request an OAuth2 token (and use HttpFS port instead, see later). This is because the WebHDFS interface authentication and authorization mechanisms were recently moved to OAuth2. There was a general announcement, nevertheless if you missed it, you have all the details in this stack overflow.com question: http://stackoverflow.com/questions/31187977/oauth2-access-to-cosmos-webhdfs -in-fiware-lab Additionally, as said, we are using the HttpFS port (14000 instead of 50070) because this allows us to hide the cluster details from the outside. Please, follow the guide above and tell me if you have any doubts. Regarding the enabler documentation, we are working hard to update it, it should be ready in the following days. Regards, Francisco El 8/8/15 0:45, "FW External User (JIRA)" <jira-help-desk@fi-ware.org> ________________________________ Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción. The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it. Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição _______________________________________________ Fiware-tech-help mailing list Fiware-tech-help@lists.fi-ware.org https://lists.fi-ware.org/listinfo/fiware-tech-help
        Hide
        fw.ext.user FW External User added a comment -

        Hi Francisco,
        now it works!
        Thanks a lot for your helpful advices.

        Best regards.
        S.

        ----Messaggio originale----
        Da: FRANCISCO ROMERO BUENO francisco.romerobueno@telefonica.com
        Inviato: martedì 11 agosto 2015 09:17
        A: fiware-tech-help@lists.fi-ware.org
        Cc: sergio.delorenzis@albaproject.it
        Oggetto: Re: [FI-WARE-JIRA] (HELP-3762) FIWARE.Request.Lab.Data.BigData-Analysis.UnsuccessfullDataSaving

        Hi Sergio,

        You are not able to perform such an operation from outside because you have to append to the request an OAuth2 token (and use HttpFS port instead, see later). This is because the WebHDFS interface authentication and authorization mechanisms were recently moved to OAuth2. There was a general announcement, nevertheless if you missed it, you have all the details in this stack overflow.com question:

        http://stackoverflow.com/questions/31187977/oauth2-access-to-cosmos-webhdfs
        -in-fiware-lab

        Additionally, as said, we are using the HttpFS port (14000 instead of
        50070) because this allows us to hide the cluster details from the outside.

        Please, follow the guide above and tell me if you have any doubts.

        Regarding the enabler documentation, we are working hard to update it, it should be ready in the following days.

        Regards,
        Francisco

        El 8/8/15 0:45, "FW External User (JIRA)" <jira-help-desk@fi-ware.org>

        ________________________________

        Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

        The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

        Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição

        _______________________________________________
        Fiware-tech-help mailing list
        Fiware-tech-help@lists.fi-ware.org
        https://lists.fi-ware.org/listinfo/fiware-tech-help

        Show
        fw.ext.user FW External User added a comment - Hi Francisco, now it works! Thanks a lot for your helpful advices. Best regards. S. ---- Messaggio originale ---- Da: FRANCISCO ROMERO BUENO francisco.romerobueno@telefonica.com Inviato: martedì 11 agosto 2015 09:17 A: fiware-tech-help@lists.fi-ware.org Cc: sergio.delorenzis@albaproject.it Oggetto: Re: [FI-WARE-JIRA] ( HELP-3762 ) FIWARE.Request.Lab.Data.BigData-Analysis.UnsuccessfullDataSaving Hi Sergio, You are not able to perform such an operation from outside because you have to append to the request an OAuth2 token (and use HttpFS port instead, see later). This is because the WebHDFS interface authentication and authorization mechanisms were recently moved to OAuth2. There was a general announcement, nevertheless if you missed it, you have all the details in this stack overflow.com question: http://stackoverflow.com/questions/31187977/oauth2-access-to-cosmos-webhdfs -in-fiware-lab Additionally, as said, we are using the HttpFS port (14000 instead of 50070) because this allows us to hide the cluster details from the outside. Please, follow the guide above and tell me if you have any doubts. Regarding the enabler documentation, we are working hard to update it, it should be ready in the following days. Regards, Francisco El 8/8/15 0:45, "FW External User (JIRA)" <jira-help-desk@fi-ware.org> ________________________________ Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción. The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it. Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição _______________________________________________ Fiware-tech-help mailing list Fiware-tech-help@lists.fi-ware.org https://lists.fi-ware.org/listinfo/fiware-tech-help

          People

          • Assignee:
            frb Francisco Romero
            Reporter:
            fw.ext.user FW External User
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: