Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-14354

FIWARE.Request.Tech.Re: Regarding Issues In Cosmos Setup Locally.

    Details

    • Type: extRequest
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Fix Version/s: 2021
    • Component/s: FIWARE-TECH-HELP
    • Labels:
      None
    • HD-Enabler:
      Cosmos

      Description

      Dear Andres,

      Thank you very much for the help.

      We need you support again.

      We have managed to set up the cosmos server.
      We thought we have also successfully connected cygnus with the cosmos.
      There were some errors, that we have solved and achieved the position of no
      error in the connection.

      However we had tried to trabsfer data from Orion > Cygnus > Cosmos.

      The data was successfully transferred from Orion to Cygnus *but didn't
      arrive from **Cygnus to Cosmos. *

      There was no errors or anything else, so we aren't sure how to solve this
      ourselves.

      Are you able to review the cosmos server and advise?

      Thanks
      Tal

      On Tue, 26 Jun 2018, 15:15 Andres Muñoz, <andres.munoza@alumnos.upm.es>
      wrote:

      > Hi Tal,
      >
      > Your account have been created, please follow the official guide for
      > knowing how to storage and run mapreduce jobs.
      >
      >
      > https://github.com/ging/fiware-cosmos/blob/master/doc/manuals/quick_start_guide_new.md
      >
      > Best regards,
      >
      > On 06/26/2018 03:47 PM, Tal Zohar, ArtuData wrote:
      >
      > Thanks Andres,
      >
      > We have created an id:
      > "umangsuthar"
      >
      > Looking forward to the user access.
      >
      > Please continue reply all.
      >
      > Thanks
      > Tal
      >
      >
      >
      > On Tue, 26 Jun 2018, 15:09 Andres Muñoz, <andres.munoza@alumnos.upm.es>
      > wrote:
      >
      >> Hi Tal,
      >>
      >> I don't see in your email that you want access to the FIWARE Cosmos
      >> instance, but if you need that, please follow the following instructions:
      >>
      >>
      >> ##Step by step guide ###Step 1: Get an OAuth2 token All APIs in FIWARE
      >> Lab are protected by means of OAuth2 tokens. Cosmos is not an exception, so
      >> you will need to request to the Computing Endpoint a valid token for your
      >> FIWARE Lab user. curl tool can be used for that purpose:
      >>
      >> $ curl -k -X POST
      >> "https://computing.cosmos.lab.fiware.org:13000/cosmos-auth/v1/token"
      >> <https://computing.cosmos.lab.fiware.org:13000/cosmos-auth/v1/token> -H
      >> "Content-Type: application/x-www-form-urlencoded" -d
      >> "grant_type=password&username=<YOUR_USER_EMAIL>&password=<YOUR_PASSWORD>”
      >>
      >> Where username and password are the email and password you used when you
      >> registered in FIWARE Lab. You should get something like:
      >>
      >>

      {"access_token": "3azH09G1PdaGmgBNODLOtxy52f5a00", "token_type": >> "Bearer", "expires_in": 3600, "refresh_token": >> "V2Wlk7aFCnElKlW9BOmRzGhBtqgR2z"}

      >>
      >> The access_token field is the OAuth2 token.
      >>
      >> ###Step 2: Create a Cosmos account At the moment of writing, deploying a
      >> Cosmos Portal for FIWARE Lab is in the roadmap, but not yet done.
      >>
      >> Thus, in order to create an account you will have to send an email to
      >> francisco.romerobueno@telefonica.com specifying your FIWARE Lab ID.
      >>
      >> Such an ID can be obtained by querying FIWARE Lab's Identity Manager:
      >>
      >> $ curl -X GET "https://account.lab.fiware.org/user?access_token=<YOUR_OAUTH2_TOKEN>"
      >>
      >>
      >> The result of such a query for the user frb is:
      >>
      >> {"organizations": [], "displayName": "frb", "roles": [

      {"name": >> "provider", "id": "106"}

      ], "app_id": “9556cc76154361b3b43d7b31f0600982",
      >> "email": "frb@tid.es" <frb@tid.es>, "id": "frb”}
      >>
      >> The interesting part is the "id" field, in the above example frb.
      >>
      >> Hence, when you have the id, please send me an email with it and I will
      >> create you the user to access the different resources to the cosmos
      >> platform. As a mention previously we are still testing the infrastructure,
      >> for this reason, we are planning to give a full access the next week, but
      >> for saving time we can go forward in the user creation and resources
      >> assignation.
      >>
      >> Best regards,
      >>
      >> On 06/26/2018 02:57 PM, Tal Zohar, ArtuData wrote:
      >>
      >> Andres,
      >>
      >> Just confirming that you have seen our second question about getting
      >> access to Cosmos server?
      >>
      >> Thanks
      >> Tal
      >>
      >>
      >> On Tue, 26 Jun 2018, 14:52 Tal Zohar, ArtuData, <tal@artudata.com> wrote:
      >>
      >>> Andres,
      >>>
      >>> Thank you very much.
      >>>
      >>> Looping in our tech team.
      >>>
      >>> Thanks
      >>> Tal
      >>>
      >>>
      >>>
      >>> On Tue, 26 Jun 2018, 14:44 Andres Muñoz, <andres.munoza@alumnos.upm.es>
      >>> wrote:
      >>>
      >>>> Hi Tal,
      >>>>
      >>>> I will answer you to your problems in the same order that you presented:
      >>>>
      >>>> First problem 1 is regarding to the local setup of the cosmos instance,
      >>>> but if you look at the official documentation you have to Identify whether
      >>>> you really need to deploy all the big data architecture, a figure that can
      >>>> help you with that is :
      >>>>
      >>>> [image: Figure 1 - Which kind of Big Data user am I?]
      >>>>
      >>>> Thus, use only one instance for installing and deploying all the Cosmos
      >>>> infrastructure is not possible because you need a lot of resources and
      >>>> clustering components.
      >>>>
      >>>>
      >>>> On the other hand, the problem 2 regarding to Docker container of
      >>>> Cygnus, you can change the agent.conf file and then restart then service
      >>>> of Cygnus inside of the container in order that you have all the service
      >>>> active with the new configuration. Hence, if you are using the official
      >>>> docker images of cygnus you can re-run the Cygnus service using this
      >>>> example command:
      >>>>
      >>>> /opt/apache-flume/bin/cygnus-flume-ng agent --conf conf -f
      >>>> /opt/apache-flume/conf/agent.conf -n cygnus-ngsi
      >>>> -Dflume.root.logger=DEBUG,console
      >>>>
      >>>> Moreover, if you are loosing your configuration data you have to check
      >>>> how you are running the rocker container, one option for keeping the
      >>>> changes is using the -v option and send to the container a configuration
      >>>> file for Cygnus, but it is more related to docker usage than Cygnus.
      >>>>
      >>>> If you have more questions please let me know.
      >>>>
      >>>> Best Regards,
      >>>>
      >>>> Andres
      >>>>
      >>>>
      >>>>
      >>
      >

      __________________________________________________________________________________________

      You can get more information about our cookies and privacy policies on the following links:

      Fiware-tech-help mailing list
      Fiware-tech-help@lists.fiware.org
      https://lists.fiware.org/listinfo/fiware-tech-help

      [Created via e-mail received from: "Tal Zohar, ArtuData" <tal@artudata.com>]

      1. docker-compose.yml
        1 kB
        FW External User
      2. Error log FIWARE Cosmos 28062018.txt
        0.8 kB
        FW External User

        Activity

        Hide
        andres.munoza Andres Muñoz added a comment -

        Hi Tal,

        Please tell me if you are trying to store your data in the HDFS of Cosmos? and how you configure you agent.conf in Cygnus. Also the Cygnus logs will be helpful to identify the error

        Show
        andres.munoza Andres Muñoz added a comment - Hi Tal, Please tell me if you are trying to store your data in the HDFS of Cosmos? and how you configure you agent.conf in Cygnus. Also the Cygnus logs will be helpful to identify the error
        Hide
        fw.ext.user FW External User added a comment -

        Hi Andres,

        Good morning, hope you have a nice evening.

        As we continue our exploration of fixing it, please find enclosed the
        Docker code file and the error log.

        Looking forward to your urgent assistance, as we need to present the
        working solution of FIWARE Jury very soon.

        Thanks
        Tal

        Best Regards,

        Tal Zohar
        ArtuData
        Artificial Intelligence for everyone.

        www.ArtuData.com <http://www.ArtuData.com>

        On Wed, Jun 27, 2018 at 5:26 PM, Tal Zohar, ArtuData <tal@artudata.com>

        __________________________________________________________________________________________

        You can get more information about our cookies and privacy policies on the following links:

        Fiware-tech-help mailing list
        Fiware-tech-help@lists.fiware.org
        https://lists.fiware.org/listinfo/fiware-tech-help

        Show
        fw.ext.user FW External User added a comment - Hi Andres, Good morning, hope you have a nice evening. As we continue our exploration of fixing it, please find enclosed the Docker code file and the error log. Looking forward to your urgent assistance, as we need to present the working solution of FIWARE Jury very soon. Thanks Tal Best Regards, Tal Zohar ArtuData Artificial Intelligence for everyone. www.ArtuData.com < http://www.ArtuData.com > On Wed, Jun 27, 2018 at 5:26 PM, Tal Zohar, ArtuData <tal@artudata.com> __________________________________________________________________________________________ You can get more information about our cookies and privacy policies on the following links: http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE_Privacy_Policy http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/Cookies_Policy_FIWARE Fiware-tech-help mailing list Fiware-tech-help@lists.fiware.org https://lists.fiware.org/listinfo/fiware-tech-help
        Hide
        andres.munoza Andres Muñoz added a comment -

        Hi Tal,

        I have been checked your logs, and acoording to it cygnus cnanot stablish the conecction to the HDFS service hosted in COSMOS. Thus, and also checking your docker compose, your cygnus configuration have some bad parameters.

        First, The instance of FIWARE cosmos has 2 separate components, Computing and Storage so , if you want tu storage data you need to point to the storage component. Second, in a previous email I include the link to the oficial documentation please use that link ( https://github.com/ging/fiware-cosmos/blob/master/doc/manuals/quick_start_guide_new.md).

        Thus, in your docker compose you have to use:
        "CYGNUS_HDFS_HOST=storage.dit.upm.es"
        "CYGNUS_HDFS_PORT=14000"

        Moreover, you can stop and start the cygnus service getting into the container and use the commands to run the Cygnus service, you don't need to restart all the container. A good practice is stop the Cygnus service before make any change in the agent conf file and then start the service again.

        Show
        andres.munoza Andres Muñoz added a comment - Hi Tal, I have been checked your logs, and acoording to it cygnus cnanot stablish the conecction to the HDFS service hosted in COSMOS. Thus, and also checking your docker compose, your cygnus configuration have some bad parameters. First, The instance of FIWARE cosmos has 2 separate components, Computing and Storage so , if you want tu storage data you need to point to the storage component. Second, in a previous email I include the link to the oficial documentation please use that link ( https://github.com/ging/fiware-cosmos/blob/master/doc/manuals/quick_start_guide_new.md ). Thus, in your docker compose you have to use: "CYGNUS_HDFS_HOST=storage.dit.upm.es" "CYGNUS_HDFS_PORT=14000" Moreover, you can stop and start the cygnus service getting into the container and use the commands to run the Cygnus service, you don't need to restart all the container. A good practice is stop the Cygnus service before make any change in the agent conf file and then start the service again.
        Hide
        andres.munoza Andres Muñoz added a comment -

        Hi Tal,

        Also , You have to consider that The FIWARE instance of cosmos is only for testing purposes so , I think that the better way for you setting up your test is first using Cygnus to store your data in your local HDFS and then upload only the files that you need to analyze to the Cosmos platform because the Cosmos platform, only do batch processing and trying to sent real time data to Cosmos is not the best way to use it.

        Show
        andres.munoza Andres Muñoz added a comment - Hi Tal, Also , You have to consider that The FIWARE instance of cosmos is only for testing purposes so , I think that the better way for you setting up your test is first using Cygnus to store your data in your local HDFS and then upload only the files that you need to analyze to the Cosmos platform because the Cosmos platform, only do batch processing and trying to sent real time data to Cosmos is not the best way to use it.

          People

          • Assignee:
            andres.munoza Andres Muñoz
            Reporter:
            fw.ext.user FW External User
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: