Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-5815

FIWARE.Request.Tech.Data.BigData-Analysis.Cosmos questions from FI-ADOPT SME, Mileyenda.

    Details

      Description

      Dear Francisco,

      I am the FI-ADOPT coach. miLeyenda team, a FI-ADOPT Accelerator SME, wants to do some big data processing, but need some tips to get started. We had a conversation with them in FLUA-2221, and in the end it may be better to proceed with your guidance. Here are some details from their original community account request ticket:

      "miLeyenda works with a large volume of information, including scores, classifications, teams data and players data, etc. Because of this, we thought of using this enabler for the need to process that amount of information for statistical and informational purposes.

      Based on the above, our line of work consists of three phases:

      • Phase 1. Studying and understanding the operation, configuration and startup of the three components that form part of our BigData system.
      • Phase 2. Designing and developing a representative scenario of miLeyenda to benefit from the advantages provided by Cosmos.
      • Phase 3. Integrating BigData system in miLeyenda to achieve the statistical and informational purposes."

      I pointed them towards the global instance of Cosmos in Fiware Lab (is that a good suggestion based on their purposes outlined above?) . Then they asked me the follow up questions below:

      "What is the computational power of the global instance?.
      With your instructions, I can access to the global instance, thanks.
      When can I admin my own dashboard to manage my datasets?"

      Could you kindly help the miLeyenda team with these questions?

      Many thanks in advance.

      ilknur

        Activity

        Hide
        ichulani ilknur chulani added a comment -

        Dear Francisco Romero,

        and here is the contact from miLeyenda team that asked these questions in the community account ticket (FLUA-2221): farcas@ucam.edu (Francsico Arcas)

        Hope you can help him with some guidance.

        Kind regards,

        ilknur

        Show
        ichulani ilknur chulani added a comment - Dear Francisco Romero, and here is the contact from miLeyenda team that asked these questions in the community account ticket (FLUA-2221): farcas@ucam.edu (Francsico Arcas) Hope you can help him with some guidance. Kind regards, ilknur
        Hide
        fw.ext.user FW External User added a comment -

        Hi all,

        Sorry for the late reply, your ticket was under a mountain of emails.

        The Big Data enabler is suited for the kind of analysis you want to
        perform, of course. Both in terms of storage (a large distributed cluster
        in charge of holding large files of data, based on HDFS) and computing (a
        large distributed cluster where to run MapReduce applications).

        Regarding the global instance of FIWARE, we can say currently we are in
        the middle of a migration from the ³old² instance to the ³new² one. The
        old instance has become small because of the large number of users that
        have chosen it for their developments; it was based on 10 virtual nodes
        with a total storage amount of 1 TB. Now, we are finishing the deployment
        of a couple of clusters, one specifically addressing storage and the other
        one specifically addressing computing, with higher capabilities: 52.5 TB
        distributed all along 11 physical nodes, and 14 computing nodes,
        respectively. Users usually have a limited quota of 5 GB, but this could
        be increased for sure in the new deployment.

        All the details about the new deployment can be found here (work in
        progress):
        https://github.com/telefonicaid/fiware-cosmos/blob/master/doc/deployment_ex
        amples/cosmos/fiware_lab.md

        Regarding the administration of your datasets, we have adopted the
        WebHDFS/HttpFS, as REST API that will allow you manage the HDFS I/O.
        Nevertheless, it is under our roadmap to create a section within the
        Cosmos portal in order to mange the data in a graphical way:
        https://github.com/telefonicaid/fiware-cosmos/issues/94

        Hope this can answer your questions.

        Best regards,
        Francisco

        El 4/2/16 13:32, "ilknur chulani (JIRA)" <jira-help-desk@fi-ware.org>

        ________________________________

        Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

        The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

        Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
        Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost.
        Please, send your messages using the new domain (Fiware-tech-help@lists.fiware.org) instead of the old one.
        _______________________________________________
        Fiware-tech-help mailing list
        Fiware-tech-help@lists.fiware.org
        https://lists.fiware.org/listinfo/fiware-tech-help

        Show
        fw.ext.user FW External User added a comment - Hi all, Sorry for the late reply, your ticket was under a mountain of emails. The Big Data enabler is suited for the kind of analysis you want to perform, of course. Both in terms of storage (a large distributed cluster in charge of holding large files of data, based on HDFS) and computing (a large distributed cluster where to run MapReduce applications). Regarding the global instance of FIWARE, we can say currently we are in the middle of a migration from the ³old² instance to the ³new² one. The old instance has become small because of the large number of users that have chosen it for their developments; it was based on 10 virtual nodes with a total storage amount of 1 TB. Now, we are finishing the deployment of a couple of clusters, one specifically addressing storage and the other one specifically addressing computing, with higher capabilities: 52.5 TB distributed all along 11 physical nodes, and 14 computing nodes, respectively. Users usually have a limited quota of 5 GB, but this could be increased for sure in the new deployment. All the details about the new deployment can be found here (work in progress): https://github.com/telefonicaid/fiware-cosmos/blob/master/doc/deployment_ex amples/cosmos/fiware_lab.md Regarding the administration of your datasets, we have adopted the WebHDFS/HttpFS, as REST API that will allow you manage the HDFS I/O. Nevertheless, it is under our roadmap to create a section within the Cosmos portal in order to mange the data in a graphical way: https://github.com/telefonicaid/fiware-cosmos/issues/94 Hope this can answer your questions. Best regards, Francisco El 4/2/16 13:32, "ilknur chulani (JIRA)" <jira-help-desk@fi-ware.org> ________________________________ Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción. The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it. Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost. Please, send your messages using the new domain (Fiware-tech-help@lists.fiware.org) instead of the old one. _______________________________________________ Fiware-tech-help mailing list Fiware-tech-help@lists.fiware.org https://lists.fiware.org/listinfo/fiware-tech-help
        Hide
        ichulani ilknur chulani added a comment -

        The issue has been emailed:

        • Time sent: 23/Feb/16 4:36 PM
        • To: farcas@ucam.edu
        • with subject: *(HELP-5815) Cosmos questions from FI-ADOPT SME, Mileyenda *

        Dear Francisco,

        Here is the answer from the GE owner (Francisco Romero):

        "Hi all,
        Sorry for the late reply, your ticket was under a mountain of emails.
        The Big Data enabler is suited for the kind of analysis you want to
        perform, of course. Both in terms of storage (a large distributed cluster
        in charge of holding large files of data, based on HDFS) and computing (a
        large distributed cluster where to run MapReduce applications).
        Regarding the global instance of FIWARE, we can say currently we are in
        the middle of a migration from the ³old² instance to the ³new² one. The
        old instance has become small because of the large number of users that
        have chosen it for their developments; it was based on 10 virtual nodes
        with a total storage amount of 1 TB. Now, we are finishing the deployment
        of a couple of clusters, one specifically addressing storage and the other
        one specifically addressing computing, with higher capabilities: 52.5 TB
        distributed all along 11 physical nodes, and 14 computing nodes,
        respectively. Users usually have a limited quota of 5 GB, but this could
        be increased for sure in the new deployment.
        All the details about the new deployment can be found here (work in
        progress):
        https://github.com/telefonicaid/fiware-cosmos/blob/master/doc/deployment_ex
        amples/cosmos/fiware_lab.md
        Regarding the administration of your datasets, we have adopted the
        WebHDFS/HttpFS, as REST API that will allow you manage the HDFS I/O.
        Nevertheless, it is under our roadmap to create a section within the
        Cosmos portal in order to mange the data in a graphical way:
        https://github.com/telefonicaid/fiware-cosmos/issues/94
        Hope this can answer your questions.
        Best regards,
        Francisco"

        Hope this helps.

        ilknur

        Show
        ichulani ilknur chulani added a comment - The issue has been emailed: Time sent: 23/Feb/16 4:36 PM To: farcas@ucam.edu with subject: *( HELP-5815 ) Cosmos questions from FI-ADOPT SME, Mileyenda * Dear Francisco, Here is the answer from the GE owner (Francisco Romero): "Hi all, Sorry for the late reply, your ticket was under a mountain of emails. The Big Data enabler is suited for the kind of analysis you want to perform, of course. Both in terms of storage (a large distributed cluster in charge of holding large files of data, based on HDFS) and computing (a large distributed cluster where to run MapReduce applications). Regarding the global instance of FIWARE, we can say currently we are in the middle of a migration from the ³old² instance to the ³new² one. The old instance has become small because of the large number of users that have chosen it for their developments; it was based on 10 virtual nodes with a total storage amount of 1 TB. Now, we are finishing the deployment of a couple of clusters, one specifically addressing storage and the other one specifically addressing computing, with higher capabilities: 52.5 TB distributed all along 11 physical nodes, and 14 computing nodes, respectively. Users usually have a limited quota of 5 GB, but this could be increased for sure in the new deployment. All the details about the new deployment can be found here (work in progress): https://github.com/telefonicaid/fiware-cosmos/blob/master/doc/deployment_ex amples/cosmos/fiware_lab.md Regarding the administration of your datasets, we have adopted the WebHDFS/HttpFS, as REST API that will allow you manage the HDFS I/O. Nevertheless, it is under our roadmap to create a section within the Cosmos portal in order to mange the data in a graphical way: https://github.com/telefonicaid/fiware-cosmos/issues/94 Hope this can answer your questions. Best regards, Francisco" Hope this helps. ilknur

          People

          • Assignee:
            frb Francisco Romero
            Reporter:
            ichulani ilknur chulani
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: