Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-5815

FIWARE.Request.Tech.Data.BigData-Analysis.Cosmos questions from FI-ADOPT SME, Mileyenda.

    Details

      Description

      Dear Francisco,

      I am the FI-ADOPT coach. miLeyenda team, a FI-ADOPT Accelerator SME, wants to do some big data processing, but need some tips to get started. We had a conversation with them in FLUA-2221, and in the end it may be better to proceed with your guidance. Here are some details from their original community account request ticket:

      "miLeyenda works with a large volume of information, including scores, classifications, teams data and players data, etc. Because of this, we thought of using this enabler for the need to process that amount of information for statistical and informational purposes.

      Based on the above, our line of work consists of three phases:

      • Phase 1. Studying and understanding the operation, configuration and startup of the three components that form part of our BigData system.
      • Phase 2. Designing and developing a representative scenario of miLeyenda to benefit from the advantages provided by Cosmos.
      • Phase 3. Integrating BigData system in miLeyenda to achieve the statistical and informational purposes."

      I pointed them towards the global instance of Cosmos in Fiware Lab (is that a good suggestion based on their purposes outlined above?) . Then they asked me the follow up questions below:

      "What is the computational power of the global instance?.
      With your instructions, I can access to the global instance, thanks.
      When can I admin my own dashboard to manage my datasets?"

      Could you kindly help the miLeyenda team with these questions?

      Many thanks in advance.

      ilknur

        Activity

        ichulani ilknur chulani created issue -
        ichulani ilknur chulani made changes -
        Field Original Value New Value
        Link This issue clones FLUA-2221 [ FLUA-2221 ]
        ichulani ilknur chulani made changes -
        Preferred FIWARE Lab Node Spain [ 10428 ]
        total harddisk 40
        Project FIWARE LAB Upgrade Account [ 11101 ] Help-Desk [ 10841 ]
        Key FLUA-2241 HELP-5815
        Issue Type UpgradeAccount [ 10200 ] eRequest [ 10101 ]
        Workflow FIWARE Lab User Account Upgrade [ 41867 ] FW eRequest Workflow [ 41867 ]
        Component/s FIWARE-TECH-HELP [ 10278 ]
        Component/s FI-ADOPT [ 10609 ]
        Sender Email farcas@ucam.edu
        Company Mileyenda
        Number of VMs 1
        # public IPs 1
        Account status Pending [ 10602 ]
        Accelerator submission name FP7-ICT-632868
        total # vCPUs 2
        Are you already registered in FIWARE Lab? Yes [ 10603 ]
        total RAM 4
        Number of developers 4
        ichulani ilknur chulani made changes -
        Labels fiware-cosmos
        ichulani ilknur chulani made changes -
        Summary BigData questions from FI-ADOPT SME Cosmos questions from FI-ADOPT SME, Mileyenda
        ichulani ilknur chulani made changes -
        Description miLeyenda works with a large volume of information, including scores, classifications, teams data and players data, etc. Because of this, we thought of using this enabler for the need to process that amount of information for statistical and informational purposes.

        Based on the above, our line of work consists of three phases:
        - Phase 1. Studying and understanding the operation, configuration and startup of the three components that form part of our BigData system.

        - Phase 2. Designing and developing a representative scenario of miLeyenda to benefit from the advantages provided by Cosmos.

        - Phase 3. Integrating BigData system in miLeyenda to achieve the statistical and informational purposes.
        Dear Francisco,

        I am the FI-ADOPT coach. miLeyenda team, a FI-ADOPT Accelerator SME, wants to do some big data processing, but need some tips to get started. We had a conversation with them in FLUA-2221, and in the end it may be better to proceed with your guidance. Here are some details from their original community account request ticket:
         
        "miLeyenda works with a large volume of information, including scores, classifications, teams data and players data, etc. Because of this, we thought of using this enabler for the need to process that amount of information for statistical and informational purposes.

        Based on the above, our line of work consists of three phases:
        - Phase 1. Studying and understanding the operation, configuration and startup of the three components that form part of our BigData system.

        - Phase 2. Designing and developing a representative scenario of miLeyenda to benefit from the advantages provided by Cosmos.

        - Phase 3. Integrating BigData system in miLeyenda to achieve the statistical and informational purposes."

        I pointed them towards the global instance of Cosmos in Fiware Lab (is that a good suggestion based on their purposes outlined above?) . Then they asked me the follow up questions below:

        "What is the computational power of the global instance?.
        With your instructions, I can access to the global instance, thanks.
        When can I admin my own dashboard to manage my datasets?"

        Could you kindly help the miLeyenda team with these questions?

        Many thanks in advance.

        ilknur
        ichulani ilknur chulani made changes -
        Assignee ilknur chulani [ ilknur.chulani ] Francisco Romero [ frb ]
        Hide
        ichulani ilknur chulani added a comment -

        Dear Francisco Romero,

        and here is the contact from miLeyenda team that asked these questions in the community account ticket (FLUA-2221): farcas@ucam.edu (Francsico Arcas)

        Hope you can help him with some guidance.

        Kind regards,

        ilknur

        Show
        ichulani ilknur chulani added a comment - Dear Francisco Romero, and here is the contact from miLeyenda team that asked these questions in the community account ticket (FLUA-2221): farcas@ucam.edu (Francsico Arcas) Hope you can help him with some guidance. Kind regards, ilknur
        Hide
        fw.ext.user FW External User added a comment -

        Hi all,

        Sorry for the late reply, your ticket was under a mountain of emails.

        The Big Data enabler is suited for the kind of analysis you want to
        perform, of course. Both in terms of storage (a large distributed cluster
        in charge of holding large files of data, based on HDFS) and computing (a
        large distributed cluster where to run MapReduce applications).

        Regarding the global instance of FIWARE, we can say currently we are in
        the middle of a migration from the ³old² instance to the ³new² one. The
        old instance has become small because of the large number of users that
        have chosen it for their developments; it was based on 10 virtual nodes
        with a total storage amount of 1 TB. Now, we are finishing the deployment
        of a couple of clusters, one specifically addressing storage and the other
        one specifically addressing computing, with higher capabilities: 52.5 TB
        distributed all along 11 physical nodes, and 14 computing nodes,
        respectively. Users usually have a limited quota of 5 GB, but this could
        be increased for sure in the new deployment.

        All the details about the new deployment can be found here (work in
        progress):
        https://github.com/telefonicaid/fiware-cosmos/blob/master/doc/deployment_ex
        amples/cosmos/fiware_lab.md

        Regarding the administration of your datasets, we have adopted the
        WebHDFS/HttpFS, as REST API that will allow you manage the HDFS I/O.
        Nevertheless, it is under our roadmap to create a section within the
        Cosmos portal in order to mange the data in a graphical way:
        https://github.com/telefonicaid/fiware-cosmos/issues/94

        Hope this can answer your questions.

        Best regards,
        Francisco

        El 4/2/16 13:32, "ilknur chulani (JIRA)" <jira-help-desk@fi-ware.org>

        ________________________________

        Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

        The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

        Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
        Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost.
        Please, send your messages using the new domain (Fiware-tech-help@lists.fiware.org) instead of the old one.
        _______________________________________________
        Fiware-tech-help mailing list
        Fiware-tech-help@lists.fiware.org
        https://lists.fiware.org/listinfo/fiware-tech-help

        Show
        fw.ext.user FW External User added a comment - Hi all, Sorry for the late reply, your ticket was under a mountain of emails. The Big Data enabler is suited for the kind of analysis you want to perform, of course. Both in terms of storage (a large distributed cluster in charge of holding large files of data, based on HDFS) and computing (a large distributed cluster where to run MapReduce applications). Regarding the global instance of FIWARE, we can say currently we are in the middle of a migration from the ³old² instance to the ³new² one. The old instance has become small because of the large number of users that have chosen it for their developments; it was based on 10 virtual nodes with a total storage amount of 1 TB. Now, we are finishing the deployment of a couple of clusters, one specifically addressing storage and the other one specifically addressing computing, with higher capabilities: 52.5 TB distributed all along 11 physical nodes, and 14 computing nodes, respectively. Users usually have a limited quota of 5 GB, but this could be increased for sure in the new deployment. All the details about the new deployment can be found here (work in progress): https://github.com/telefonicaid/fiware-cosmos/blob/master/doc/deployment_ex amples/cosmos/fiware_lab.md Regarding the administration of your datasets, we have adopted the WebHDFS/HttpFS, as REST API that will allow you manage the HDFS I/O. Nevertheless, it is under our roadmap to create a section within the Cosmos portal in order to mange the data in a graphical way: https://github.com/telefonicaid/fiware-cosmos/issues/94 Hope this can answer your questions. Best regards, Francisco El 4/2/16 13:32, "ilknur chulani (JIRA)" <jira-help-desk@fi-ware.org> ________________________________ Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción. The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it. Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost. Please, send your messages using the new domain (Fiware-tech-help@lists.fiware.org) instead of the old one. _______________________________________________ Fiware-tech-help mailing list Fiware-tech-help@lists.fiware.org https://lists.fiware.org/listinfo/fiware-tech-help
        frb Francisco Romero made changes -
        Status Open [ 1 ] Answered [ 10104 ]
        Hide
        ichulani ilknur chulani added a comment -

        The issue has been emailed:

        • Time sent: 23/Feb/16 4:36 PM
        • To: farcas@ucam.edu
        • with subject: *(HELP-5815) Cosmos questions from FI-ADOPT SME, Mileyenda *

        Dear Francisco,

        Here is the answer from the GE owner (Francisco Romero):

        "Hi all,
        Sorry for the late reply, your ticket was under a mountain of emails.
        The Big Data enabler is suited for the kind of analysis you want to
        perform, of course. Both in terms of storage (a large distributed cluster
        in charge of holding large files of data, based on HDFS) and computing (a
        large distributed cluster where to run MapReduce applications).
        Regarding the global instance of FIWARE, we can say currently we are in
        the middle of a migration from the ³old² instance to the ³new² one. The
        old instance has become small because of the large number of users that
        have chosen it for their developments; it was based on 10 virtual nodes
        with a total storage amount of 1 TB. Now, we are finishing the deployment
        of a couple of clusters, one specifically addressing storage and the other
        one specifically addressing computing, with higher capabilities: 52.5 TB
        distributed all along 11 physical nodes, and 14 computing nodes,
        respectively. Users usually have a limited quota of 5 GB, but this could
        be increased for sure in the new deployment.
        All the details about the new deployment can be found here (work in
        progress):
        https://github.com/telefonicaid/fiware-cosmos/blob/master/doc/deployment_ex
        amples/cosmos/fiware_lab.md
        Regarding the administration of your datasets, we have adopted the
        WebHDFS/HttpFS, as REST API that will allow you manage the HDFS I/O.
        Nevertheless, it is under our roadmap to create a section within the
        Cosmos portal in order to mange the data in a graphical way:
        https://github.com/telefonicaid/fiware-cosmos/issues/94
        Hope this can answer your questions.
        Best regards,
        Francisco"

        Hope this helps.

        ilknur

        Show
        ichulani ilknur chulani added a comment - The issue has been emailed: Time sent: 23/Feb/16 4:36 PM To: farcas@ucam.edu with subject: *( HELP-5815 ) Cosmos questions from FI-ADOPT SME, Mileyenda * Dear Francisco, Here is the answer from the GE owner (Francisco Romero): "Hi all, Sorry for the late reply, your ticket was under a mountain of emails. The Big Data enabler is suited for the kind of analysis you want to perform, of course. Both in terms of storage (a large distributed cluster in charge of holding large files of data, based on HDFS) and computing (a large distributed cluster where to run MapReduce applications). Regarding the global instance of FIWARE, we can say currently we are in the middle of a migration from the ³old² instance to the ³new² one. The old instance has become small because of the large number of users that have chosen it for their developments; it was based on 10 virtual nodes with a total storage amount of 1 TB. Now, we are finishing the deployment of a couple of clusters, one specifically addressing storage and the other one specifically addressing computing, with higher capabilities: 52.5 TB distributed all along 11 physical nodes, and 14 computing nodes, respectively. Users usually have a limited quota of 5 GB, but this could be increased for sure in the new deployment. All the details about the new deployment can be found here (work in progress): https://github.com/telefonicaid/fiware-cosmos/blob/master/doc/deployment_ex amples/cosmos/fiware_lab.md Regarding the administration of your datasets, we have adopted the WebHDFS/HttpFS, as REST API that will allow you manage the HDFS I/O. Nevertheless, it is under our roadmap to create a section within the Cosmos portal in order to mange the data in a graphical way: https://github.com/telefonicaid/fiware-cosmos/issues/94 Hope this can answer your questions. Best regards, Francisco" Hope this helps. ilknur
        frb Francisco Romero made changes -
        Resolution Done [ 10000 ]
        Status Answered [ 10104 ] Done [ 10000 ]
        mev Manuel Escriche made changes -
        Issue Type eRequest [ 10101 ] extRequest [ 10104 ]
        Workflow FW eRequest Workflow [ 41868 ] FW extRequest Workflow-II [ 41868 ]
        Status Done [ 10000 ] Closed [ 6 ]
        backlogmanager Backlog Manager made changes -
        Summary Cosmos questions from FI-ADOPT SME, Mileyenda FIWARE.Request.Tech.Cosmos questions from FI-ADOPT SME, Mileyenda.
        mev Manuel Escriche made changes -
        HD-Enabler Cosmos [ 10872 ]
        HD-Chapter Data [ 10838 ]
        backlogmanager Backlog Manager made changes -
        Summary FIWARE.Request.Tech.Cosmos questions from FI-ADOPT SME, Mileyenda. FIWARE.Request.Tech.Data.BigData-Analysis.Cosmos questions from FI-ADOPT SME, Mileyenda.
        fla Fernando Lopez made changes -
        Fix Version/s 2021 [ 12600 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Answered Answered
        18d 20h 21m 1 Francisco Romero 23/Feb/16 9:39 AM
        Answered Answered Done Done
        6d 1h 19m 1 Francisco Romero 29/Feb/16 10:58 AM
        Done Done Closed Closed
        38d 23h 1m 1 Manuel Escriche 08/Apr/16 11:00 AM

          People

          • Assignee:
            frb Francisco Romero
            Reporter:
            ichulani ilknur chulani
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: