Details
-
Type:
extRequest
-
Status: Closed
-
Resolution: Done
-
Fix Version/s: 2021
-
Component/s: FIWARE-TECH-HELP
-
Labels:
-
Sender Email:
-
HD-Chapter:Data
-
HD-Enabler:Cosmos
Description
Dear Francisco,
I am the FI-ADOPT coach. miLeyenda team, a FI-ADOPT Accelerator SME, wants to do some big data processing, but need some tips to get started. We had a conversation with them in FLUA-2221, and in the end it may be better to proceed with your guidance. Here are some details from their original community account request ticket:
"miLeyenda works with a large volume of information, including scores, classifications, teams data and players data, etc. Because of this, we thought of using this enabler for the need to process that amount of information for statistical and informational purposes.
Based on the above, our line of work consists of three phases:
- Phase 1. Studying and understanding the operation, configuration and startup of the three components that form part of our BigData system.
- Phase 2. Designing and developing a representative scenario of miLeyenda to benefit from the advantages provided by Cosmos.
- Phase 3. Integrating BigData system in miLeyenda to achieve the statistical and informational purposes."
I pointed them towards the global instance of Cosmos in Fiware Lab (is that a good suggestion based on their purposes outlined above?) . Then they asked me the follow up questions below:
"What is the computational power of the global instance?.
With your instructions, I can access to the global instance, thanks.
When can I admin my own dashboard to manage my datasets?"
Could you kindly help the miLeyenda team with these questions?
Many thanks in advance.
ilknur
Activity
Hi all,
Sorry for the late reply, your ticket was under a mountain of emails.
The Big Data enabler is suited for the kind of analysis you want to
perform, of course. Both in terms of storage (a large distributed cluster
in charge of holding large files of data, based on HDFS) and computing (a
large distributed cluster where to run MapReduce applications).
Regarding the global instance of FIWARE, we can say currently we are in
the middle of a migration from the ³old² instance to the ³new² one. The
old instance has become small because of the large number of users that
have chosen it for their developments; it was based on 10 virtual nodes
with a total storage amount of 1 TB. Now, we are finishing the deployment
of a couple of clusters, one specifically addressing storage and the other
one specifically addressing computing, with higher capabilities: 52.5 TB
distributed all along 11 physical nodes, and 14 computing nodes,
respectively. Users usually have a limited quota of 5 GB, but this could
be increased for sure in the new deployment.
All the details about the new deployment can be found here (work in
progress):
https://github.com/telefonicaid/fiware-cosmos/blob/master/doc/deployment_ex
amples/cosmos/fiware_lab.md
Regarding the administration of your datasets, we have adopted the
WebHDFS/HttpFS, as REST API that will allow you manage the HDFS I/O.
Nevertheless, it is under our roadmap to create a section within the
Cosmos portal in order to mange the data in a graphical way:
https://github.com/telefonicaid/fiware-cosmos/issues/94
Hope this can answer your questions.
Best regards,
Francisco
El 4/2/16 13:32, "ilknur chulani (JIRA)" <jira-help-desk@fi-ware.org>
________________________________
Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.
The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.
Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost.
Please, send your messages using the new domain (Fiware-tech-help@lists.fiware.org) instead of the old one.
_______________________________________________
Fiware-tech-help mailing list
Fiware-tech-help@lists.fiware.org
https://lists.fiware.org/listinfo/fiware-tech-help
The issue has been emailed:
- Time sent: 23/Feb/16 4:36 PM
- To: farcas@ucam.edu
- with subject: *(
HELP-5815) Cosmos questions from FI-ADOPT SME, Mileyenda *
Dear Francisco,
Here is the answer from the GE owner (Francisco Romero):
"Hi all,
Sorry for the late reply, your ticket was under a mountain of emails.
The Big Data enabler is suited for the kind of analysis you want to
perform, of course. Both in terms of storage (a large distributed cluster
in charge of holding large files of data, based on HDFS) and computing (a
large distributed cluster where to run MapReduce applications).
Regarding the global instance of FIWARE, we can say currently we are in
the middle of a migration from the ³old² instance to the ³new² one. The
old instance has become small because of the large number of users that
have chosen it for their developments; it was based on 10 virtual nodes
with a total storage amount of 1 TB. Now, we are finishing the deployment
of a couple of clusters, one specifically addressing storage and the other
one specifically addressing computing, with higher capabilities: 52.5 TB
distributed all along 11 physical nodes, and 14 computing nodes,
respectively. Users usually have a limited quota of 5 GB, but this could
be increased for sure in the new deployment.
All the details about the new deployment can be found here (work in
progress):
https://github.com/telefonicaid/fiware-cosmos/blob/master/doc/deployment_ex
amples/cosmos/fiware_lab.md
Regarding the administration of your datasets, we have adopted the
WebHDFS/HttpFS, as REST API that will allow you manage the HDFS I/O.
Nevertheless, it is under our roadmap to create a section within the
Cosmos portal in order to mange the data in a graphical way:
https://github.com/telefonicaid/fiware-cosmos/issues/94
Hope this can answer your questions.
Best regards,
Francisco"
Hope this helps.
ilknur
Dear Francisco Romero,
and here is the contact from miLeyenda team that asked these questions in the community account ticket (FLUA-2221): farcas@ucam.edu (Francsico Arcas)
Hope you can help him with some guidance.
Kind regards,
ilknur