Details
-
Type:
extRequest
-
Status: Closed
-
Resolution: Done
-
Fix Version/s: 2021
-
Component/s: FIWARE-TECH-HELP
-
Labels:
-
Sender Email:
-
HD-Chapter:Data
-
HD-Enabler:Cosmos
Description
Dear Francisco,
I am the FI-ADOPT coach. miLeyenda team, a FI-ADOPT Accelerator SME, wants to do some big data processing, but need some tips to get started. We had a conversation with them in FLUA-2221, and in the end it may be better to proceed with your guidance. Here are some details from their original community account request ticket:
"miLeyenda works with a large volume of information, including scores, classifications, teams data and players data, etc. Because of this, we thought of using this enabler for the need to process that amount of information for statistical and informational purposes.
Based on the above, our line of work consists of three phases:
- Phase 1. Studying and understanding the operation, configuration and startup of the three components that form part of our BigData system.
- Phase 2. Designing and developing a representative scenario of miLeyenda to benefit from the advantages provided by Cosmos.
- Phase 3. Integrating BigData system in miLeyenda to achieve the statistical and informational purposes."
I pointed them towards the global instance of Cosmos in Fiware Lab (is that a good suggestion based on their purposes outlined above?) . Then they asked me the follow up questions below:
"What is the computational power of the global instance?.
With your instructions, I can access to the global instance, thanks.
When can I admin my own dashboard to manage my datasets?"
Could you kindly help the miLeyenda team with these questions?
Many thanks in advance.
ilknur
The issue has been emailed:
HELP-5815) Cosmos questions from FI-ADOPT SME, Mileyenda *Dear Francisco,
Here is the answer from the GE owner (Francisco Romero):
"Hi all,
Sorry for the late reply, your ticket was under a mountain of emails.
The Big Data enabler is suited for the kind of analysis you want to
perform, of course. Both in terms of storage (a large distributed cluster
in charge of holding large files of data, based on HDFS) and computing (a
large distributed cluster where to run MapReduce applications).
Regarding the global instance of FIWARE, we can say currently we are in
the middle of a migration from the ³old² instance to the ³new² one. The
old instance has become small because of the large number of users that
have chosen it for their developments; it was based on 10 virtual nodes
with a total storage amount of 1 TB. Now, we are finishing the deployment
of a couple of clusters, one specifically addressing storage and the other
one specifically addressing computing, with higher capabilities: 52.5 TB
distributed all along 11 physical nodes, and 14 computing nodes,
respectively. Users usually have a limited quota of 5 GB, but this could
be increased for sure in the new deployment.
All the details about the new deployment can be found here (work in
progress):
https://github.com/telefonicaid/fiware-cosmos/blob/master/doc/deployment_ex
amples/cosmos/fiware_lab.md
Regarding the administration of your datasets, we have adopted the
WebHDFS/HttpFS, as REST API that will allow you manage the HDFS I/O.
Nevertheless, it is under our roadmap to create a section within the
Cosmos portal in order to mange the data in a graphical way:
https://github.com/telefonicaid/fiware-cosmos/issues/94
Hope this can answer your questions.
Best regards,
Francisco"
Hope this helps.
ilknur