Support for COSMOS installation for a SME selected by European Pioneers

Hi all,

(update to ticket: https://jira.fiware.org/browse/HELC-525 )

we started a discussion regarding installation of OpenVZ and have found following results.

• This GE uses OpenVZ. Is this a mandatory requirement?
• If not, can we setup this component in Docker environment to do integration testing?
• If yes, how should it be possible/makes sense to run OpenVZ on FIWARE which is OpenStack based? (Both are paravirtualization technologies)
• Please help to setup communication with the GE owner to clarify this, GameGenetics is blocked since March 18th!
• Additionally Wiki is very hard to read and is not very well structured
• What ist the state of the art here, Puppet or Chef? Both are available
• Why is an IP hardcoded in a Chef recipe?
• How to dynamically scale the cluster?

GameGenetics has provided an Ansible automation and a Dockerfile.
https://github.com/gamegenetics/cosmos

We have also tried a Docker integration and after copying first Wiki docs line we got an error regarding missing components
[0m [91merror: Failed dependencies:
[0m [91m /sbin/grubby is needed by vzkernel-2.6.32-042stab079.6.x86_64
[0m [91m /sbin/new-kernel-pkg is needed by vzkernel-2.6.32-042stab079.6.x86_64
[0m [91m /sbin/mkinitrd is needed by vzkernel-2.6.32-042stab079.6.x86_64

Kind regards / viele Grüße & Servus
Peter Muryshkin, M. Sc.

Skype ID: peter_iais
LinkedIn: http://de.linkedin.com/in/muryshkin

----------------------------------------------------------------------------------
NetMedia / B3-342
Fraunhofer IAIS
Schloss Birlinghoven 1
53754 St. Augustin, Germany
T: +49 2241 14 3413
Email: Peter.Muryshkin@iais.fraunhofer.de

Hi all,

first of all I'd like to apologize regarding the dealy in answering you. Last two weeks I was partially or totally out of the office because of the FIWARE Developer's Week that took place at Brussels and the Easter holidays.

Regarding Cosmos, this is a software designed for providing "Hadoop As A Service", i.e. a tool that manages a large pool of physical infrastructure in order to allocate/deallocate machines for creating/destroying private Hadoop clusters on demand.

Being said that, due to the large amount of resources it requires it is not usually installed by the users. Instead of that, people install their own private Hadoop cluster in their own infrastructure, or they use the shared Hadoop cluster at FIWARE LAB working as a global instance of the Big Data enabler. This is stated in the "Creating Instances" tab of the Catalogue (http://catalogue.fiware.org/enablers/bigdata-analysis-cosmos/creating-instances). You only have to create an account at http://cosmos.lab.fi-ware.org/cosmos-gui/ by using your FIWARE LAB credentials.

Around the global instance, we started creating a complete ecosystem of tools that are effectively available for the community:

• Cygnus, a connector between Orion CB and HDFS (https://github.com/telefonicaid/fiware-cygnus)
• Tidoop, some Hadoop extensions for working with CKAN Open Data (https://github.com/telefonicaid/fiware-tidoop)

Regards,
Francisco

thank you so far, will discuss this with SME.

Hi Francisco,
Thank you for the clarification. At GameGenetics, we already run a hadoop cluster. We are looking for a way to manage an arbitrary number of hadoop clusters in flexible configurations. The basic idea is to provide different teams and projects with on-demand hadoop resources as they need (i.e. even for short amounts of time). Our infrastructure is extremely flexible and therefore we want to be able to use automatic software resource allocation whereever possible. Arguably, it would be useless to have an API for allocating hardware when the software doesn't have such an API.
From a high level perspective, we understood that the Cosmos project provides such functionality?

Either way, how can we proceed?
Kind regards,
-paul

(Francisco)
Dear all,

We have been discussing this internally for a few days, before giving you a definitive answer. Since some time ago we have been thinking on migrating to Sahara, the Openstack’s platform for Hadoop (and Spark) cluster provisioning (https://wiki.openstack.org/wiki/Sahara). Thus, we are at a point where we think the best is not to start supporting new Cosmos installations, and start redefining the Big Data GE implementation in order future deployments are effectively done by using Sahara instead of Cosmos.

Thus, regarding your needs I think there are two options:
1. To wait until such Big Data GE Sahara-based redefinition is accomplished. This can take a couple of sprints, but then you will have access to a fully supported implementation including Sahara and some plugins we are envisioning.
2. To install by your own Sahara, if you cannot wait until the above is accomplished.
Hope you can understand our point of view.

Regards,
Francisco

Dear all,
Thank you for the update. From what I can tell, it seems to make sense from an architectural/technical point of view to build on top of an existing open source project like Sahara. However, I fail to understand the technical pivot happening at this point in time. We are looking for a specific set of tools/technologies that enables us to work with big data/hadoop clusters. The first hurdle we ran into is now left unaddressed. One of the two options provided is to implement everything by ourselves, whereas the other one is to wait for an unspecified amount of time in the hope that any hurdles will actually be addressed.
I am not sure if this is what we expected from the promised GE featureset description. Are there any other options at all?

kind regards,
-paul kremer

De: Jan Thomas <jan.thomas@gamegenetics.de>
Fecha: martes, 28 de abril de 2015, 16:01
Para: "thierry.nagellen@orange.com" <thierry.nagellen@orange.com>, Paul Kremer <paul.kremer@gamegenetics.de>
CC: Francisco Romero Bueno <francisco.romerobueno@telefonica.com>, "Muryshkin, Peter" <Peter.Muryshkin@iais.fraunhofer.de>, Maximilian Weber <maximilian.weber@gamegenetics.de>, SERGIO GARCIA GOMEZ <sergio.garciagomez@telefonica.com>, SANTIAGO MARTINEZ GARCIA <santiago.martinezgarcia@telefonica.com>, Moti Tal <moti.tal@gamegenetics.de>
Asunto: Re: REMINDER`AW: Discussion on Cosmos GE

Hi Thierry,

Thanks for responding. We have updated our European Pioneers Programme Managers about the problem (Peter is included on this as well) and asked for their advise, as we are unsure how to proceed at the moment.

Please let us know if there are any updates regarding a timeline for the new version.

Best, Jan

Am 28.04.2015 um 11:32 schrieb thierry.nagellen@orange.com:
Dear Jan,

As European Piooners coach, I think FIWARE cannot provide a better answer for the moment. We are pushing to prioritize the new version of Cosmos but I do not have any deadline currently.

BR
Thierry

De : Jan Thomas jan.thomas@gamegenetics.de
Envoyé : jeudi 23 avril 2015 23:01
À : Paul Kremer
Cc : FRANCISCO ROMERO BUENO; Muryshkin, Peter; NAGELLEN Thierry IMT/OLPS; Maximilian Weber; SERGIO GARCIA GOMEZ; SANTIAGO MARTINEZ GARCIA; Moti Tal
Objet : Re: REMINDER`AW: Discussion on Cosmos GE

Hi Peter & Francisco,

Can you please confirm that there are really no other options for us to use Cosmos GE than
1. figuring out how to deal with the issues encountered in the current version by ourselves
2. waiting until ? for a new version, where we'd still have see that it works

We're part of the European Pioneers funding programme and are investing time trying to utilise this technology. We need to complete this within the next couple of months, and can't afford to start from scratch by again trying to evaluate which GE could make sense for our business. Is there really nothing else than could be done?

Appreciate your help to resolve this quickly.

Best, Jan

Head of Product
GameGenetics

Additional information: we run a reference virtual integration environment within FIC2Lab and get the following error:

time="2015-06-22T13:04:12Z" level=info msg="The command [/bin/sh -c rpm -Uvh vzkernel-2.6.32-042stab079.6.x86_64.rpm] returned a non-zero code: 1"

Please contact peter.muryshkin@iais.fraunhofer.de; maybe in this case Docker integration does not make sense at all and we should run a pure FIWARE VM test fixture?