Details
-
Type:
extRequest
-
Status: Closed
-
Priority:
Major
-
Resolution: Done
-
Fix Version/s: Sprint 4.1.1
-
Component/s: FIWARE-TECH-HELP
-
Labels:None
-
Sender Email:
-
HD-Chapter:Data
-
HD-Enabler:Cosmos
Description
Hello can you please direct this question to right person.
Thank you.
---------- Forwarded message ----------
From: Dave Watson <watson.dave@gmail.com>
Date: Mon, Sep 29, 2014 at 7:07 PM
Subject: Question re FIWARE COSMOS
To: francisco.romerobueno@telefonica.com, frb@tid.es
Hello Francisco:
We attended FI-WARE meeting today in London and it was suggested we direct
question to you.
1) Do you support Parquet files in the HDFS? (Your documentation only shows
example of CSV file)
2) Will the version of Hadoop in Cosmos be upgraded so we can work with
Spark and Drill?
3) What is the memory configuration for nodes in the FIWARE Lab and Testbed
instances?
See below regarding Drill 0.5 version support:
Drill now uses the Hadoop 2.4.1 APIs. This includes upgrading Parquet to
use direct memory and the ability to write larger Parquet files
https://blogs.apache.org/drill/entry/apache_drill_beta_release_see
https://cwiki.apache.org/confluence/display/DRILL/Apache+Drill+0.5.0+Release+Notes
Thanks for your help here.
Dave
–
Dave Watson
+447741675325
–
Dave Watson
+447741675325
_______________________________________________
Fiware-tech-help mailing list
Fiware-tech-help@lists.fi-ware.org
https://lists.fi-ware.org/listinfo/fiware-tech-help
Activity
| Field | Original Value | New Value |
|---|---|---|
| Assignee | Manuel Escriche [Adm] [ mev ] | Francisco Romero Bueno [ frb ] |
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
| Resolution | Done [ 10000 ] | |
| Status | In Progress [ 3 ] | Done [ 10000 ] |
| Component/s | FIWARE-TECH-HELP [ 10278 ] |
| Workflow | Basic Workflow [ 15419 ] | FW eRequest Workflow [ 16038 ] |
| Issue Type | Question [ 10004 ] | eRequest [ 10101 ] |
| FI-WARE Environment | FI-LAB [ 10100 ] | |
| Resolution | Done [ 10000 ] |
| Resolution | Done [ 10000 ] |
| Workflow | FW eRequest Workflow [ 16038 ] | eRequest Workflow-2 [ 16354 ] |
| Status | Done [ 10000 ] | In Progress [ 3 ] |
| Status | In Progress [ 3 ] | Done [ 10000 ] |
| Workflow | eRequest Workflow-2 [ 16354 ] | FW eRequest Workflow [ 16981 ] |
| Fix Version/s | Sprint 4.1.1 [ 10737 ] |
| Reporter | FIWARE-TECH-HELP [ fiware-tech-help ] | FW External User [ fw.external.urser ] |
| Summary | [Fiware-tech-help] Fwd: Question re FIWARE COSMOS | FIWARE.Request.Lab.Data.BigData-Analysis.QuestionFIWARE COSMOS |
| Summary | FIWARE.Request.Lab.Data.BigData-Analysis.QuestionFIWARE COSMOS | FIWARE.Request.Lab.Data.BigData-Analysis.Parquet |
| Summary | FIWARE.Request.Lab.Data.BigData-Analysis.Parquet | FIWARE.Request.Tech.Data.BigData-Analysis.Parquet |
| Workflow | FW eRequest Workflow [ 16981 ] | FW extRequest Workflow-II [ 40645 ] |
| Status | Done [ 10000 ] | Closed [ 6 ] |
| Issue Type | eRequest [ 10101 ] | extRequest [ 10104 ] |
| Sender Email | watson.dave@gmail.com |
| HD-Enabler | Cosmos [ 10872 ] | |
| HD-Chapter | Data [ 10838 ] |
| Transition | Time In Source Status | Execution Times | Last Executer | Last Execution Date | |||||
|---|---|---|---|---|---|---|---|---|---|
|
4d 4h 16m | 1 | Francisco Romero | 09/Oct/14 3:46 PM | |||||
|
13d 6h 43m | 1 | Manuel Escriche | 22/Oct/14 10:30 PM | |||||
|
1m 38s | 2 | Manuel Escriche | 22/Oct/14 10:31 PM | |||||
|
446d 11h 36m | 1 | Manuel Escriche | 12/Jan/16 9:08 AM |
Hi Dave,
Sorry for the delay in answering you, I’ve been participating in several FIWARE dissemination events.
Regarding Parquet and Drill, if you have entered the global instance of Cosmos you’ll probably have seen the Hadoop version we are currently using is pretty old. In addition, it is quite limited in resources. It was initially addressed to hackathonists and newbies trying to get an easy approach to the Hadoop ecosystem, but not for more advanced usage. We are working on deploying a, let’s say, more “professional” Big Data platform, and I hope it is ready before the end of the year. In the meantime, I’ll try to stress the current deployment and I’ll study if any of those technologies can be exposed to the FILAB users, but as you are pointing, Drill requires recent Hadoop APIs.
Regards,
Francisco