Details
-
Type:
extRequest
-
Status: Closed
-
Priority:
Major
-
Resolution: Done
-
Fix Version/s: 2021
-
Component/s: FIWARE-TECH-HELP
-
Labels:None
-
HD-Chapter:Data
-
HD-Enabler:Cygnus
Description
Hi ,
I have configured the Cygnus instance with the following configuration for Hive:
cygnus-ngsi.sinks.hdfs-sink.hive.host = cosmos.lab.fiware.org
cygnus-ngsi.sinks.hdfs-sink.hive.server_version = 2
cygnus-ngsi.sinks.hdfs-sink.file_format = json-column
cygnus-ngsi.sinks.hdfs-sink.csv_separator = ,
cygnus-ngsi.sinks.hdfs-sink.hdfs_host = storage.cosmos.lab.fiware.org
cygnus-ngsi.sinks.hdfs-sink.hdfs_port = 14000
cygnus-ngsi.sinks.hdfs-sink.hdfs_username = <username>
cygnus-ngsi.sinks.hdfs-sink.hdfs_password = <password>
cygnus-ngsi.sinks.hdfs-sink.oauth2_token = <token>
cygnus-ngsi.sinks.hdfs-sink.hive = true
After table creation, I have received the below error message:
Error message:
SQL Error [1] [08S01]: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.openx.data.jsonserde.JsonSerDe
SQL Error [1] [08S01]: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.openx.data.jsonserde.JsonSerDe
Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.openx.data.jsonserde.JsonSerDe
Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.openx.data.jsonserde.JsonSerDe
Ddl hive table:
create external table if not exists default.mellusom_defaultx002fx0053x0043x0053x002d004_x0053martx0043ityx0053ensor_column
(
recvx0054ime string,
fiwarex0053ervicex0050ath string,
entityx0049d string,
entityx0054ype string,
brightness string,
brightness_md array<struct<name:string,type:string,value:string>>,
city string,city_md array<struct<name:string,type:string,value:string>>,
continent string,continent_md array<struct<name:string,type:string,value:string>>,
country string,country_md array<struct<name:string,type:string,value:string>>,
location string,location_md array<struct<name:string,type:string,value:string>>,
owner string,owner_md array<struct<name:string,type:string,value:string>>,
region string,region_md array<struct<name:string,type:string,value:string>>,
temperature_c string,temperature_c_md array<struct<name:string,type:string,value:string>>,
zip string,zip_md array<struct<name:string,type:string,value:string>>)
row format serde 'org.openx.data.jsonserde.JsonSerDe' with serdeproperties ("dots.in.keys" = "true")
location '/user/mellusom/default/SCS-004_SmartCitySensor'
I solved the issue regarding the error message using this link:
https://community.cloudera.com/t5/Batch-SQL-Apache-Hive/Where-to-put-json-serde-1-3-7-jar-with-dependencies-jar-file/td-p/48073<http://cp.mcafee.com/d/avndygsd3gOrhvKYY-U-NsTsSyyPtBBBVBMSyyOyevKUYyrhhp76zBWXVEVd79J55VNdNZcQszzrItxrWmHOSHr7PPZ1lGNcWuvaDiCrJm9DjPVkWkPuE9LK-qen-LOrz1EVd7bPTnKnjpujVxUsUOUVR4mlHFGTppVkffGhBrwqrhdI6XYCyOzsQsIL8I6XCOsVHkiP5RmUiCmWNZmQfzLbVd_1DpS9BNw1MGn45JIjYK5V6vM1ivNYLeFtpKpshFm1uDNgYKynpbzyUzIiVykOpt3RLwxYbi-xshS9tSh_Vd0KJz3RNP9IhHjV4orqluJIo_fUR1jIz6N_w09JAsYYyCr1vF6y0bOc-q82VEw0woH0o-a7Cy0-h2eNb6y0Kq818IhwTgSOedBjl2gIAwSd_>
and after that I have tried to deploy on my local Hive istance the jar library recovered by http://www.congiu.net/hive-json-serde/<http://cp.mcafee.com/d/FZsS93gwrhvKYY-U-NsTsSyyPtBBBVBMSyyOyevKUYyrhhp76zBWXVEVd79J55VNdNZcQszzrItxrWmHOSHr7PPZ1lGNcWuvaDiCrJm9DjPVkWkPuE9LK-qen-LOrz1EVd7bPTnKnjpujVxUsUOUVR4mlHFGTppVkffGhBrwqrjdI6XYCyOzsQsIL8I6XCM0lGM8OrU6UjApYK8QH0LjUEvxIhHjV4orqluJIo_fUR1jIz6N_w09JAsYYyCr1vF6y0bOc-q82VEw0woH0o-a7Cy0-h2eNb6y0Kq818IhwTgSOedFPnyNnzG> (version 1.3.8)
and now It's correctly working in local.
Best regards.
Francesco Paparella
Ricerca e Sviluppo
Corvallis S.p.A. a Socio Unico
Attività di direzione e coordinamento Corvallis Holding SpA
Gruppo Corvallis
Viale Aldo Moro, 36
73100 Lecce
Italia
Telefono +39 0832 069901
Telefax +39 0832 069911
francesco.paparella@corvallis.it<francesco.paparella@corvallis.it
>
www.corvallis.it<http://www.corvallis.it/>
Le informazioni contenute in questo messaggio di posta elettronica e negli eventuali allegati sono riservate e confidenziali e sono indirizzate esclusivamente al destinatario. Si prega di non fare copia, inoltrare a terzi o conservare tale messaggio se non si è il legittimo destinatario dello stesso. Qualora questo messaggio sia stato ricevuto per errore, si prega di rinviarlo al mittente e di cancellarlo permanentemente dal proprio computer.
The information contained in this message and in any attachment is intended exclusively for the recipient. If you are not the intended recipient you are hereby notified not to copy, save, disclose, or distribute it to any third party. If you erroneously received this message you are kindly requested to return it to the sender and eliminate it permanently from your computer.
__________________________________________________________________________________________
You can get more information about our cookies and privacy policies on the following links:
- http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE_Privacy_Policy
- http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/Cookies_Policy_FIWARE
Fiware-lab-help mailing list
Fiware-lab-help@lists.fiware.org
https://lists.fiware.org/listinfo/fiware-lab-help
[Created via e-mail received from: Paparella Francesco <Francesco.Paparella@corvallis.it>]
Activity
Comment by jira-help-desk@jira.fiware.org :
Spam detection software, running on the system "fiware-forge-01", has
identified this incoming email as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: [ https://jira.fiware.org/browse/HELP-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
] FW External User updated HELP-12848: Attachment: [FIWARE-JIRA] (HELP-12848)
[Fiware-general-help] [SPAM] [FIWARE-JIRA] (HELP-12870) [SPAM] Website Job
Application [...]
Content analysis details: (7.7 points, 4.5 required)
pts rule name description
---- ---------------------- --------------------------------------------------
2.5 URIBL_DBL_SPAM Contains a spam URL listed in the DBL blocklist
[URIs: kzarea.com]
1.7 URIBL_BLACK Contains an URL listed in the URIBL blacklist
[URIs: kzarea.com]
0.0 FSL_HELO_NON_FQDN_1 FSL_HELO_NON_FQDN_1
3.8 HELO_LOCALHOST HELO_LOCALHOST
1.6 DATE_IN_PAST_03_06 Date: is 3 to 6 hours before Received: date
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
Comment by jira-help-desk@jira.fiware.org :
Spam detection software, running on the system "fiware-forge-01", has
identified this incoming email as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: [ https://jira.fiware.org/browse/HELP-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
] FW External User updated HELP-12848: Attachment: [FIWARE-JIRA] (HELP-12848)
[Fiware-lab-help] Hive - JSonSerde library - Cygnus [...]
Content analysis details: (7.7 points, 4.5 required)
pts rule name description
---- ---------------------- --------------------------------------------------
2.5 URIBL_DBL_SPAM Contains a spam URL listed in the DBL blocklist
[URIs: kzarea.com]
1.7 URIBL_BLACK Contains an URL listed in the URIBL blacklist
[URIs: kzarea.com]
0.0 FSL_HELO_NON_FQDN_1 FSL_HELO_NON_FQDN_1
3.8 HELO_LOCALHOST HELO_LOCALHOST
1.6 DATE_IN_PAST_03_06 Date: is 3 to 6 hours before Received: date
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
Comment by jira-help-desk@jira.fiware.org :
Spam detection software, running on the system "fiware-forge-01", has
identified this incoming email as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: [ https://jira.fiware.org/browse/HELP-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
] FW External User updated HELP-12848: Attachment: [FIWARE-JIRA] (HELP-12848)
[Fiware-lab-help] Hive - JSonSerde library - Cygnus [...]
Content analysis details: (7.7 points, 4.5 required)
pts rule name description
---- ---------------------- --------------------------------------------------
2.5 URIBL_DBL_SPAM Contains a spam URL listed in the DBL blocklist
[URIs: kzarea.com]
1.7 URIBL_BLACK Contains an URL listed in the URIBL blacklist
[URIs: kzarea.com]
0.0 FSL_HELO_NON_FQDN_1 FSL_HELO_NON_FQDN_1
3.8 HELO_LOCALHOST HELO_LOCALHOST
1.6 DATE_IN_PAST_03_06 Date: is 3 to 6 hours before Received: date
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
Hi Francesco,
I notice that you solve your problem in a local instance, but I have a question, Are you want to store your information in the cloud Hive service of FIWARE Cosmos instance?, Also, I could notice that you are using in your agent.conf the FQDN of the old cosmos platform, the new platform is located in storage.dit.upm.es.
Best Regards,
This issue is related to the use of the old Fiware Cosmos platform, Thus, using the new cloud platform this issue is solved
Comment by jira-help-desk@jira.fiware.org :
Spam detection software, running on the system "fiware-forge-01", has
identified this incoming email as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: [ https://jira.fiware.org/browse/HELP-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
] FW External User updated
HELP-12848: > [Fiware-general-help] [SPAM] [FIWARE-JIRA](HELP-12870) [SPAM] Website Job Application > > > Key:
HELP-12848> URL:https://jira.fiware.org/browse/HELP-12848 > Project: Help-Desk > Issue Type:
extRequest > Reporter: FW External User > Attachments: [FIWARE-JIRA] (HELP-12870)
[Fiware-general-help] [SPAM] Website Job Application > > > Spam detection
software, running on the system "fiware-forge-01", has > identified this
incoming email as possible spam. The original message > has been attached
to this so you can view it (if it isn't spam) or label > similar future email.
If you have any questions, see > the administrator of that system for details.
> Content preview: [ https://jira.fiware.org/browse/HELP-12870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ] FW External User updated HELP-12870: Attachment: [FIWARE-JIRA] (HELP-12870)
> [Fiware-general-help] [SPAM] Website Job Application [...] > Content analysis
details: (7.7 points, 4.5 required) > pts rule name description > ---- >
2.5 URIBL_DBL_SPAM Contains a spam URL listed in the DBL blocklist > [URIs:
kzarea.com] > 1.7 URIBL_BLACK Contains an URL listed in the URIBL blacklist
> [URIs: kzarea.com] > 0.0 FSL_HELO_NON_FQDN_1 FSL_HELO_NON_FQDN_1 > 3.8
HELO_LOCALHOST HELO_LOCALHOST > 1.6 DATE_IN_PAST_03_06 Date: is 3 to 6 hours
before Received: date > -1.9 BAYES_00 BODY: Bayes spam probability is 0 to
1% > [score: 0.0000] > > You can get more information about our cookies and
privacy policies on the following links: > - http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE_Privacy_Policy
> - http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/Cookies_Policy_FIWARE
> Fiware-general-help mailing list > Fiware-general-help@lists.fiware.org
> [...]
Content analysis details: (6.9 points, 4.5 required)
pts rule name description
---- ---------------------- --------------------------------------------------
2.5 URIBL_DBL_SPAM Contains a spam URL listed in the DBL blocklist
[URIs: kzarea.com]
1.7 URIBL_BLACK Contains an URL listed in the URIBL blacklist
[URIs: kzarea.com]
0.0 FSL_HELO_NON_FQDN_1 FSL_HELO_NON_FQDN_1
3.8 HELO_LOCALHOST HELO_LOCALHOST
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
0.8 RDNS_NONE Delivered to internal network by a host with no rDNS