[HELP-3894] FIWARE.Request.Tech.Data.BigData-Analysis.HiveServer2Error - JIRA

Details

Type: extRequest
Status: Closed
Priority: Major
Resolution: Done
Fix Version/s: 2021
Component/s: FIWARE-TECH-HELP
Labels:
None

Sender Email:
dario.pellegrino@eng.it
HD-Chapter:
Data
HD-Enabler:
Cosmos

Description

Dear all,
a few days ago we have received your mail about the Hive Server upgrade (HiveServer2 instead of Shark) and we have modified our Java code as you recommended.
In particular we have load the new driver "org.apache.hive.jdbc.HiveDriver" , we have modified the JDBC connection "return DriverManager.getConnection("jdbc:hive2://" + hiveServer + ":" + hivePort + "/default", hdfsUser, hdfsPwd);" and we have changed the file POM.xml (dependencies Hive 0.13.0).
Unfortunately, after changes our application doesn't work.
You can find our error message below:

25-ago-2015 15.34.01 org.apache.catalina.core.StandardWrapperValve invoke
GRAVE: Servlet.service() for servlet [eu.finesce.emarketplace.RestHiveInputApplication] in context with path [/rest2cosmos] threw exception [java.lang.IllegalMonitorStateException] with root cause
java.lang.IllegalMonitorStateException
at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(Unknown Source)
at java.util.concurrent.locks.ReentrantLock.unlock(Unknown Source)
at org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:175)
at org.apache.hive.jdbc.HiveQueryResultSet.close(HiveQueryResultSet.java:293)
at eu.finesce.emarketplace.client.HiveClient.getmeterDetails(HiveClient.java:1386)
at eu.finesce.emarketplace.RestHive2Cosmos.getMeterDetails(RestHive2Cosmos.java:299)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:402)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:349)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:106)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:259)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:318)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:236)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1010)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:382)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:345)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:220)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Waiting for your feedback, thank you in advance.
Best regards,
Dario Pellegrino

Dario Pellegrino
Direzione Ricerca e Innovazione - R&D Lab
dario.pellegrino@eng.it<dario.pellegrino@eng.it>

Engineering Ingegneria Informatica spa
Viale Regione Siciliana, 7275 - 90146 Palermo
Tel. +39-091.7511847
Mob. +39-346.5325257
www.eng.it<http://www.eng.it/>

_______________________________________________
Fiware-lab-help mailing list
Fiware-lab-help@lists.fi-ware.org
https://lists.fi-ware.org/listinfo/fiware-lab-help

[Created via e-mail received from: Pellegrino Dario <dario.pellegrino@eng.it>]

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Aristi Galani (Inactive) added a comment - 25/Aug/15 6:30 PM

Dear Dario,

We forwarded your request to second level support.

Kind regards
IWAVE team, on behalf of helpdesk team

_______________________________________________
Fiware-lab-help mailing list
Fiware-lab-help@lists.fi-ware.org
https://lists.fi-ware.org/listinfo/fiware-lab-help

Show

Aristi Galani (Inactive) added a comment - 25/Aug/15 6:30 PM Dear Dario, We forwarded your request to second level support. Kind regards IWAVE team, on behalf of helpdesk team _______________________________________________ Fiware-lab-help mailing list Fiware-lab-help@lists.fi-ware.org https://lists.fi-ware.org/listinfo/fiware-lab-help

9 older comments

Hide

Permalink

FW External User added a comment - 31/Aug/15 5:58 PM

Hi Francisco,
the Hive Server does not seem to be working for me. Could you check it?
Regards,
Dario
Dario Pellegrino
Direzione Ricerca e Innovazione - R&D Lab
dario.pellegrino@eng.it<dario.pellegrino@eng.it>

Engineering Ingegneria Informatica spa
Viale Regione Siciliana, 7275 - 90146 Palermo
Tel. +39-091.7511847
Mob. +39-346.5325257
www.eng.it<http://www.eng.it/>

Da: FRANCISCO ROMERO BUENO francisco.romerobueno@telefonica.com
Inviato: lunedì 31 agosto 2015 14:45
A: Pasquale Andriani
Cc: Pellegrino Dario; Leandro Lombardo; Massimiliano Nigrelli; Luigi Briguglio; fiware-lab-help@lists.fi-ware.org; SERGIO GARCIA GOMEZ; SANTIAGO MARTINEZ GARCIA; Stefano De Panfilis; JUAN JOSE HIERRO SUREDA
Oggetto: Re: [Fiware-lab-help] [FINESCE-WP4] COSMOS : Error after upgrading to HiveServer2

Dear Pasquale, did you see my last email? I was saying I can setup a Spark/Shark deployment, exclusively for you, next monday 7th (despite I will be on holydays). Currently I am out of Spain with limited access to the Internet. Regards, Francisco

Show

FW External User added a comment - 31/Aug/15 5:58 PM Hi Francisco, the Hive Server does not seem to be working for me. Could you check it? Regards, Dario Dario Pellegrino Direzione Ricerca e Innovazione - R&D Lab dario.pellegrino@eng.it< dario.pellegrino@eng.it > Engineering Ingegneria Informatica spa Viale Regione Siciliana, 7275 - 90146 Palermo Tel. +39-091.7511847 Mob. +39-346.5325257 www.eng.it< http://www.eng.it/ > Da: FRANCISCO ROMERO BUENO francisco.romerobueno@telefonica.com Inviato: lunedì 31 agosto 2015 14:45 A: Pasquale Andriani Cc: Pellegrino Dario; Leandro Lombardo; Massimiliano Nigrelli; Luigi Briguglio; fiware-lab-help@lists.fi-ware.org; SERGIO GARCIA GOMEZ; SANTIAGO MARTINEZ GARCIA; Stefano De Panfilis; JUAN JOSE HIERRO SUREDA Oggetto: Re: [Fiware-lab-help] [FINESCE-WP4] COSMOS : Error after upgrading to HiveServer2 Dear Pasquale, did you see my last email? I was saying I can setup a Spark/Shark deployment, exclusively for you, next monday 7th (despite I will be on holydays). Currently I am out of Spain with limited access to the Internet. Regards, Francisco

Hide

Permalink

FW External User added a comment - 07/Sep/15 10:13 AM

Hi all,

As promised, I’ve setup a Spark/Shark deployment just for you. This has required the installation of a whole new Hive metastore since the existing one was recently tuned for Hive 0.13.0 (due to HiveServer2) and the Shark we had installed was compiled for Hive 0.9.0 (in any case, I’ve been looking for the more recent version of Shark and the latest one, before the project was discontinued, was designed to work with Hive 0.11.0, thus installing a new version would not solve the problem).

A couple of remarks:

Shark server now runs on port TCP/9999, don’t forget to change this in your client.
As any other user, your default Hive home within the Cosmos instance is /usr/local/apache-hive-0.13.0-bin . Nevertheless, your Hive metastore is related to Hive 0.9.0, thus my recommendation is you locally change both your PATH and your HIVE_HOME in order you always refer to Hive 0.9.0 and not Hive 0.13.0 when using, for instance, the CLI. Basically, add these lines to your /<your_user>/.bash_profile file:
export HIVE_HOME=/usr/local/hive-0.9.0-shark-0.8.0-bin/
export PATH=/usr/local/hive-0.9.0-shark-0.8.0-bin/bin/:/usr/local/shark-0.8/bin/:/usr/local/node-v0.12.4-linux-x64/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
Finally, the new metastore specifically created for you is empty: there is no tables nor databases (except for the default one and my personal db, named “frb"). Don’t panic! As its names denotes, is a storage for metadata, it does not contains real data; the real data continues stored in your HDFS space. So, you just need to recreate your tables by executing the command "create external table etc etc location ‘/path/to/data/in/hdfs/…’"; I guess you know the command because you already created the old tables by your own. If you don’t remember some detail regarding the tables, you can ask for it to Hive (0.13.0): “describe extended|formatted <table_name>"

I’ll keep an eye out on the email if you have any doubt.

Regards,
Francisco

De: Pasquale Andriani <pasquale.andriani@eng.it<pasquale.andriani@eng.it>>
Fecha: lunes, 31 de agosto de 2015, 14:55
Para: Francisco Romero Bueno <francisco.romerobueno@telefonica.com<francisco.romerobueno@telefonica.com>>
CC: Pellegrino Dario <dario.pellegrino@eng.it<dario.pellegrino@eng.it>>, Leandro Lombardo <Leandro.Lombardo@eng.it<Leandro.Lombardo@eng.it>>, Massimiliano Nigrelli <massimiliano.nigrelli@eng.it<massimiliano.nigrelli@eng.it>>, Luigi Briguglio <Luigi.Briguglio@eng.it<Luigi.Briguglio@eng.it>>, "fiware-lab-help@lists.fi-ware.org<fiware-lab-help@lists.fi-ware.org>" <fiware-lab-help@lists.fi-ware.org<fiware-lab-help@lists.fi-ware.org>>, SERGIO GARCIA GOMEZ <sergio.garciagomez@telefonica.com<sergio.garciagomez@telefonica.com>>, SANTIAGO MARTINEZ GARCIA <santiago.martinezgarcia@telefonica.com<santiago.martinezgarcia@telefonica.com>>, "stefano.depanfilis@eng.it<stefano.depanfilis@eng.it>" <stefano.depanfilis@eng.it<stefano.depanfilis@eng.it>>, JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com<juanjose.hierro@telefonica.com>>
Asunto: Re: [Fiware-lab-help] [FINESCE-WP4] COSMOS : Error after upgrading to HiveServer2

Thanks. Waiting for your news about Spark/Shark on Monday 7th.

Kind regards,
P.

Pasquale Andriani
Direzione Ricerca e Innovazione - Research & Development Lab
pasquale.andriani@eng.it<pasquale.andriani@eng.it>

Engineering Ingegneria Informatica spa
Via Riccardo Morandi, 32 - 00148 Roma
Tel. +39-06.87594138
Mob. +39 3924698746
Fax. +39-06.83074408
www.eng.it<http://www.eng.it>

On Mon, Aug 31, 2015 at 2:44 PM, FRANCISCO ROMERO BUENO <francisco.romerobueno@telefonica.com<francisco.romerobueno@telefonica.com>> wrote:
Dear Pasquale, did you see my last email? I was saying I can setup a Spark/Shark deployment, exclusively for you, next monday 7th (despite I will be on holydays). Currently I am out of Spain with limited access to the Internet. Regards, Francisco

Show

FW External User added a comment - 07/Sep/15 10:13 AM Hi all, As promised, I’ve setup a Spark/Shark deployment just for you. This has required the installation of a whole new Hive metastore since the existing one was recently tuned for Hive 0.13.0 (due to HiveServer2) and the Shark we had installed was compiled for Hive 0.9.0 (in any case, I’ve been looking for the more recent version of Shark and the latest one, before the project was discontinued, was designed to work with Hive 0.11.0, thus installing a new version would not solve the problem). A couple of remarks: Shark server now runs on port TCP/9999, don’t forget to change this in your client. As any other user, your default Hive home within the Cosmos instance is /usr/local/apache-hive-0.13.0-bin . Nevertheless, your Hive metastore is related to Hive 0.9.0, thus my recommendation is you locally change both your PATH and your HIVE_HOME in order you always refer to Hive 0.9.0 and not Hive 0.13.0 when using, for instance, the CLI. Basically, add these lines to your /<your_user>/.bash_profile file: export HIVE_HOME=/usr/local/hive-0.9.0-shark-0.8.0-bin/ export PATH=/usr/local/hive-0.9.0-shark-0.8.0-bin/bin/:/usr/local/shark-0.8/bin/:/usr/local/node-v0.12.4-linux-x64/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin Finally, the new metastore specifically created for you is empty: there is no tables nor databases (except for the default one and my personal db, named “frb"). Don’t panic! As its names denotes, is a storage for metadata, it does not contains real data; the real data continues stored in your HDFS space. So, you just need to recreate your tables by executing the command "create external table etc etc location ‘/path/to/data/in/hdfs/…’"; I guess you know the command because you already created the old tables by your own. If you don’t remember some detail regarding the tables, you can ask for it to Hive (0.13.0): “describe extended|formatted <table_name>" I’ll keep an eye out on the email if you have any doubt. Regards, Francisco De: Pasquale Andriani <pasquale.andriani@eng.it< pasquale.andriani@eng.it >> Fecha: lunes, 31 de agosto de 2015, 14:55 Para: Francisco Romero Bueno <francisco.romerobueno@telefonica.com< francisco.romerobueno@telefonica.com >> CC: Pellegrino Dario <dario.pellegrino@eng.it< dario.pellegrino@eng.it >>, Leandro Lombardo <Leandro.Lombardo@eng.it< Leandro.Lombardo@eng.it >>, Massimiliano Nigrelli <massimiliano.nigrelli@eng.it< massimiliano.nigrelli@eng.it >>, Luigi Briguglio <Luigi.Briguglio@eng.it< Luigi.Briguglio@eng.it >>, "fiware-lab-help@lists.fi-ware.org< fiware-lab-help@lists.fi-ware.org >" <fiware-lab-help@lists.fi-ware.org< fiware-lab-help@lists.fi-ware.org >>, SERGIO GARCIA GOMEZ <sergio.garciagomez@telefonica.com< sergio.garciagomez@telefonica.com >>, SANTIAGO MARTINEZ GARCIA <santiago.martinezgarcia@telefonica.com< santiago.martinezgarcia@telefonica.com >>, "stefano.depanfilis@eng.it< stefano.depanfilis@eng.it >" <stefano.depanfilis@eng.it< stefano.depanfilis@eng.it >>, JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com< juanjose.hierro@telefonica.com >> Asunto: Re: [Fiware-lab-help] [FINESCE-WP4] COSMOS : Error after upgrading to HiveServer2 Thanks. Waiting for your news about Spark/Shark on Monday 7th. Kind regards, P. Pasquale Andriani Direzione Ricerca e Innovazione - Research & Development Lab pasquale.andriani@eng.it< pasquale.andriani@eng.it > Engineering Ingegneria Informatica spa Via Riccardo Morandi, 32 - 00148 Roma Tel. +39-06.87594138 Mob. +39 3924698746 Fax. +39-06.83074408 www.eng.it< http://www.eng.it > On Mon, Aug 31, 2015 at 2:44 PM, FRANCISCO ROMERO BUENO <francisco.romerobueno@telefonica.com< francisco.romerobueno@telefonica.com >> wrote: Dear Pasquale, did you see my last email? I was saying I can setup a Spark/Shark deployment, exclusively for you, next monday 7th (despite I will be on holydays). Currently I am out of Spain with limited access to the Internet. Regards, Francisco

Hide

Permalink

FW External User added a comment - 08/Sep/15 11:24 AM

Hi Francisco,
thanks for your support.
I am going to implement your solution.
I will let you know.
Best regards,
Dario
Dario Pellegrino
Direzione Ricerca e Innovazione - R&D Lab
dario.pellegrino@eng.it<dario.pellegrino@eng.it>

Engineering Ingegneria Informatica spa
Viale Regione Siciliana, 7275 - 90146 Palermo
Tel. +39-091.7511847
Mob. +39-346.5325257
www.eng.it<http://www.eng.it/>

Da: FRANCISCO ROMERO BUENO francisco.romerobueno@telefonica.com
Inviato: domenica 6 settembre 2015 19:16
A: Pasquale Andriani
Cc: Pellegrino Dario; Leandro Lombardo; Massimiliano Nigrelli; Luigi Briguglio; fiware-lab-help@lists.fi-ware.org; SERGIO GARCIA GOMEZ; SANTIAGO MARTINEZ GARCIA; Stefano De Panfilis; JUAN JOSE HIERRO SUREDA
Oggetto: Re: [Fiware-lab-help] [FINESCE-WP4] COSMOS : Error after upgrading to HiveServer2

Hi all,

As promised, I've setup a Spark/Shark deployment just for you. This has required the installation of a whole new Hive metastore since the existing one was recently tuned for Hive 0.13.0 (due to HiveServer2) and the Shark we had installed was compiled for Hive 0.9.0 (in any case, I've been looking for the more recent version of Shark and the latest one, before the project was discontinued, was designed to work with Hive 0.11.0, thus installing a new version would not solve the problem).

A couple of remarks:

Shark server now runs on port TCP/9999, don't forget to change this in your client.
As any other user, your default Hive home within the Cosmos instance is /usr/local/apache-hive-0.13.0-bin . Nevertheless, your Hive metastore is related to Hive 0.9.0, thus my recommendation is you locally change both your PATH and your HIVE_HOME in order you always refer to Hive 0.9.0 and not Hive 0.13.0 when using, for instance, the CLI. Basically, add these lines to your /<your_user>/.bash_profile file:
export HIVE_HOME=/usr/local/hive-0.9.0-shark-0.8.0-bin/
export PATH=/usr/local/hive-0.9.0-shark-0.8.0-bin/bin/:/usr/local/shark-0.8/bin/:/usr/local/node-v0.12.4-linux-x64/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
Finally, the new metastore specifically created for you is empty: there is no tables nor databases (except for the default one and my personal db, named "frb"). Don't panic! As its names denotes, is a storage for metadata, it does not contains real data; the real data continues stored in your HDFS space. So, you just need to recreate your tables by executing the command "create external table etc etc location '/path/to/data/in/hdfs/...'"; I guess you know the command because you already created the old tables by your own. If you don't remember some detail regarding the tables, you can ask for it to Hive (0.13.0): "describe extended|formatted <table_name>"
I'll keep an eye out on the email if you have any doubt.

Regards,
Francisco

Thanks. Waiting for your news about Spark/Shark on Monday 7th.

Kind regards,
P.

Pasquale Andriani
Direzione Ricerca e Innovazione - Research & Development Lab
pasquale.andriani@eng.it<pasquale.andriani@eng.it>

Engineering Ingegneria Informatica spa
Via Riccardo Morandi, 32 - 00148 Roma
Tel. +39-06.87594138
Mob. +39 3924698746
Fax. +39-06.83074408
www.eng.it<http://www.eng.it>

Show

FW External User added a comment - 08/Sep/15 11:24 AM Hi Francisco, thanks for your support. I am going to implement your solution. I will let you know. Best regards, Dario Dario Pellegrino Direzione Ricerca e Innovazione - R&D Lab dario.pellegrino@eng.it< dario.pellegrino@eng.it > Engineering Ingegneria Informatica spa Viale Regione Siciliana, 7275 - 90146 Palermo Tel. +39-091.7511847 Mob. +39-346.5325257 www.eng.it< http://www.eng.it/ > Da: FRANCISCO ROMERO BUENO francisco.romerobueno@telefonica.com Inviato: domenica 6 settembre 2015 19:16 A: Pasquale Andriani Cc: Pellegrino Dario; Leandro Lombardo; Massimiliano Nigrelli; Luigi Briguglio; fiware-lab-help@lists.fi-ware.org; SERGIO GARCIA GOMEZ; SANTIAGO MARTINEZ GARCIA; Stefano De Panfilis; JUAN JOSE HIERRO SUREDA Oggetto: Re: [Fiware-lab-help] [FINESCE-WP4] COSMOS : Error after upgrading to HiveServer2 Hi all, As promised, I've setup a Spark/Shark deployment just for you. This has required the installation of a whole new Hive metastore since the existing one was recently tuned for Hive 0.13.0 (due to HiveServer2) and the Shark we had installed was compiled for Hive 0.9.0 (in any case, I've been looking for the more recent version of Shark and the latest one, before the project was discontinued, was designed to work with Hive 0.11.0, thus installing a new version would not solve the problem). A couple of remarks: Shark server now runs on port TCP/9999, don't forget to change this in your client. As any other user, your default Hive home within the Cosmos instance is /usr/local/apache-hive-0.13.0-bin . Nevertheless, your Hive metastore is related to Hive 0.9.0, thus my recommendation is you locally change both your PATH and your HIVE_HOME in order you always refer to Hive 0.9.0 and not Hive 0.13.0 when using, for instance, the CLI. Basically, add these lines to your /<your_user>/.bash_profile file: export HIVE_HOME=/usr/local/hive-0.9.0-shark-0.8.0-bin/ export PATH=/usr/local/hive-0.9.0-shark-0.8.0-bin/bin/:/usr/local/shark-0.8/bin/:/usr/local/node-v0.12.4-linux-x64/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin Finally, the new metastore specifically created for you is empty: there is no tables nor databases (except for the default one and my personal db, named "frb"). Don't panic! As its names denotes, is a storage for metadata, it does not contains real data; the real data continues stored in your HDFS space. So, you just need to recreate your tables by executing the command "create external table etc etc location '/path/to/data/in/hdfs/...'"; I guess you know the command because you already created the old tables by your own. If you don't remember some detail regarding the tables, you can ask for it to Hive (0.13.0): "describe extended|formatted <table_name>" I'll keep an eye out on the email if you have any doubt. Regards, Francisco De: Pasquale Andriani <pasquale.andriani@eng.it< pasquale.andriani@eng.it >> Fecha: lunes, 31 de agosto de 2015, 14:55 Para: Francisco Romero Bueno <francisco.romerobueno@telefonica.com< francisco.romerobueno@telefonica.com >> CC: Pellegrino Dario <dario.pellegrino@eng.it< dario.pellegrino@eng.it >>, Leandro Lombardo <Leandro.Lombardo@eng.it< Leandro.Lombardo@eng.it >>, Massimiliano Nigrelli <massimiliano.nigrelli@eng.it< massimiliano.nigrelli@eng.it >>, Luigi Briguglio <Luigi.Briguglio@eng.it< Luigi.Briguglio@eng.it >>, "fiware-lab-help@lists.fi-ware.org< fiware-lab-help@lists.fi-ware.org >" <fiware-lab-help@lists.fi-ware.org< fiware-lab-help@lists.fi-ware.org >>, SERGIO GARCIA GOMEZ <sergio.garciagomez@telefonica.com< sergio.garciagomez@telefonica.com >>, SANTIAGO MARTINEZ GARCIA <santiago.martinezgarcia@telefonica.com< santiago.martinezgarcia@telefonica.com >>, "stefano.depanfilis@eng.it< stefano.depanfilis@eng.it >" <stefano.depanfilis@eng.it< stefano.depanfilis@eng.it >>, JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com< juanjose.hierro@telefonica.com >> Asunto: Re: [Fiware-lab-help] [FINESCE-WP4] COSMOS : Error after upgrading to HiveServer2 Thanks. Waiting for your news about Spark/Shark on Monday 7th. Kind regards, P. Pasquale Andriani Direzione Ricerca e Innovazione - Research & Development Lab pasquale.andriani@eng.it< pasquale.andriani@eng.it > Engineering Ingegneria Informatica spa Via Riccardo Morandi, 32 - 00148 Roma Tel. +39-06.87594138 Mob. +39 3924698746 Fax. +39-06.83074408 www.eng.it< http://www.eng.it > On Mon, Aug 31, 2015 at 2:44 PM, FRANCISCO ROMERO BUENO <francisco.romerobueno@telefonica.com< francisco.romerobueno@telefonica.com >> wrote: Dear Pasquale, did you see my last email? I was saying I can setup a Spark/Shark deployment, exclusively for you, next monday 7th (despite I will be on holydays). Currently I am out of Spain with limited access to the Internet. Regards, Francisco

Hide

Permalink

FW External User added a comment - 11/Sep/15 9:42 AM

Hi Francisco,
I confirm you that Hive 0.9 with Spark/Shark works properly with reasonable query response time.
Please, could you have a "special" look at Shark/Spark system from Monday 14th to Wednesday 16th during the FINESCE final event?
Thank you again for your kind support.
Best regards,
Dario
Dario Pellegrino
Direzione Ricerca e Innovazione - R&D Lab
dario.pellegrino@eng.it<dario.pellegrino@eng.it>

Engineering Ingegneria Informatica spa
Viale Regione Siciliana, 7275 - 90146 Palermo
Tel. +39-091.7511847
Mob. +39-346.5325257
www.eng.it<http://www.eng.it/>

Da: Pellegrino Dario
Inviato: martedì 8 settembre 2015 11:07
A: 'FRANCISCO ROMERO BUENO'
Cc: Leandro Lombardo; Massimiliano Nigrelli; Luigi Briguglio; fiware-lab-help@lists.fi-ware.org; SERGIO GARCIA GOMEZ; SANTIAGO MARTINEZ GARCIA; Stefano De Panfilis; JUAN JOSE HIERRO SUREDA; Pasquale Andriani
Oggetto: R: [Fiware-lab-help] [FINESCE-WP4] COSMOS : Error after upgrading to HiveServer2

Engineering Ingegneria Informatica spa
Viale Regione Siciliana, 7275 - 90146 Palermo
Tel. +39-091.7511847
Mob. +39-346.5325257
www.eng.it<http://www.eng.it/>

Da: FRANCISCO ROMERO BUENO francisco.romerobueno@telefonica.com
Inviato: domenica 6 settembre 2015 19:16
A: Pasquale Andriani
Cc: Pellegrino Dario; Leandro Lombardo; Massimiliano Nigrelli; Luigi Briguglio; fiware-lab-help@lists.fi-ware.org<fiware-lab-help@lists.fi-ware.org>; SERGIO GARCIA GOMEZ; SANTIAGO MARTINEZ GARCIA; Stefano De Panfilis; JUAN JOSE HIERRO SUREDA
Oggetto: Re: [Fiware-lab-help] [FINESCE-WP4] COSMOS : Error after upgrading to HiveServer2

Hi all,

A couple of remarks:

Shark server now runs on port TCP/9999, don't forget to change this in your client.
As any other user, your default Hive home within the Cosmos instance is /usr/local/apache-hive-0.13.0-bin . Nevertheless, your Hive metastore is related to Hive 0.9.0, thus my recommendation is you locally change both your PATH and your HIVE_HOME in order you always refer to Hive 0.9.0 and not Hive 0.13.0 when using, for instance, the CLI. Basically, add these lines to your /<your_user>/.bash_profile file:
export HIVE_HOME=/usr/local/hive-0.9.0-shark-0.8.0-bin/
export PATH=/usr/local/hive-0.9.0-shark-0.8.0-bin/bin/:/usr/local/shark-0.8/bin/:/usr/local/node-v0.12.4-linux-x64/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
Finally, the new metastore specifically created for you is empty: there is no tables nor databases (except for the default one and my personal db, named "frb"). Don't panic! As its names denotes, is a storage for metadata, it does not contains real data; the real data continues stored in your HDFS space. So, you just need to recreate your tables by executing the command "create external table etc etc location '/path/to/data/in/hdfs/...'"; I guess you know the command because you already created the old tables by your own. If you don't remember some detail regarding the tables, you can ask for it to Hive (0.13.0): "describe extended|formatted <table_name>"
I'll keep an eye out on the email if you have any doubt.

Regards,
Francisco

Thanks. Waiting for your news about Spark/Shark on Monday 7th.

Kind regards,
P.

Pasquale Andriani
Direzione Ricerca e Innovazione - Research & Development Lab
pasquale.andriani@eng.it<pasquale.andriani@eng.it>

Engineering Ingegneria Informatica spa
Via Riccardo Morandi, 32 - 00148 Roma
Tel. +39-06.87594138
Mob. +39 3924698746
Fax. +39-06.83074408
www.eng.it<http://www.eng.it>

Show

FW External User added a comment - 11/Sep/15 9:42 AM Hi Francisco, I confirm you that Hive 0.9 with Spark/Shark works properly with reasonable query response time. Please, could you have a "special" look at Shark/Spark system from Monday 14th to Wednesday 16th during the FINESCE final event? Thank you again for your kind support. Best regards, Dario Dario Pellegrino Direzione Ricerca e Innovazione - R&D Lab dario.pellegrino@eng.it< dario.pellegrino@eng.it > Engineering Ingegneria Informatica spa Viale Regione Siciliana, 7275 - 90146 Palermo Tel. +39-091.7511847 Mob. +39-346.5325257 www.eng.it< http://www.eng.it/ > Da: Pellegrino Dario Inviato: martedì 8 settembre 2015 11:07 A: 'FRANCISCO ROMERO BUENO' Cc: Leandro Lombardo; Massimiliano Nigrelli; Luigi Briguglio; fiware-lab-help@lists.fi-ware.org; SERGIO GARCIA GOMEZ; SANTIAGO MARTINEZ GARCIA; Stefano De Panfilis; JUAN JOSE HIERRO SUREDA; Pasquale Andriani Oggetto: R: [Fiware-lab-help] [FINESCE-WP4] COSMOS : Error after upgrading to HiveServer2 Hi Francisco, thanks for your support. I am going to implement your solution. I will let you know. Best regards, Dario Dario Pellegrino Direzione Ricerca e Innovazione - R&D Lab dario.pellegrino@eng.it< dario.pellegrino@eng.it > Engineering Ingegneria Informatica spa Viale Regione Siciliana, 7275 - 90146 Palermo Tel. +39-091.7511847 Mob. +39-346.5325257 www.eng.it< http://www.eng.it/ > Da: FRANCISCO ROMERO BUENO francisco.romerobueno@telefonica.com Inviato: domenica 6 settembre 2015 19:16 A: Pasquale Andriani Cc: Pellegrino Dario; Leandro Lombardo; Massimiliano Nigrelli; Luigi Briguglio; fiware-lab-help@lists.fi-ware.org< fiware-lab-help@lists.fi-ware.org >; SERGIO GARCIA GOMEZ; SANTIAGO MARTINEZ GARCIA; Stefano De Panfilis; JUAN JOSE HIERRO SUREDA Oggetto: Re: [Fiware-lab-help] [FINESCE-WP4] COSMOS : Error after upgrading to HiveServer2 Hi all, As promised, I've setup a Spark/Shark deployment just for you. This has required the installation of a whole new Hive metastore since the existing one was recently tuned for Hive 0.13.0 (due to HiveServer2) and the Shark we had installed was compiled for Hive 0.9.0 (in any case, I've been looking for the more recent version of Shark and the latest one, before the project was discontinued, was designed to work with Hive 0.11.0, thus installing a new version would not solve the problem). A couple of remarks: Shark server now runs on port TCP/9999, don't forget to change this in your client. As any other user, your default Hive home within the Cosmos instance is /usr/local/apache-hive-0.13.0-bin . Nevertheless, your Hive metastore is related to Hive 0.9.0, thus my recommendation is you locally change both your PATH and your HIVE_HOME in order you always refer to Hive 0.9.0 and not Hive 0.13.0 when using, for instance, the CLI. Basically, add these lines to your /<your_user>/.bash_profile file: export HIVE_HOME=/usr/local/hive-0.9.0-shark-0.8.0-bin/ export PATH=/usr/local/hive-0.9.0-shark-0.8.0-bin/bin/:/usr/local/shark-0.8/bin/:/usr/local/node-v0.12.4-linux-x64/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin Finally, the new metastore specifically created for you is empty: there is no tables nor databases (except for the default one and my personal db, named "frb"). Don't panic! As its names denotes, is a storage for metadata, it does not contains real data; the real data continues stored in your HDFS space. So, you just need to recreate your tables by executing the command "create external table etc etc location '/path/to/data/in/hdfs/...'"; I guess you know the command because you already created the old tables by your own. If you don't remember some detail regarding the tables, you can ask for it to Hive (0.13.0): "describe extended|formatted <table_name>" I'll keep an eye out on the email if you have any doubt. Regards, Francisco De: Pasquale Andriani <pasquale.andriani@eng.it< pasquale.andriani@eng.it >> Fecha: lunes, 31 de agosto de 2015, 14:55 Para: Francisco Romero Bueno <francisco.romerobueno@telefonica.com< francisco.romerobueno@telefonica.com >> CC: Pellegrino Dario <dario.pellegrino@eng.it< dario.pellegrino@eng.it >>, Leandro Lombardo <Leandro.Lombardo@eng.it< Leandro.Lombardo@eng.it >>, Massimiliano Nigrelli <massimiliano.nigrelli@eng.it< massimiliano.nigrelli@eng.it >>, Luigi Briguglio <Luigi.Briguglio@eng.it< Luigi.Briguglio@eng.it >>, "fiware-lab-help@lists.fi-ware.org< fiware-lab-help@lists.fi-ware.org >" <fiware-lab-help@lists.fi-ware.org< fiware-lab-help@lists.fi-ware.org >>, SERGIO GARCIA GOMEZ <sergio.garciagomez@telefonica.com< sergio.garciagomez@telefonica.com >>, SANTIAGO MARTINEZ GARCIA <santiago.martinezgarcia@telefonica.com< santiago.martinezgarcia@telefonica.com >>, "stefano.depanfilis@eng.it< stefano.depanfilis@eng.it >" <stefano.depanfilis@eng.it< stefano.depanfilis@eng.it >>, JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com< juanjose.hierro@telefonica.com >> Asunto: Re: [Fiware-lab-help] [FINESCE-WP4] COSMOS : Error after upgrading to HiveServer2 Thanks. Waiting for your news about Spark/Shark on Monday 7th. Kind regards, P. Pasquale Andriani Direzione Ricerca e Innovazione - Research & Development Lab pasquale.andriani@eng.it< pasquale.andriani@eng.it > Engineering Ingegneria Informatica spa Via Riccardo Morandi, 32 - 00148 Roma Tel. +39-06.87594138 Mob. +39 3924698746 Fax. +39-06.83074408 www.eng.it< http://www.eng.it > On Mon, Aug 31, 2015 at 2:44 PM, FRANCISCO ROMERO BUENO <francisco.romerobueno@telefonica.com< francisco.romerobueno@telefonica.com >> wrote: Dear Pasquale, did you see my last email? I was saying I can setup a Spark/Shark deployment, exclusively for you, next monday 7th (despite I will be on holydays). Currently I am out of Spain with limited access to the Internet. Regards, Francisco

Hide

Permalink

Francisco Romero added a comment - 14/Sep/15 11:30 AM

Last emails exchanged:

Hi Dario,

Just came back from my holidays. Great to hear that. Yes, I’ll be specially checking for it all along these days.

Regards,
Francisco

De: Pellegrino Dario <dario.pellegrino@eng.it>
Fecha: jueves, 10 de septiembre de 2015, 11:37
Para: Francisco Romero Bueno <francisco.romerobueno@telefonica.com>
CC: Leandro Lombardo <Leandro.Lombardo@eng.it>, Massimiliano Nigrelli <massimiliano.nigrelli@eng.it>, Luigi Briguglio <Luigi.Briguglio@eng.it>, "fiware-lab-help@lists.fi-ware.org" <fiware-lab-help@lists.fi-ware.org>, SERGIO GARCIA GOMEZ <sergio.garciagomez@telefonica.com>, SANTIAGO MARTINEZ GARCIA <santiago.martinezgarcia@telefonica.com>, "stefano.depanfilis@eng.it" <stefano.depanfilis@eng.it>, JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com>, Pasquale Andriani <pasquale.andriani@eng.it>
Asunto: R: [Fiware-lab-help] [FINESCE-WP4] COSMOS : Error after upgrading to HiveServer2

Show

Francisco Romero added a comment - 14/Sep/15 11:30 AM Last emails exchanged: Hi Dario, Just came back from my holidays. Great to hear that. Yes, I’ll be specially checking for it all along these days. Regards, Francisco De: Pellegrino Dario <dario.pellegrino@eng.it> Fecha: jueves, 10 de septiembre de 2015, 11:37 Para: Francisco Romero Bueno <francisco.romerobueno@telefonica.com> CC: Leandro Lombardo <Leandro.Lombardo@eng.it>, Massimiliano Nigrelli <massimiliano.nigrelli@eng.it>, Luigi Briguglio <Luigi.Briguglio@eng.it>, "fiware-lab-help@lists.fi-ware.org" <fiware-lab-help@lists.fi-ware.org>, SERGIO GARCIA GOMEZ <sergio.garciagomez@telefonica.com>, SANTIAGO MARTINEZ GARCIA <santiago.martinezgarcia@telefonica.com>, "stefano.depanfilis@eng.it" <stefano.depanfilis@eng.it>, JUAN JOSE HIERRO SUREDA <juanjose.hierro@telefonica.com>, Pasquale Andriani <pasquale.andriani@eng.it> Asunto: R: [Fiware-lab-help] [FINESCE-WP4] COSMOS : Error after upgrading to HiveServer2 Hi Francisco, I confirm you that Hive 0.9 with Spark/Shark works properly with reasonable query response time. Please, could you have a "special" look at Shark/Spark system from Monday 14th to Wednesday 16th during the FINESCE final event? Thank you again for your kind support. Best regards, Dario

People

Assignee:

Francisco Romero

Reporter:

FW External User

Votes:

0 Vote for this issue

Watchers:

4 Start watching this issue

Dates

Created:

25/Aug/15 4:18 PM

Updated:

27/May/21 10:58 AM

Resolved:

27/Aug/15 4:06 PM

Agile

View on Board