Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-5779

FIWARE.Question.Tech.Data.BigData-Analysis.Cosmos Hive error entering and using map reduce

    Details

      Description

      Created question in FIWARE Q/A platform on 29-01-2016 at 13:01
      Please, ANSWER this question AT http://stackoverflow.com/questions/35084224/cosmos-hive-error-entering-and-using-map-reduce

      Question:
      Cosmos Hive error entering and using map reduce

      Description:
      I've a couple of problems executing Hive on cosmos fiware lab instance.

      First, after log into the machine, I enter in Hive command line and I get the following error (I saw other questions related to this, but I couldn't find a solution):

      $ hive
      log4j:ERROR Could not instantiate class [org.apache.hadoop.hive.shims.HiveEventCounter].
      java.lang.RuntimeException: Could not load shims in class org.apache.hadoop.log.metrics.EventCounter
      at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:123)
      at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:115)
      at org.apache.hadoop.hive.shims.ShimLoader.getEventCounter(ShimLoader.java:98)
      at org.apache.hadoop.hive.shims.HiveEventCounter.<init>(HiveEventCounter.java:34)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at java.lang.Class.newInstance0(Class.java:357)
      at java.lang.Class.newInstance(Class.java:310)
      at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:330)
      at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
      at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
      at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
      at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
      at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
      at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
      at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:354)
      at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jDefault(LogUtils.java:127)
      at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:77)
      at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:58)
      at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:641)
      at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.log.metrics.EventCounter
      at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:171)
      at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:120)
      ... 27 more
      log4j:ERROR Could not instantiate appender named "EventCounter".

      Logging initialized using configuration in jar:file:/usr/local/apache-hive-0.13.0-bin/lib/hive-common-0.13.0.jar!/hive-log4j.properties

      However, I'm able to run a query like SELECT * FROM table;

      On the other hand, if I try to run other query more specific like display only a column field, a map reduce job starts to run and it results in the following error:

      hive> SELECT table.column FROM table;
      Total jobs = 1
      Launching Job 1 out of 1
      Number of reduce tasks is set to 0 since there's no reduce operator
      Starting Job = job_201507101501_40071, Tracking URL = http://cosmosmaster-gi:50030/jobdetails.jsp?jobid=job_201507101501_40071
      Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -kill job_201507101501_40071
      Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
      2016-01-29 12:49:45,518 Stage-1 map = 0%, reduce = 0%
      2016-01-29 12:50:08,642 Stage-1 map = 100%, reduce = 100%
      Ended Job = job_201507101501_40071 with errors
      Error during job, obtaining debugging information...
      Job Tracking URL: http://cosmosmaster-gi:50030/jobdetails.jsp?jobid=job_201507101501_40071
      Examining task ID: task_201507101501_40071_m_000002 (and more) from job job_201507101501_40071

      Task with the most failures(4):


      Task ID:
      task_201507101501_40071_m_000000

      URL:
      http://cosmosmaster-gi:50030/taskdetails.jsp?jobid=job_201507101501_40071&amp;tipid=task_201507101501_40071_m_000000


      Diagnostic Messages for this Task:
      java.lang.RuntimeException: Error in configuring object
      at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
      at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
      at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:386)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:324)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
      at org.apache.hadoop.mapred.Child.main(Child.java:260)
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav

      FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
      MapReduce Jobs Launched:
      Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
      Total MapReduce CPU Time Spent: 0 msec

      Any help or suggestion is welcome.

      Thanks.

        Activity

        Hide
        backlogmanager Backlog Manager added a comment -

        2016-01-29 15:05|CREATED monitor | # answers= 1, accepted answer= False

        Show
        backlogmanager Backlog Manager added a comment - 2016-01-29 15:05|CREATED monitor | # answers= 1, accepted answer= False
        Hide
        backlogmanager Backlog Manager added a comment -

        2016-01-29 18:05|UPDATED status: transition Answer| # answers= 1, accepted answer= False

        Show
        backlogmanager Backlog Manager added a comment - 2016-01-29 18:05|UPDATED status: transition Answer| # answers= 1, accepted answer= False
        Hide
        backlogmanager Backlog Manager added a comment -

        2016-01-29 21:05|UPDATED status: transition Answered| # answers= 1, accepted answer= False

        Show
        backlogmanager Backlog Manager added a comment - 2016-01-29 21:05|UPDATED status: transition Answered| # answers= 1, accepted answer= False
        Hide
        backlogmanager Backlog Manager added a comment -

        2016-01-31 15:05|UPDATED status: transition Finish| # answers= 1, accepted answer= False

        Show
        backlogmanager Backlog Manager added a comment - 2016-01-31 15:05|UPDATED status: transition Finish| # answers= 1, accepted answer= False

          People

          • Assignee:
            frb Francisco Romero
            Reporter:
            backlogmanager Backlog Manager
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: