Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-9198

[fiware-stackoverflow] Error at defining a specific field in a Hive Query

    Details

      Description

      Created question in FIWARE Q/A platform on 08-07-2015 at 14:07
      Please, ANSWER this question AT https://stackoverflow.com/questions/31293270/error-at-defining-a-specific-field-in-a-hive-query

      Question:
      Error at defining a specific field in a Hive Query

      Description:
      I have an Orion Context Broker connected with Cosmos by cygnus.

      It works ok, I mean I send new elements to Context Broker and cygnus send them to Cosmos and save them in files.

      The problem I have is when I try to do some searchs.

      I start hive, and I see that there are some tables created related with the files that cosmos have created, so I launch some querys.

      The simple one works fine:

      select * from Table_name;

      Hive doesn't launch any mapReduce jobs.

      but when I want to filter, join, count, or get only some fields. That is what happens:

      Total MapReduce jobs = 1
      Launching Job 1 out of 1
      Number of reduce tasks determined at compile time: 1
      In order to change the average load for a reducer (in bytes):
      set hive.exec.reducers.bytes.per.reducer=<number>
      In order to limit the maximum number of reducers:
      set hive.exec.reducers.max=<number>
      In order to set a constant number of reducers:
      set mapred.reduce.tasks=<number>
      Starting Job = JOB_NAME, Tracking URL = JOB_DETAILS_URL
      Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -kill JOB_NAME
      Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
      2015-07-08 14:35:12,723 Stage-1 map = 0%, reduce = 0%
      2015-07-08 14:35:38,943 Stage-1 map = 100%, reduce = 100%
      Ended Job = JOB_NAME with errors
      Error during job, obtaining debugging information...
      Examining task ID: TASK_NAME (and more) from job JOB_NAME

      Task with the most failures(4):


      Task ID:
      task_201409031055_6337_m_000000

      URL: TASK_DETAIL_URL


      FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
      MapReduce Jobs Launched:
      Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL

      I have found that the files that are created by Cygnus has difference between tha other files, because in the case of the cygnus, they have to be deserialize with a jar.

      So, I have the doubt if in those cases I have to apply any MapReduce method or if is already any general method to do this.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open In Progress In Progress
        2h 59m 1 Backlog Manager 24/May/17 6:04 PM
        In Progress In Progress Answered Answered
        2h 59m 1 Backlog Manager 24/May/17 9:04 PM
        Answered Answered Closed Closed
        1d 13h 16m 1 Fernando Lopez 26/May/17 10:20 AM
        fla Fernando Lopez made changes -
        Fix Version/s 2021 [ 12600 ]
        fla Fernando Lopez made changes -
        Resolution Done [ 10000 ]
        Status Answered [ 10104 ] Closed [ 6 ]
        fla Fernando Lopez made changes -
        HD-Enabler Cosmos [ 10872 ]
        Description
        Created question in FIWARE Q/A platform on 08-07-2015 at 14:07
        {color: red}Please, ANSWER this question AT{color} https://stackoverflow.com/questions/31293270/error-at-defining-a-specific-field-in-a-hive-query


        +Question:+
        Error at defining a specific field in a Hive Query

        +Description:+
        I have an Orion Context Broker connected with Cosmos by cygnus.

        It works ok, I mean I send new elements to Context Broker and cygnus send them to Cosmos and save them in files.

        The problem I have is when I try to do some searchs.

        I start hive, and I see that there are some tables created related with the files that cosmos have created, so I launch some querys.

        The simple one works fine:

        select * from Table_name;


        Hive doesn't launch any mapReduce jobs.

        but when I want to filter, join, count, or get only some fields. That is what happens:

        Total MapReduce jobs = 1
        Launching Job 1 out of 1
        Number of reduce tasks determined at compile time: 1
        In order to change the average load for a reducer (in bytes):
          set hive.exec.reducers.bytes.per.reducer=&lt;number&gt;
        In order to limit the maximum number of reducers:
          set hive.exec.reducers.max=&lt;number&gt;
        In order to set a constant number of reducers:
          set mapred.reduce.tasks=&lt;number&gt;
        Starting Job = JOB_NAME, Tracking URL = JOB_DETAILS_URL
        Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -kill JOB_NAME
        Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
        2015-07-08 14:35:12,723 Stage-1 map = 0%, reduce = 0%
        2015-07-08 14:35:38,943 Stage-1 map = 100%, reduce = 100%
        Ended Job = JOB_NAME with errors
        Error during job, obtaining debugging information...
        Examining task ID: TASK_NAME (and more) from job JOB_NAME

        Task with the most failures(4):
        -----
        Task ID:
          task_201409031055_6337_m_000000

        URL: TASK_DETAIL_URL
        -----

        FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
        MapReduce Jobs Launched:
        Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL


        I have found that the files that are created by Cygnus has difference between tha other files, because in the case of the cygnus, they have to be deserialize with a jar.

        So, I have the doubt if in those cases I have to apply any MapReduce method or if is already any general method to do this.
        Created question in FIWARE Q/A platform on 08-07-2015 at 14:07
        {color: red}Please, ANSWER this question AT{color} https://stackoverflow.com/questions/31293270/error-at-defining-a-specific-field-in-a-hive-query


        +Question:+
        Error at defining a specific field in a Hive Query

        +Description:+
        I have an Orion Context Broker connected with Cosmos by cygnus.

        It works ok, I mean I send new elements to Context Broker and cygnus send them to Cosmos and save them in files.

        The problem I have is when I try to do some searchs.

        I start hive, and I see that there are some tables created related with the files that cosmos have created, so I launch some querys.

        The simple one works fine:

        select * from Table_name;


        Hive doesn't launch any mapReduce jobs.

        but when I want to filter, join, count, or get only some fields. That is what happens:

        Total MapReduce jobs = 1
        Launching Job 1 out of 1
        Number of reduce tasks determined at compile time: 1
        In order to change the average load for a reducer (in bytes):
          set hive.exec.reducers.bytes.per.reducer=&lt;number&gt;
        In order to limit the maximum number of reducers:
          set hive.exec.reducers.max=&lt;number&gt;
        In order to set a constant number of reducers:
          set mapred.reduce.tasks=&lt;number&gt;
        Starting Job = JOB_NAME, Tracking URL = JOB_DETAILS_URL
        Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -kill JOB_NAME
        Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
        2015-07-08 14:35:12,723 Stage-1 map = 0%, reduce = 0%
        2015-07-08 14:35:38,943 Stage-1 map = 100%, reduce = 100%
        Ended Job = JOB_NAME with errors
        Error during job, obtaining debugging information...
        Examining task ID: TASK_NAME (and more) from job JOB_NAME

        Task with the most failures(4):
        -----
        Task ID:
          task_201409031055_6337_m_000000

        URL: TASK_DETAIL_URL
        -----

        FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
        MapReduce Jobs Launched:
        Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL


        I have found that the files that are created by Cygnus has difference between tha other files, because in the case of the cygnus, they have to be deserialize with a jar.

        So, I have the doubt if in those cases I have to apply any MapReduce method or if is already any general method to do this.
        HD-Chapter Data [ 10838 ]
        fla Fernando Lopez made changes -
        Assignee Francisco Romero [ frb ]
        Hide
        backlogmanager Backlog Manager added a comment -

        2017-05-24 21:05|UPDATED status: transition Answered| # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2017-05-24 21:05|UPDATED status: transition Answered| # answers= 1, accepted answer= True
        backlogmanager Backlog Manager made changes -
        Status In Progress [ 3 ] Answered [ 10104 ]
        Hide
        backlogmanager Backlog Manager added a comment -

        2017-05-24 18:05|UPDATED status: transition Answer| # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2017-05-24 18:05|UPDATED status: transition Answer| # answers= 1, accepted answer= True
        backlogmanager Backlog Manager made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        backlogmanager Backlog Manager made changes -
        Field Original Value New Value
        Component/s FIWARE-TECH-HELP [ 10278 ]
        Hide
        backlogmanager Backlog Manager added a comment -

        2017-05-24 15:05|CREATED monitor | # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2017-05-24 15:05|CREATED monitor | # answers= 1, accepted answer= True
        backlogmanager Backlog Manager created issue -

          People

          • Assignee:
            frb Francisco Romero
            Reporter:
            backlogmanager Backlog Manager
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: