Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-12848

FIWARE.Request.Lab.Hive - JSonSerde library - Cygnus.

    Details

    • Type: extRequest
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Fix Version/s: 2021
    • Component/s: FIWARE-TECH-HELP
    • Labels:
      None

      Description

      Hi ,
      I have configured the Cygnus instance with the following configuration for Hive:

      cygnus-ngsi.sinks.hdfs-sink.hive.host = cosmos.lab.fiware.org
      cygnus-ngsi.sinks.hdfs-sink.hive.server_version = 2
      cygnus-ngsi.sinks.hdfs-sink.file_format = json-column
      cygnus-ngsi.sinks.hdfs-sink.csv_separator = ,
      cygnus-ngsi.sinks.hdfs-sink.hdfs_host = storage.cosmos.lab.fiware.org
      cygnus-ngsi.sinks.hdfs-sink.hdfs_port = 14000
      cygnus-ngsi.sinks.hdfs-sink.hdfs_username = <username>
      cygnus-ngsi.sinks.hdfs-sink.hdfs_password = <password>
      cygnus-ngsi.sinks.hdfs-sink.oauth2_token = <token>
      cygnus-ngsi.sinks.hdfs-sink.hive = true

      After table creation, I have received the below error message:

      Error message:
      SQL Error [1] [08S01]: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.openx.data.jsonserde.JsonSerDe
      SQL Error [1] [08S01]: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.openx.data.jsonserde.JsonSerDe
      Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.openx.data.jsonserde.JsonSerDe
      Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.openx.data.jsonserde.JsonSerDe

      Ddl hive table:
      create external table if not exists default.mellusom_defaultx002fx0053x0043x0053x002d004_x0053martx0043ityx0053ensor_column
      (
      recvx0054ime string,
      fiwarex0053ervicex0050ath string,
      entityx0049d string,
      entityx0054ype string,
      brightness string,
      brightness_md array<struct<name:string,type:string,value:string>>,
      city string,city_md array<struct<name:string,type:string,value:string>>,
      continent string,continent_md array<struct<name:string,type:string,value:string>>,
      country string,country_md array<struct<name:string,type:string,value:string>>,
      location string,location_md array<struct<name:string,type:string,value:string>>,
      owner string,owner_md array<struct<name:string,type:string,value:string>>,
      region string,region_md array<struct<name:string,type:string,value:string>>,
      temperature_c string,temperature_c_md array<struct<name:string,type:string,value:string>>,
      zip string,zip_md array<struct<name:string,type:string,value:string>>)
      row format serde 'org.openx.data.jsonserde.JsonSerDe' with serdeproperties ("dots.in.keys" = "true")
      location '/user/mellusom/default/SCS-004_SmartCitySensor'

      I solved the issue regarding the error message using this link:

      https://community.cloudera.com/t5/Batch-SQL-Apache-Hive/Where-to-put-json-serde-1-3-7-jar-with-dependencies-jar-file/td-p/48073<http://cp.mcafee.com/d/avndygsd3gOrhvKYY-U-NsTsSyyPtBBBVBMSyyOyevKUYyrhhp76zBWXVEVd79J55VNdNZcQszzrItxrWmHOSHr7PPZ1lGNcWuvaDiCrJm9DjPVkWkPuE9LK-qen-LOrz1EVd7bPTnKnjpujVxUsUOUVR4mlHFGTppVkffGhBrwqrhdI6XYCyOzsQsIL8I6XCOsVHkiP5RmUiCmWNZmQfzLbVd_1DpS9BNw1MGn45JIjYK5V6vM1ivNYLeFtpKpshFm1uDNgYKynpbzyUzIiVykOpt3RLwxYbi-xshS9tSh_Vd0KJz3RNP9IhHjV4orqluJIo_fUR1jIz6N_w09JAsYYyCr1vF6y0bOc-q82VEw0woH0o-a7Cy0-h2eNb6y0Kq818IhwTgSOedBjl2gIAwSd_>

      and after that I have tried to deploy on my local Hive istance the jar library recovered by http://www.congiu.net/hive-json-serde/<http://cp.mcafee.com/d/FZsS93gwrhvKYY-U-NsTsSyyPtBBBVBMSyyOyevKUYyrhhp76zBWXVEVd79J55VNdNZcQszzrItxrWmHOSHr7PPZ1lGNcWuvaDiCrJm9DjPVkWkPuE9LK-qen-LOrz1EVd7bPTnKnjpujVxUsUOUVR4mlHFGTppVkffGhBrwqrjdI6XYCyOzsQsIL8I6XCM0lGM8OrU6UjApYK8QH0LjUEvxIhHjV4orqluJIo_fUR1jIz6N_w09JAsYYyCr1vF6y0bOc-q82VEw0woH0o-a7Cy0-h2eNb6y0Kq818IhwTgSOedFPnyNnzG> (version 1.3.8)
      and now It's correctly working in local.

      Best regards.

      Francesco Paparella
      Ricerca e Sviluppo

      Corvallis S.p.A. a Socio Unico
      Attività di direzione e coordinamento Corvallis Holding SpA
      Gruppo Corvallis
      Viale Aldo Moro, 36
      73100 Lecce
      Italia

      Telefono +39 0832 069901
      Telefax +39 0832 069911

      francesco.paparella@corvallis.it<francesco.paparella@corvallis.it>

      www.corvallis.it<http://www.corvallis.it/>

      Le informazioni contenute in questo messaggio di posta elettronica e negli eventuali allegati sono riservate e confidenziali e sono indirizzate esclusivamente al destinatario. Si prega di non fare copia, inoltrare a terzi o conservare tale messaggio se non si è il legittimo destinatario dello stesso. Qualora questo messaggio sia stato ricevuto per errore, si prega di rinviarlo al mittente e di cancellarlo permanentemente dal proprio computer.

      The information contained in this message and in any attachment is intended exclusively for the recipient. If you are not the intended recipient you are hereby notified not to copy, save, disclose, or distribute it to any third party. If you erroneously received this message you are kindly requested to return it to the sender and eliminate it permanently from your computer.

      __________________________________________________________________________________________

      You can get more information about our cookies and privacy policies on the following links:

      Fiware-lab-help mailing list
      Fiware-lab-help@lists.fiware.org
      https://lists.fiware.org/listinfo/fiware-lab-help

      [Created via e-mail received from: Paparella Francesco <Francesco.Paparella@corvallis.it>]

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open In Progress In Progress
        85d 13h 40m 1 Andres Muñoz 18/Jan/18 5:05 PM
        In Progress In Progress Impeded Impeded
        6d 17h 6m 1 Andres Muñoz 25/Jan/18 10:12 AM
        Impeded Impeded In Progress In Progress
        7d 1h 49m 1 Andres Muñoz 01/Feb/18 12:01 PM
        In Progress In Progress Answered Answered
        2s 1 Andres Muñoz 01/Feb/18 12:01 PM
        Answered Answered Closed Closed
        1s 1 Andres Muñoz 01/Feb/18 12:01 PM

          People

          • Assignee:
            andres.munoza Andres Muñoz
            Reporter:
            fw.ext.user FW External User
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: