Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-4790

FIWARE.Question.Tech.Data.BigData-Analysis.Cygnus MySQL sink error (attr_persistence = column)

    Details

      Description

      Created question in FIWARE Q/A platform on 11-08-2015 at 11:08
      Please, ANSWER this question AT http://stackoverflow.com/questions/31937580/cygnus-mysql-sink-error-attr-persistence-column

      Question:
      Cygnus MySQL sink error (attr_persistence = column)

      Description:
      I'm using cygnus to store data in both MySQL Database and Cosmos.
      Storing the data in cosmos works ok, but when storing in MySQL, especifically with the attribute "cygnusagent.sinks.mysql-sink.attr_persistence = column" instead of "cygnusagent.sinks.mysql-sink.attr_persistence = row" the data is not stored and I'm getting some errors in flume log.

      As the table needs to be created previously when using the column attribute(more info : https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/design/OrionMySQLSink.md#important-notes-regarding-the-persistence-mode), I create the table:

      CREATE TABLE def_servpath_sensorreading4_sensorreading(systemid int,value float, sensorid int, nodeid int);

      DESCRIBE def_servpath_sensorreading4_sensorreading(systemid int,value float, sensorid int, nodeid int);
      mysql> DESCRIBE def_servpath_sensorreading4_sensorreading;
      ------------------------------------+

      Field Type Null Key Default Extra

      ------------------------------------+

      systemid int(11) YES   NULL  
      value float YES   NULL  
      sensorid int(11) YES   NULL  
      nodeid int(11) YES   NULL  

      ------------------------------------+
      4 rows in set (0.00 sec)

      This is the flume log:

      11 Aug 2015 11:40:31,977 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionMySQLSink.persist:240) - [mysql-sink] Persisting data at OrionMySQLSink. Database: def_serv, Table: def_servpath_sensorreading3_sensorreading, Timestamp: 2015-08-11T08:40:31.969, Data (attrs):

      {systemid=1, value=29.2, sensorid=2, nodeid=1}

      , (metadata):

      {sensorid_md=[], systemid_md=[], value_md=[], nodeid_md=[]}

      11 Aug 2015 11:40:31,978 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:182) - Bad context data (Unknown column 'systemid' in 'field list')
      11 Aug 2015 11:40:31,978 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:193) - Finishing transaction (1439278647-655-0000000005)
      11 Aug 2015 11:40:32,369 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persist:356) - [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (def_serv/def_servpath/sensorreading3_sensorreading/sensorreading3_sensorreading.txt), Data (

      {"recvTime":"2015-08-11T08:40:31.969Z","nodeid":"1", "nodeid_md":[],"sensorid":"2", "sensorid_md":[],"systemid":"1", "systemid_md":[],"value":"29.2", "value_md":[]}

      )
      11 Aug 2015 11:40:32,858 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:193) - Finishing transaction (1439278647-655-0000000005)

      Where im getting the following error:

      Bad context data (Unknown column 'systemid' in 'field list')

      Do I need to create any more columns in the table related with the metadata? Am I creating the table correctly?

      Thanks in advance.

        Activity

        Hide
        backlogmanager Backlog Manager added a comment -

        2015-10-14 15:05|CREATED monitor | # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2015-10-14 15:05|CREATED monitor | # answers= 1, accepted answer= True
        Hide
        backlogmanager Backlog Manager added a comment -

        2015-10-14 18:05|UPDATED status: transition Answer| # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2015-10-14 18:05|UPDATED status: transition Answer| # answers= 1, accepted answer= True
        Hide
        backlogmanager Backlog Manager added a comment -

        2015-10-14 21:05|UPDATED status: transition Answered| # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2015-10-14 21:05|UPDATED status: transition Answered| # answers= 1, accepted answer= True
        Hide
        backlogmanager Backlog Manager added a comment -

        2015-10-15 00:05|UPDATED status: transition Finish| # answers= 1, accepted answer= True

        Show
        backlogmanager Backlog Manager added a comment - 2015-10-15 00:05|UPDATED status: transition Finish| # answers= 1, accepted answer= True

          People

          • Assignee:
            frb Francisco Romero
            Reporter:
            backlogmanager Backlog Manager
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: