Uploaded image for project: 'Help-Coaches-Desk'
  1. Help-Coaches-Desk
  2. HELC-986

FIWARE.Request.Coach.FINODEX.FIWARE.Request.Lab.Data.BigData-Analysis.CygnusChannelFullError

        Details

        • Type: extRequest
        • Status: Closed
        • Priority: Major
        • Resolution: Done
        • Component/s: FINODEX
        • Labels:
          None

          Description

          Hello

          We are having troubles with Cygnus. When we have some devices sending
          information to Context Broker (and context broker subscripted to Cygnus,
          sending information to Cosmos and CKAN), the log show errors

          There are errors both ckan-channel and hdfs-channel

          Here a subset of log.

          02 Sep 2015 06:46:07,948 WARN [1701535537@qtp-1850408381-327]
          (org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost:203) -
          Error appending event to channel. Channel might be full. Consider
          increasing the channel capacity or make sure the sinks perform faster.
          org.apache.flume.ChannelException: Unable to put batch on required channel:
          org.apache.flume.channel.MemoryChannel

          {name: ckan-channel}

          at
          org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
          at
          org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:201)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
          at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
          at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
          at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
          at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
          at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
          at org.mortbay.jetty.Server.handle(Server.java:326)
          at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
          at
          org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
          at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
          at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
          at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
          at
          org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
          at
          org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
          Caused by: org.apache.flume.ChannelException: Space for commit to queue
          couldn't be acquired Sinks are likely not keeping up with sources, or the
          buffer size is too tight
          at org.apache.flume.channel.MemoryChannel

          The complete log is available here:
          https://dl.dropboxusercontent.com/u/2537618/cygnus_a.log

          –

          _______________________________________________
          Fiware-finodex-coaching mailing list
          Fiware-finodex-coaching@lists.fi-ware.org
          https://lists.fi-ware.org/listinfo/fiware-finodex-coaching

          [Created via e-mail received from: MAPT Talkykar <miguel@talkykar.com>]

            Issue Links

            relates to

            extRequest - Request from external stakeholder received in the help desk HELP-4512 FIWARE.Request.Tech.Data.BigData-Analysis.CygnusChannelFullError

            • Major - Major loss of function.
            • Closed

              Activity

              Descending order - Click to sort in ascending order
              Transition Time In Source Status Execution Times Last Executer Last Execution Date
              Open Open In Progress In Progress
              		12d 1h 31m 1 Francisco Romero 14/Sep/15 11:07 AM
              In Progress In Progress Answered Answered
              		17m 16s 1 Francisco Romero 14/Sep/15 11:25 AM
              Answered Answered Closed Closed
              		16d 4h 15m 1 Pasquale Vitale 30/Sep/15 3:40 PM
              backlogmanager Backlog Manager made changes -
              Summary FIWARE.Request.Lab.Data.BigData-Analysis.CygnusChannelFullError FIWARE.Request.Coach.FINODEX.FIWARE.Request.Lab.Data.BigData-Analysis.CygnusChannelFullError
              pasquale.vitale Pasquale Vitale made changes -
              Resolution Done [ 10000 ]
              Status Answered [ 10104 ] Closed [ 6 ]
              backlogmanager Backlog Manager made changes -
              Component/s _TECH_ [ 11400 ]
              backlogmanager Backlog Manager made changes -
              Link This issue relates to HELP-4512 [ HELP-4512 ]
              mev Manuel Escriche made changes -
              Component/s _TECH_ [ 11400 ]
              mev Manuel Escriche made changes -
              Assignee Francisco Romero [ frb ] Pasquale Vitale [ pasquale.vitale ]
              Hide
              Permalink
              frb Francisco Romero added a comment -

              The issue has been emailed:

              • Time sent: 16/Sep/15 11:42 AM
              • To: miguel@talkykar.com
              • with subject: *(HELC-986) FIWARE.Request.Lab.Data.BigData-Analysis.CygnusChannelFullError *

              Hi again Miguel,

              we have found a bug in Cygnus 0.8.2 related to the automatic creation of Hive tables that may be, as a side effect, reducing the performance of the HDFS sink. You can see the details at https://github.com/telefonicaid/fiware-cygnus/issues/513

              Not creating the Hive tables is a minor problem, the real problem is the Hive API invocation for creating the table needs a large timeout to finally find out the table cannot be created. That must be reducing the performace for sure.

              Currently working on fixing it. Next release (0.9.0) should be ready in a few days. I'll send a notification to all FIWARE users regarding that.

              Regards,
              Francisco

              Show
              frb Francisco Romero added a comment - The issue has been emailed: Time sent: 16/Sep/15 11:42 AM To: miguel@talkykar.com with subject: *( HELC-986 ) FIWARE.Request.Lab.Data.BigData-Analysis.CygnusChannelFullError * Hi again Miguel, we have found a bug in Cygnus 0.8.2 related to the automatic creation of Hive tables that may be, as a side effect, reducing the performance of the HDFS sink. You can see the details at https://github.com/telefonicaid/fiware-cygnus/issues/513 Not creating the Hive tables is a minor problem, the real problem is the Hive API invocation for creating the table needs a large timeout to finally find out the table cannot be created. That must be reducing the performace for sure. Currently working on fixing it. Next release (0.9.0) should be ready in a few days. I'll send a notification to all FIWARE users regarding that. Regards, Francisco
              frb Francisco Romero made changes -
              Status In Progress [ 3 ] Answered [ 10104 ]
              Hide
              Permalink
              frb Francisco Romero added a comment - - edited

              This was sent by email to the user:

              Hi Miguel,

              Sorry for the late reply, I was on holidays.

              That kind of error appears when the data producer (the NGSI-like notifications originator) is faster than the data consumer (Cygnus’s sinks). Please consider this guidelines for improving the performance:
              https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/operation/performance_tuning_tips.md

              In addition, if you are not using the Grouping feature, simply avoid that part in the configuration (even when there are no grouping rules, the notified events are iterated if configured the interceptor):

              cygnusagent.sources.http-source.interceptors = ts
              cygnusagent.sources.http-source.interceptors.ts.type = timestamp
              #the following lines are not necessary
              #cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
              #cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf

              Finally, we are considering the usage of “binary” APIs instead of REST-based ones when dealing with the backends in order to improve the performance, please check:
              https://github.com/telefonicaid/fiware-cygnus/issues/301

              Hope this helps. Regards,
              Francisco

              Show
              frb Francisco Romero added a comment - - edited This was sent by email to the user: Hi Miguel, Sorry for the late reply, I was on holidays. That kind of error appears when the data producer (the NGSI-like notifications originator) is faster than the data consumer (Cygnus’s sinks). Please consider this guidelines for improving the performance: https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/operation/performance_tuning_tips.md In addition, if you are not using the Grouping feature, simply avoid that part in the configuration (even when there are no grouping rules, the notified events are iterated if configured the interceptor): cygnusagent.sources.http-source.interceptors = ts cygnusagent.sources.http-source.interceptors.ts.type = timestamp #the following lines are not necessary #cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder #cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf Finally, we are considering the usage of “binary” APIs instead of REST-based ones when dealing with the backends in order to improve the performance, please check: https://github.com/telefonicaid/fiware-cygnus/issues/301 Hope this helps. Regards, Francisco
              frb Francisco Romero made changes -
              Status Open [ 1 ] In Progress [ 3 ]
              frb Francisco Romero made changes -
              Summary [Fiware-finodex-coaching] Problems with Cygnus v0.8.2 - Error appending event to channel. Channel might be full FIWARE.Request.Lab.Data.BigData-Analysis.CygnusChannelFullError
              pasquale.vitale Pasquale Vitale made changes -
              Component/s FINODEX [ 10364 ]
              pasquale.vitale Pasquale Vitale made changes -
              Field Original Value New Value
              Assignee Francisco Romero [ frb ]
              fw.ext.user FW External User created issue -

                People

                • Assignee:
                  pasquale.vitale Pasquale Vitale
                  Reporter:
                  fw.ext.user FW External User
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: