Uploaded image for project: 'Help-Coaches-Desk'
  1. Help-Coaches-Desk
  2. HELC-986

FIWARE.Request.Coach.FINODEX.FIWARE.Request.Lab.Data.BigData-Analysis.CygnusChannelFullError

        Details

        • Type: extRequest
        • Status: Closed
        • Priority: Major
        • Resolution: Done
        • Component/s: FINODEX
        • Labels:
          None

          Description

          Hello

          We are having troubles with Cygnus. When we have some devices sending
          information to Context Broker (and context broker subscripted to Cygnus,
          sending information to Cosmos and CKAN), the log show errors

          There are errors both ckan-channel and hdfs-channel

          Here a subset of log.

          02 Sep 2015 06:46:07,948 WARN [1701535537@qtp-1850408381-327]
          (org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost:203) -
          Error appending event to channel. Channel might be full. Consider
          increasing the channel capacity or make sure the sinks perform faster.
          org.apache.flume.ChannelException: Unable to put batch on required channel:
          org.apache.flume.channel.MemoryChannel

          {name: ckan-channel}

          at
          org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
          at
          org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:201)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
          at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
          at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
          at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
          at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
          at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
          at org.mortbay.jetty.Server.handle(Server.java:326)
          at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
          at
          org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
          at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
          at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
          at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
          at
          org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
          at
          org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
          Caused by: org.apache.flume.ChannelException: Space for commit to queue
          couldn't be acquired Sinks are likely not keeping up with sources, or the
          buffer size is too tight
          at org.apache.flume.channel.MemoryChannel

          The complete log is available here:
          https://dl.dropboxusercontent.com/u/2537618/cygnus_a.log

          –

          _______________________________________________
          Fiware-finodex-coaching mailing list
          Fiware-finodex-coaching@lists.fi-ware.org
          https://lists.fi-ware.org/listinfo/fiware-finodex-coaching

          [Created via e-mail received from: MAPT Talkykar <miguel@talkykar.com>]

            Issue Links

            relates to

            extRequest - Request from external stakeholder received in the help desk HELP-4512 FIWARE.Request.Tech.Data.BigData-Analysis.CygnusChannelFullError

            • Major - Major loss of function.
            • Closed

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              frb Francisco Romero added a comment - - edited

              This was sent by email to the user:

              Hi Miguel,

              Sorry for the late reply, I was on holidays.

              That kind of error appears when the data producer (the NGSI-like notifications originator) is faster than the data consumer (Cygnus’s sinks). Please consider this guidelines for improving the performance:
              https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/operation/performance_tuning_tips.md

              In addition, if you are not using the Grouping feature, simply avoid that part in the configuration (even when there are no grouping rules, the notified events are iterated if configured the interceptor):

              cygnusagent.sources.http-source.interceptors = ts
              cygnusagent.sources.http-source.interceptors.ts.type = timestamp
              #the following lines are not necessary
              #cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
              #cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf

              Finally, we are considering the usage of “binary” APIs instead of REST-based ones when dealing with the backends in order to improve the performance, please check:
              https://github.com/telefonicaid/fiware-cygnus/issues/301

              Hope this helps. Regards,
              Francisco

              Show
              frb Francisco Romero added a comment - - edited This was sent by email to the user: Hi Miguel, Sorry for the late reply, I was on holidays. That kind of error appears when the data producer (the NGSI-like notifications originator) is faster than the data consumer (Cygnus’s sinks). Please consider this guidelines for improving the performance: https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/operation/performance_tuning_tips.md In addition, if you are not using the Grouping feature, simply avoid that part in the configuration (even when there are no grouping rules, the notified events are iterated if configured the interceptor): cygnusagent.sources.http-source.interceptors = ts cygnusagent.sources.http-source.interceptors.ts.type = timestamp #the following lines are not necessary #cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder #cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf Finally, we are considering the usage of “binary” APIs instead of REST-based ones when dealing with the backends in order to improve the performance, please check: https://github.com/telefonicaid/fiware-cygnus/issues/301 Hope this helps. Regards, Francisco
              Hide
              Permalink
              frb Francisco Romero added a comment -

              The issue has been emailed:

              • Time sent: 16/Sep/15 11:42 AM
              • To: miguel@talkykar.com
              • with subject: *(HELC-986) FIWARE.Request.Lab.Data.BigData-Analysis.CygnusChannelFullError *

              Hi again Miguel,

              we have found a bug in Cygnus 0.8.2 related to the automatic creation of Hive tables that may be, as a side effect, reducing the performance of the HDFS sink. You can see the details at https://github.com/telefonicaid/fiware-cygnus/issues/513

              Not creating the Hive tables is a minor problem, the real problem is the Hive API invocation for creating the table needs a large timeout to finally find out the table cannot be created. That must be reducing the performace for sure.

              Currently working on fixing it. Next release (0.9.0) should be ready in a few days. I'll send a notification to all FIWARE users regarding that.

              Regards,
              Francisco

              Show
              frb Francisco Romero added a comment - The issue has been emailed: Time sent: 16/Sep/15 11:42 AM To: miguel@talkykar.com with subject: *( HELC-986 ) FIWARE.Request.Lab.Data.BigData-Analysis.CygnusChannelFullError * Hi again Miguel, we have found a bug in Cygnus 0.8.2 related to the automatic creation of Hive tables that may be, as a side effect, reducing the performance of the HDFS sink. You can see the details at https://github.com/telefonicaid/fiware-cygnus/issues/513 Not creating the Hive tables is a minor problem, the real problem is the Hive API invocation for creating the table needs a large timeout to finally find out the table cannot be created. That must be reducing the performace for sure. Currently working on fixing it. Next release (0.9.0) should be ready in a few days. I'll send a notification to all FIWARE users regarding that. Regards, Francisco

                People

                • Assignee:
                  pasquale.vitale Pasquale Vitale
                  Reporter:
                  fw.ext.user FW External User
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: