Details
-
Type:
extRequest
-
Status: Closed
-
Priority:
Major
-
Resolution: Done
-
Component/s: FINODEX
-
Labels:None
Description
Hello
We are having troubles with Cygnus. When we have some devices sending
information to Context Broker (and context broker subscripted to Cygnus,
sending information to Cosmos and CKAN), the log show errors
There are errors both ckan-channel and hdfs-channel
Here a subset of log.
02 Sep 2015 06:46:07,948 WARN [1701535537@qtp-1850408381-327]
(org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost:203) -
Error appending event to channel. Channel might be full. Consider
increasing the channel capacity or make sure the sinks perform faster.
org.apache.flume.ChannelException: Unable to put batch on required channel:
org.apache.flume.channel.MemoryChannel
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
at
org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:201)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.flume.ChannelException: Space for commit to queue
couldn't be acquired Sinks are likely not keeping up with sources, or the
buffer size is too tight
at org.apache.flume.channel.MemoryChannel
The complete log is available here:
https://dl.dropboxusercontent.com/u/2537618/cygnus_a.log
–
_______________________________________________
Fiware-finodex-coaching mailing list
Fiware-finodex-coaching@lists.fi-ware.org
https://lists.fi-ware.org/listinfo/fiware-finodex-coaching
[Created via e-mail received from: MAPT Talkykar <miguel@talkykar.com>]
Issue Links
- relates to
-
HELP-4512
FIWARE.Request.Tech.Data.BigData-Analysis.CygnusChannelFullError
-
- Closed
-
Activity
| Field | Original Value | New Value |
|---|---|---|
| Assignee | Francisco Romero [ frb ] |
| Component/s | FINODEX [ 10364 ] |
| Summary | [Fiware-finodex-coaching] Problems with Cygnus v0.8.2 - Error appending event to channel. Channel might be full | FIWARE.Request.Lab.Data.BigData-Analysis.CygnusChannelFullError |
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Status | In Progress [ 3 ] | Answered [ 10104 ] |
The issue has been emailed:
- Time sent: 16/Sep/15 11:42 AM
- To: miguel@talkykar.com
- with subject: *(
HELC-986) FIWARE.Request.Lab.Data.BigData-Analysis.CygnusChannelFullError *
Hi again Miguel,
we have found a bug in Cygnus 0.8.2 related to the automatic creation of Hive tables that may be, as a side effect, reducing the performance of the HDFS sink. You can see the details at https://github.com/telefonicaid/fiware-cygnus/issues/513
Not creating the Hive tables is a minor problem, the real problem is the Hive API invocation for creating the table needs a large timeout to finally find out the table cannot be created. That must be reducing the performace for sure.
Currently working on fixing it. Next release (0.9.0) should be ready in a few days. I'll send a notification to all FIWARE users regarding that.
Regards,
Francisco
| Assignee | Francisco Romero [ frb ] | Pasquale Vitale [ pasquale.vitale ] |
| Component/s | _TECH_ [ 11400 ] |
| Component/s | _TECH_ [ 11400 ] |
| Resolution | Done [ 10000 ] | |
| Status | Answered [ 10104 ] | Closed [ 6 ] |
| Summary | FIWARE.Request.Lab.Data.BigData-Analysis.CygnusChannelFullError | FIWARE.Request.Coach.FINODEX.FIWARE.Request.Lab.Data.BigData-Analysis.CygnusChannelFullError |
| Transition | Time In Source Status | Execution Times | Last Executer | Last Execution Date | |||||
|---|---|---|---|---|---|---|---|---|---|
|
12d 1h 31m | 1 | Francisco Romero | 14/Sep/15 11:07 AM | |||||
|
17m 16s | 1 | Francisco Romero | 14/Sep/15 11:25 AM | |||||
|
16d 4h 15m | 1 | Pasquale Vitale | 30/Sep/15 3:40 PM |
This was sent by email to the user:
Hi Miguel,
Sorry for the late reply, I was on holidays.
That kind of error appears when the data producer (the NGSI-like notifications originator) is faster than the data consumer (Cygnus’s sinks). Please consider this guidelines for improving the performance:
https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/operation/performance_tuning_tips.md
In addition, if you are not using the Grouping feature, simply avoid that part in the configuration (even when there are no grouping rules, the notified events are iterated if configured the interceptor):
cygnusagent.sources.http-source.interceptors = ts
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
#the following lines are not necessary
#cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
#cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
Finally, we are considering the usage of “binary” APIs instead of REST-based ones when dealing with the backends in order to improve the performance, please check:
https://github.com/telefonicaid/fiware-cygnus/issues/301
Hope this helps. Regards,
Francisco