Details
-
Type:
extRequest
-
Status: Closed
-
Priority:
Major
-
Resolution: Done
-
Fix Version/s: 2021
-
Component/s: FIWARE-TECH-HELP
-
Labels:None
-
Sender Email:
-
HD-Chapter:Data
-
HD-Enabler:Kurento
Description
Hi there!
We are implementig a on2many videoconference using Kurento. The
procedure is like this:
- The transmitter opens the videoconference and emits video and audio
using Firefox or Chrome - A receiver joins the conference and gets de video and audio on his
Chrome or Firefox - 18 minutes approx. after initiating the conference, Kurento crashes
And syslog shows:
kernel: [136447.115593] traps: KmsLoop[20422] general protection
ip:7f6c9ed66d36 sp:7f6c3bffeb90 error:0 in
libglib-2.0.so.0.4200.1[7f6c9ed0e000+10a000]
kernel: [158988.522731] KmsLoop[9007]: segfault at 43 ip
00007ffb8d669ec8 sp 00007ffb6277dbb0 error 4 in
libnice.so.10.4.1[7ffb8d65f000+3a000]
Why Kurento crashes in a so predictably way? Any clue?
Our environment:
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty
Version: 6.1.1
Found modules:
Module: 'backgroundextractor' version '6.1.1'
Module: 'chroma' version '6.1.1'
Module: 'core' version '6.1.2'
Module: 'crowddetector' version '6.1.1'
Module: 'elements' version '6.1.1'
Module: 'facesegmentator' version '6.1.1'
Module: 'filters' version '6.1.1'
Module: 'markerdetector' version '6.1.1'
Module: 'platedetector' version '6.1.1'
Module: 'pointerdetector' version '6.1.1'
ii gstreamer1.5-libav:amd64 1.5.2.1~20150901123759.34.g07a7b16.trusty
amd64 libav plugin for GStreamer
ii gstreamer1.5-nice:amd64 0.1.7.1~20151001151322.10.ge060eb5.trusty
amd64 ICE library (GStreamer plugin)
ii gstreamer1.5-plugins-bad:amd64
1.5.91~20151001151316.36.gdaf52b5.trusty amd64 GStreamer plugins
from the "bad" set
ii gstreamer1.5-plugins-base:amd64
1.5.91~20151001151055.18.g79a42eb.trusty amd64 GStreamer plugins
from the "base" set
ii gstreamer1.5-plugins-good:amd64
1.5.91~20151001151133.27.g3b1cc74.trusty amd64 GStreamer plugins
from the "good" set
ii gstreamer1.5-plugins-ugly:amd64
1.5.91~20151001151113.10.g1b92d89.trusty amd64 GStreamer plugins
from the "ugly" set
ii gstreamer1.5-pulseaudio:amd64
1.5.91~20151001151133.27.g3b1cc74.trusty amd64 GStreamer plugin
for PulseAudio
ii gstreamer1.5-x:amd64 1.5.91~20151001151055.18.g79a42eb.trusty
amd64 GStreamer plugins for X11 and Pango
ii kms-background-extractor-6.0
6.1.1.trusty amd64 Kurento
Background Extractor Filter
ii kms-chroma-6.0 6.1.1.trusty
amd64 Kurento Chroma Filter
ii kms-core-6.0 6.1.2.trusty amd64
Kurento core module
ii kms-crowddetector-6.0 6.1.1.trusty
amd64 Kurento Crowd Detector Filter
ii kms-elements-6.0 6.1.1.trusty
amd64 Kurento elements module
ii kms-face-segmentator-6.0 6.1.1.trusty
amd64 Kurento Face Segmentator Filter
ii kms-filters-6.0 6.1.1.trusty
amd64 Kurento filters module
ii kms-jsonrpc-1.0 1.0.0.trusty
amd64 Kurento jsonrpc library
ii kms-markerdetector-6.0 6.1.1.trusty
amd64 AR Marker detector for Kurento
ii kms-platedetector-6.0 6.1.1.trusty
amd64 Kurento Plate Detector Filter
ii kms-pointerdetector-6.0 6.1.1.trusty
amd64 Kurento Pointer Detector Filter
ii kmsjsoncpp 1.6.3~20151001151058.38.g263929e.trusty amd64
Kurento jsoncpp library
ii kurento-media-server-6.0 6.1.1.trusty
amd64 Kurento Media Server
ii libgstreamer-plugins-bad1.5-0:amd64
1.5.91~20151001151316.36.gdaf52b5.trusty amd64 GStreamer
development files for libraries from the "bad" set
ii libgstreamer-plugins-base1.5-0:amd64
1.5.91~20151001151055.18.g79a42eb.trusty amd64 GStreamer
libraries from the "base" set
ii libgstreamer1.5-0:amd64 1.5.91~20151001151012.86.g29e3bbd.trusty
amd64 Core GStreamer libraries and elements
ii libnice10:amd64 0.1.7.1~20151001151322.10.ge060eb5.trusty
amd64 ICE library (shared library)
ii openwebrtc-gst-plugins 0.10.0~20151001151244.100.g61cc54b.trusty
amd64 OpenWebRTC specific GStreamer plugins
/etc/default/kurento-media-server-6.0
START_DAEMON=true
- Additional options that are passed to the Daemon.
DAEMON_OPTS=""
unset GST_PLUGIN_PATH
#enable core dump
ulimit -c unlimited
#unlimit open files
ulimit -n $((($(cat /proc/sys/fs/file-max) * 50) / 100 ))
- Next line enables debug for some kurento classes it can be modified to
change - log level and logged tags
#export GST_DEBUG=1,Kurento*:5
export GST_DEBUG=2,Kurento*:5
- whom the daemons should run as
DAEMON_USER=nobody
DAEMON_LOG=/var/log/kurento-media-server
export KURENTO_LOGS_PATH=$DAEMON_LOG
#export KURENTO_LOG_FILE_SIZE=300
#export KURENTO_NUMBER_LOG_FILES=20
/etc/kurento/kurento.conf.json
{
"mediaServer" : {
"resources":
,
"net" : {
// Uncomment just one of them
/*
"rabbitmq":
*/
"websocket": {
"port": 8888,
//"secure":
,
//"registrar":
,
"path": "kurento",
// "threads": 10
"threads": 20
}
}
}
}
Any comment is more than welcome.
Thanks in advance,
Alberto.
Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost.
Please, send your messages using the new domain (Fiware-tech-help@lists.fiware.org) instead of the old one.
_______________________________________________
Fiware-tech-help mailing list
Fiware-tech-help@lists.fiware.org
https://lists.fiware.org/listinfo/fiware-tech-help
[Created via e-mail received from: Alberto Bambala Arbea <alberto.bambala@vento.tv>]
Activity
Hi,
I've performed the tests again and after 18minutes approx. Kurento crashes.
This is what I saw in syslog after the crash:
Nov 11 00:04:07 kurento01 kernel: [1550487.700124] traps: KmsLoop[14382]
general protection ip:7f5024deed36 sp:7f4fb47efb90 error:0 in
libglib-2.0.so.0.4200.1[7f5024d96000+10a000]
As suggested I grabbed the stack trace:
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/kurento-media-server'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 g_queue_pop_head (queue=0x7f4fb47efbb0) at
/build/buildd/glib2.0-2.42.1/./glib/gqueue.c:516
516 /build/buildd/glib2.0-2.42.1/./glib/gqueue.c: No such file or
directory.
(gdb) bt
#0 g_queue_pop_head (queue=0x7f4fb47efbb0) at
/build/buildd/glib2.0-2.42.1/./glib/gqueue.c:516
#1 0x00007f501ffd4ee7 in ?? () from
/tmp/apport_sandbox_ofEb4J/usr/lib/x86_64-linux-gnu/libnice.so.10
#2 0x00007f501ffdcc2e in ?? () from
/tmp/apport_sandbox_ofEb4J/usr/lib/x86_64-linux-gnu/libnice.so.10
#3 0x00007f5024ddf4d3 in g_timeout_dispatch (source=0x7f4fd00c4570,
callback=<optimized out>, user_data=<optimized out>)
at /build/buildd/glib2.0-2.42.1/./glib/gmain.c:4520
#4 0x00007f5024ddeab5 in g_main_dispatch (context=0x7f4fd0037500) at
/build/buildd/glib2.0-2.42.1/./glib/gmain.c:3111
#5 g_main_context_dispatch (context=context@entry=0x7f4fd0037500) at
/build/buildd/glib2.0-2.42.1/./glib/gmain.c:3710
#6 0x00007f5024ddee18 in g_main_context_iterate
(context=0x7f4fd0037500, block=block@entry=1, dispatch=dispatch@entry=1,
self=<optimized out>)
at /build/buildd/glib2.0-2.42.1/./glib/gmain.c:3781
#7 0x00007f5024ddf0da in g_main_loop_run (loop=0x7f4fd0001910) at
/build/buildd/glib2.0-2.42.1/./glib/gmain.c:3975
#8 0x00007f502354432d in loop_thread_init (data=<optimized out>)
at
/var/lib/jenkins/workspace/util-generate-debian-package/label/media-server-64/slave_version/master/version/master/kms-core/src/gst-plugins/commons/kmsloop.c:102
#9 0x00007f5024e03e45 in g_thread_proxy (data=0x7f4fdc0c55e0) at
/build/buildd/glib2.0-2.42.1/./glib/gthread.c:764
#10 0x00007f5026445182 in start_thread (arg=0x7f4fb47f0700) at
pthread_create.c:312
#11 0x00007f5023c7147d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thanks,
Alberto.
Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost.
Please, send your messages using the new domain (Fiware-tech-help@lists.fiware.org) instead of the old one.
Hi again,
I also reproduced the crash with Kurento's one2many example in the tutorial.
- FF 41.0.2 as presenter
- Chrome 46.0.2490.80 as viewer
After 17 minutes or so Kurento crashes with this error in syslog:
Nov 11 12:09:05 kurento01 kernel: [1593985.842165] KmsLoop[8300]:
segfault at 24 ip 00007fc3fc407d45 sp 00007fc395ffab90 error 6 in
libglib-2.0.so.0.4200.1[7fc3fc3af000+10a000]
And from media-server_error.log
libdc1394 error: Failed to initialize libdc1394
0:00:00.047828598 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstsid.so' has invalid
datetime '2015-09-18/created>'
0:00:00.053614933 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgsttwolame.so' has
invalid datetime '2015-09-18/created>'
0:00:00.078000344 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstcdio.so' has invalid
datetime '2015-09-18/created>'
0:00:00.085643833 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstamrwbdec.so' has
invalid datetime '2015-09-18/created>'
0:00:00.086944351 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstasf.so' has invalid
datetime '2015-09-18/created>'
0:00:00.108846026 9326 0x20f6a40 WARN default
gstsf.c:98:gst_sf_create_audio_template_caps: format 0x120000: 'AVR
(Audio Visual Research)' is not mapped
0:00:00.108887257 9326 0x20f6a40 WARN default
gstsf.c:98:gst_sf_create_audio_template_caps: format 0x180000: 'CAF
(Apple Core Audio File)' is not mapped
0:00:00.108900916 9326 0x20f6a40 WARN default
gstsf.c:98:gst_sf_create_audio_template_caps: format 0x100000: 'HTK (HMM
Tool Kit)' is not mapped
0:00:00.108912642 9326 0x20f6a40 WARN default
gstsf.c:98:gst_sf_create_audio_template_caps: format 0xc0000: 'MAT4 (GNU
Octave 2.0 / Matlab 4.2)' is not mapped
0:00:00.108923028 9326 0x20f6a40 WARN default
gstsf.c:98:gst_sf_create_audio_template_caps: format 0xd0000: 'MAT5 (GNU
Octave 2.1 / Matlab 5.0)' is not mapped
0:00:00.108932501 9326 0x20f6a40 WARN default
gstsf.c:98:gst_sf_create_audio_template_caps: format 0x210000: 'MPC
(Akai MPC 2k)' is not mapped
0:00:00.108984209 9326 0x20f6a40 WARN default
gstsf.c:98:gst_sf_create_audio_template_caps: format 0xe0000: 'PVF
(Portable Voice Format)' is not mapped
0:00:00.108996643 9326 0x20f6a40 WARN default
gstsf.c:98:gst_sf_create_audio_template_caps: format 0x160000: 'SD2
(Sound Designer II)' is not mapped
0:00:00.109008553 9326 0x20f6a40 WARN default
gstsf.c:98:gst_sf_create_audio_template_caps: format 0x190000: 'WVE
(Psion Series 3)' is not mapped
0:00:00.149821092 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstmpeg2dec.so' has
invalid datetime '2015-09-18/created>'
0:00:00.150606867 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstdvdlpcmdec.so' has
invalid datetime '2015-09-18/created>'
0:00:00.167420332 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstamrnb.so' has invalid
datetime '2015-09-18/created>'
0:00:00.168654193 9326 0x20f6a40 WARN ladspa
gstladspa.c:455:plugin_init: no LADSPA plugins found, check LADSPA_PATH
0:00:00.229372059 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgsta52dec.so' has
invalid datetime '2015-09-18/created>'
0:00:00.262183479 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstdvdread.so' has
invalid datetime '2015-09-18/created>'
0:00:00.263097622 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstdvdsub.so' has
invalid datetime '2015-09-18/created>'
0:00:00.263754737 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstrmdemux.so' has
invalid datetime '2015-09-18/created>'
0:00:00.285222719 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstx264.so' has invalid
datetime '2015-09-18/created>'
0:00:00.291322454 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstlame.so' has invalid
datetime '2015-09-18/created>'
0:00:00.366747790 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstmad.so' has invalid
datetime '2015-09-18/created>'
0:00:00.369949901 9326 0x20f6a40 ERROR GST_PLUGIN_LOADING
gstplugin.c:811:_priv_gst_plugin_load_file_for_registry: GstPluginDesc
for '/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstxingmux.so' has
invalid datetime '2015-09-18/created>'
0:00:00.381012199 9326 0x20f6a40 WARN audioresample
gstaudioresample.c:1559:plugin_init: Orc disabled, can't benchmark int
vs. float resampler
0:00:00.381083609 9326 0x20f6a40 WARN GST_PERFORMANCE
gstaudioresample.c:1562:plugin_init: orc disabled, no benchmarking done
0:00:00.467176166 9325 0xe53b30 WARN GST_REGISTRY
gstregistry.c:1830:gst_update_registry: registry update failed: Error
writing registry cache to
/nonexistent/.cache/gstreamer-1.5/registry.x86_64.bin: No such file or
directory
The stack trace is as follows:
(gdb) bt
#0 g_queue_pop_head (queue=0x7fc395ffabb0) at
/build/buildd/glib2.0-2.42.1/./glib/gqueue.c:518
#1 0x00007fc3f75edee7 in ?? () from
/tmp/apport_sandbox_sdwyy0/usr/lib/x86_64-linux-gnu/libnice.so.10
#2 0x00007fc3f75f5c2e in ?? () from
/tmp/apport_sandbox_sdwyy0/usr/lib/x86_64-linux-gnu/libnice.so.10
#3 0x00007fc3fc3f84d3 in g_timeout_dispatch (source=0x7fc3d4005720,
callback=<optimized out>, user_data=<optimized out>)
at /build/buildd/glib2.0-2.42.1/./glib/gmain.c:4520
#4 0x00007fc3fc3f7ab5 in g_main_dispatch (context=0x7fc3d40010a0) at
/build/buildd/glib2.0-2.42.1/./glib/gmain.c:3111
#5 g_main_context_dispatch (context=context@entry=0x7fc3d40010a0) at
/build/buildd/glib2.0-2.42.1/./glib/gmain.c:3710
#6 0x00007fc3fc3f7e18 in g_main_context_iterate
(context=0x7fc3d40010a0, block=block@entry=1, dispatch=dispatch@entry=1,
self=<optimized out>)
at /build/buildd/glib2.0-2.42.1/./glib/gmain.c:3781
#7 0x00007fc3fc3f80da in g_main_loop_run (loop=0x7fc3d4001890) at
/build/buildd/glib2.0-2.42.1/./glib/gmain.c:3975
#8 0x00007fc3fab5d32d in loop_thread_init (data=<optimized out>)
at
/var/lib/jenkins/workspace/util-generate-debian-package/label/media-server-64/slave_version/master/version/master/kms-core/src/gst-plugins/commons/kmsloop.c:102
#9 0x00007fc3fc41ce45 in g_thread_proxy (data=0x7fc3c00051e0) at
/build/buildd/glib2.0-2.42.1/./glib/gthread.c:764
#10 0x00007fc3fda5e182 in start_thread (arg=0x7fc395ffb700) at
pthread_create.c:312
#11 0x00007fc3fb28a47d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Regards,
Alberto.
Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost.
Please, send your messages using the new domain (Fiware-tech-help@lists.fiware.org) instead of the old one.
_______________________________________________
Fiware-tech-help mailing list
Fiware-tech-help@lists.fiware.org
https://lists.fiware.org/listinfo/fiware-tech-help
We cannot reproduce the issue in our facilities. For being able to track the problem we need this information:
1. Where are you deploying Kurento? Is it in the FILAB? In which flavour? If it is out of FILAB, please, provide detailed information on the type of machine where you are deploying including type of CPU, available memory and available HD.
2. Are you using a STUN or TURN server for Kurento? You can find this information in the /etc/kurento/webrtcendpoint configuration files. Please, share them with us.
3. Which browser are you using for both presenter and the viewer? Are you using a STUN / TURN server at the browser-side?
Luis, you have to follow the procedure described in the link below in order to contact the user. Otherwise, he doesn't receive any email. Thanks.
http://backlog.fiware.org/guide/tracker.html#email-issue
We cannot reproduce the issue in our facilities. For being able to track the problem we need this information:
1. Where are you deploying Kurento? Is it in the FILAB? In which flavour? If it is out of FILAB, please, provide detailed information on the type of machine where you are deploying including type of CPU, available memory and available HD.
2. Are you using a STUN or TURN server for Kurento? You can find this information in the /etc/kurento/webrtcendpoint configuration files. Please, share them with us.
3. Which browser are you using for both presenter and the viewer? Are you using a STUN / TURN server at the browser-side?
We have been testing in our environment and we cannot reproduce the problem. We maintained a one-to-many call with 1 presenter and 10 viewers during 3 hours without running into any kind of error. Our test was based on
Our guess is that your application can have problems such as memory leaks or incorrect management of KMS resources. However, in order to find out if there is really a KMS error we would need you to execute the following procedure for recovering the stack-trace of the segfault
Install apport-retrace
Run your application until KMS crashes. Then, get the stack trace
For the next forms, press the bold key
What would you like to do? Your options are:
S: Send report (1.1 MB)
E: Examine locally
V: View report
K: Keep report file for sending later or copying to somewhere else
I: Cancel and ignore future crashes of this program version
C: Cancel
Please choose (S/E/V/K/I/C): E
Choices:
1: Run gdb session
2: Run gdb session without downloading debug symbols
3: Update /var/crash/_usr_bin_kurento-media-server.1000.crash with fully symbolic stack trace
C: Cancel
Please choose (1/2/3/C): 1
Wait for packages installation
Get the stack trace in gdb console
(gdb) bt
Copy and send the output of bt to us